Top 5 Reasons for CI Failure

Agile software development can't be perfect without an effective Continuous Integration process. CI is a process of continuously analyzing, building, testing, and deploying software. Continuous Integration checks the internal quality of the code and tests the business logic of the product before it's released to Production.

Ideally, we shouldn't allow software to deploy to Production when the build is broken. However, Continuous Integration doesn't always work for every Agile team. Some Agile teams take CI practices very seriously, some teams do it for sake of doing Agile, some teams ignore it completely, and some doesn't have a CI server set up yet.

There are various reasons that CI practices might be ignored within the team. Businesses have different priorities, Product Owners doesn't always understand the importance of internal quality, testing processes, and clean builds. The Technical Manager can't buy time to implement the CI practices or fix a broken CI. Production and Technology Management fail to understand each other's priorities and they end up deploying the broken build to end users. The Production people are happy — they delivered business value with broken CI.

This approach can give temporary satisfaction, but it's very dangerous. It can later throw severe bugs into Production, which can heavily impact business. The severity of an impact is unpredictable, ranging from the loss of money to the loss of reputation or, in an extreme case, the loss of the business completely.

However, even if the Production and Technology teams agree to invest time and money to implement or fix CI issues, some teams never succeed. Let's discuss top five reasons for CI failures and potential solutions to overcome the problems.

1. Wrong Selection of CI Server

There are various Continuous Integration tools available in the market. The CI server solution can be self-hosted or on the cloud. You can get a whole list of CI servers here with recommendations.

Jenkins is one of the popular CI servers, and people have tended to use it blindly. We've had to adjust our project to work with Jenkins and the team has to compromise with the services that Jenkins offers. Now, that situation has changed and there are various promising CI services in the market. Looking at the wide range of CI servers in the market, it's challenging to select one that suits our project need.

The process of setting up a CI server takes time and money. If you choose CI server without doing any research and it doesn't work well for the team, then all the efforts you put in to get CI server will be in vain. A common mistake that management makes is selecting a generic CI server or service for all platforms like that doesn't work quite well. Imagine that your application has a website, iOS app, and Android app; finding a generic solution might not work well. We have to be very careful about selecting our CI server.

Potential Solutions

• Research the market very well and evaluate options on pilot projects. Slant has already evaluated major CI servers with pros and cons of each.
• Look for the features like pipeline support, pipeline as code, Docker container support, platform support, ease of use, availability, etc. while evaluating CI servers.
• Don't try to find a generic solution for all of the platforms to reduce cost. Each platform has different technical requirements and challenges.
• Discuss it with the team and ask for past experiences so that we don't need to train engineers again.

2. CI Amateur Engineers

An engineer working in the Agile team should have great coding skills, but delivering a working software is more than just writing and testing code. It also involves setting up the appropriate environment to make sure we can deliver it easily. This requires strong command line and scripting skills for build automation as well as sound knowledge of build automation tools and dependency/package management tools.

Recently, companies have been moving the infrastructure to cloud, so there is a need to learn DevOps skills, i.e., cloud services like AWS, Azure, and Heroku; provisioning tools like bash, Ansible, and Chef; and container services like Docker and Kubernetes. The most important thing is having knowledge of one of the scripting languages, i.e., Bash, Ruby, or Python.

This doesn't mean that you should learn everything in the world, but you should learn everything for your platform. Let's assume that an engineer is working on iOS application development. The engineer should know tools like Cocoapods, Carthage, and Swift Package Manager for dependency management.

Also, build automation tools like Fastlane, Rake, and Make have a strong command over Apple's native command line tools. An engineer should be aware of recent tools related to the relevant platform.

There are different types of engineers in the market; some of them are good at writing native language code (i.e., Java, Objective-C, and Swift) and have a strong knowledge of DevOps and build automation tools. Some engineers just write code in an IDE (i.e., Eclipse, IntelliJ, and Xcode) are not capable of running commands from the terminal. Some engineers are better at build automation tools but are not that good at writing native language code.

CI amateurs are those who cannot get out of IDE and are not capable of using command line (scripting) tools. They prefer GUI tools to everything than command line or scripting. However, the CI server doesn't have a GUI interface to interact with, so everything has to be scripted.

If you have CI amateur engineers on your team, CI practices are never going to be successful. The CI amateur developers might end up writing poor quality build automation scripts that break all the time. The team ends up spending time on learning, improving build automation, and switching between the CI servers rather than building features that are useful for the business.

Potential Solutions

• Hire engineers who have basic knowledge of Continuous Integration and DevOps.
• Train CI amateur engineers. A good idea is to send them to external training or train them with in-house CI experts.
• Hire experts on short-term contracts to set up the CI process and share knowledge.

3. Pipeline Isn't Coded

Most CI servers allow users to change the build configuration from the web interface. This approach makes it easy for engineers to create and edit the CI builds. However, changing build configuration frequently can cause a lot of issues, such as disabling important build steps. Additionally, everyone has permission to access the build machines, which can cause confusion as to who did what and when they did it. It takes long to figure out build failures if engineers are not aware of build setting changes. Frequent changes in the CI server machine can cause problems and confusions within the team.

Potential Solution

• Take control of build configuration by putting the config file, bash script, or similar file inside the source code.
• Avoid manual configurations on CI server.
• Restrict access to the CI server machine to a few people and give them the responsibility to manage the server.
• Do not allow users to modify particular build steps.

4. Poor Quality Build Server Machines

Developers working on the team frequently check in code in the source control system, which triggers a build on the CI server. This means that the CI server machine is continuously running jobs that can involve heavy tasks such as downloading dependencies from remote servers, backing up databases, running Docker containers, etc. In order to perform those task effectively, the CI server must be fast, reliable, and connected to the network. Using a poor-quality server wastes everyone's time because the build takes too long to finish, resulting in intermittent test results and frustrated engineers.

Potential Solutions

• Get a quick and reliable server for Continuous Integration; avoid cheap servers.
• Never leave the CI server machine on WiFi.
• Never install unnecessary software on the CI server machine.
• Smartly share the CI server machine by assigning specific jobs.
• Never install any software manually.
• Avoid giving GUI access to machines. SSH access should be sufficient.

5. Lack of Management Support

Project management plays a key role in enforcing CI practices within the team. The management should set strict guidelines about the CI builds and should have zero tolerance for broken builds. The software shouldn't be released with a broken build in any case. A broken CI build should be treated as an emergency and the whole team should work to fix it before they commit any further code. The technical manager or relevant person from management can set this guideline and implement it strictly within the team. Managers without any understanding of Continuous Integration are likely to carry on with feature development regardless of code quality. Continuous Integration practices won't be successful under such management.

Potential Solutions

• Set up CI practices with team and apply them strictly within team.
• Train Project Managers on CI practices.

Conclusion

It's challenging to implement Continuous Integration practices within an Agile team, but following some strict rules and avoiding common mistakes can make the CI process much effective. What are your experiences with CI servers? Do you feel your CI practices are working well? Weigh in with a comment below!