CI/CD Best Practices: Top 10 Practices for Financial Services

As you probably already know, creating and releasing software has become challenging as teams, applications, and deployment infrastructure gets more sophisticated. Developers and organizations have come up with three interconnected but independent ways to organize and automate the processes involved in software development, testing, and release so that they can be done quickly and consistently.

Furthermore, these techniques involve little or even zero human implication while creating new code and subsequent deployment. We are referring to the three primary approaches: Continuous Integration, Continuous Delivery, and Continuous Deployment. This article won’t go into the details of CI/CD but will rather take a look at CI/CD Best Practices. 

In this post, we aim to introduce you to CI/CD Best Practices that should be followed when implementing CI/CD Pipelines in your organization or when planning to enhance the existing CI/CD Pipelines. 

Before we go ahead and introduce you to our top 10 CI/CD Best Practices, let’s take a quick look at the three aforementioned approaches in order to understand better why it’s important to have CI/CD in place and in line with the best practices.

What Is CI (Continuous Integration)?

Continuous Integration is a coding philosophy and set of procedures that encourage development teams to frequently execute tiny code modifications and check them into a version control repository. 

Although “CI” may have other connotations, it always refers to Continuous Integration when used in CI/CD. CI is a process of automation for developers that denotes the process of routinely building, testing, and merging new code modifications to an app into a shared repository. Continuous Integration creates an automated system for developing, packaging, and testing applications. It’s an approach that encourages developers to undertake code changes more frequently, thus improving collaboration and code quality.

In Short: Continuous Integration is responsible for code commits, builds, and automated tests.

What Is CD (Continuous Delivery)? 

The “CD” in CI/CD refers to Continuous Delivery and/or Continuous Deployment. First off, let’s take a look at Continuous Delivery. 

Continuous Delivery often refers to a process in which an application developer makes changes that are automatically tested and uploaded to a repository so that the operations team can deploy those changes to a real-world production environment. Continuous Delivery is a continuation of Continuous Integration in that it puts a lot of emphasis on automating the software delivery process so that teams can quickly and confidently release their work to any environment at any time, whereas releases to a live production environment are done manually.

Simply put, there is an automatic release process on top of the automated testing process. It allows developers to deploy their changes whenever they want by simply hitting a button or once CI is complete.

In Short: Continuous Delivery aims to maintain a codebase that is constantly prepared for deployment to any environment where deployment to the Production environment requires manual approval.

What Is the Other Meaning of CD (Continuous Deployment)?

Continuous Deployment moves the process one step further than Continuous Delivery. Only a failing verification step will stop the modifications from being pushed to the production environment, as the procedure is entirely automated. Continuous Deployment’s goal is to introduce new code into the production environment continuously.

In Short: Unlike Continuous Integration, which requires manual approval to release the code to the production environment, Continuous Deployment automatically deploys the code to the production environment without any explicit approval. 

Why Is It Important To Have a CI/CD in Place?

CI/CD eliminates conventional barriers while automating the software integration, release, and deployment process. It supports the DevOps strategy of unifying the development and operation teams and the bigger objective of agile methodology to shorten the software development lifecycle.

The fact that CI/CD frees up resources makes it a great DevOps practice. Automation allows teams to release to customers immediately, test as necessary, and integrate changes swiftly. Usually, these tasks would require distinct procedures taking more time and effort if CI/CD weren’t in place.

Continuous Testing, Quicker Release, Real-Time Feedback, Customer Satisfaction, and Reduced risk of faulty code in Production are a few of the many benefits of having CI/CD in place.

Consideration

In this article, we will be considering the following tools around which our best practices revolve. There are various alternatives available on the market for each tool mentioned below. However, our list of CI/CD best practices remains the same regardless. 

1. Git 
2. GitHub
3. Jenkins

You can either go through the following CI/CD Pipeline Best Practices now on a theoretical level and integrate them into your CI/CD Pipelines later, or you can try and create a sample CI/CD Pipeline while reading this article. If you want to try the pipeline on your own as you read, you need the toolset mentioned above.

Now, let’s go ahead and take a look at the 10 CI/CD Pipeline Best Practices that you should consider in your CI/CD Pipelines.

Top 10 CI/CD Best Practices

CI/CD Best Practice 1: Strict Version Control

The first step to starting with CI/CD Pipelines is ensuring the codebase is under version control. This is the most crucial prerequisite. Every modification made to the codebase needs to be securely kept in a separate Version Control System (VCS). When the code is version controlled, it allows for a CI/CD tool like Jenkins to access it.

Typically, developers only utilize Version Control Systems for an application’s source code and ignore other supporting files such as installation scripts, configuration settings, and test results. That said, every aspect of the application life cycle involving source control should be checked in. The ultimate goal of having a Version Control System (VCS) is for anyone to be able to verify everything related to an application and recreate it locally or in any other environment.

Also, it is to be noted that CI/CD Pipelines are often configured to track and test changes made to just one or a small number of branches. Therefore, in order to benefit from CI, it’s best to restrict the number and scope of branches in your repository and have a branching strategy in place.

Tip 1: Our first CI/CD best practice and one of the most important CI/CD best practices is to achieve strict version control. To do so, you can use Git as a version control tool which will enable you to manage and monitor the history of your source code, and GitHub as a cloud-based hosting service to manage your Git repositories.

CI/CD Best Practice 2: Commit Early and Commit Often

Make small, frequent commits. Iterative processes rely on small, manageable commits. When testing is done in small batches, teams can iterate more successfully and enhance code quality.

Larger commits tend to become unmanageable and are more susceptible to catastrophic failure. Moreover, large builds have the potential to overcommit resources.

Iterative processes are built on small, frequent commits, as this means less code to look through. When a development team submits code changes frequently and early on, bugs are simpler to find. Remember, the more frequently you commit changes, the quicker you will receive feedback on the changes you bring. In order to ensure that everyone on the team is aware of all of the changes in the works, the common norm is to commit at least once every day.

Tip 2: You should commit at least once a day and share your code with other team members frequently in order to ensure that they are up to date. This will help improve code quality, reduce merge conflicts, and help identify bugs early on. 

CI/CD Best Practice 3: Save Your CI/CD Pipeline Configuration as Code

In the same way, source control for your application’s source code is vital, and so is CI/CD configuration. Therefore, you should keep CI/CD configuration files alongside your application source code in your source control management system. This practice is known as “Configuration as Code.”

When it comes to Jenkins, you can create CI/CD pipelines in two ways:

1. Freestyle projects: These are configured via the Jenkins UI. These types of pipelines are called scripted pipelines.
2. Pipeline projects: These are configured via a file named Jenkinsfile in the root of the repo. These types of pipelines are called declarative pipelines.

Out of the two aforementioned options, you should always prioritize creating pipelines using Jenkinsfile, i.e., you should always aim to create declarative pipelines.

A declarative pipeline makes pipeline code easier to read and write while allowing developers to access, update, and check it at all times. Jenkinsfiles, which can be checked into source control management systems, contain the configuration code. The declarative pipeline is defined within a block with the label “pipeline.” 

Here is a sample template of what the declarative pipeline looks like. Click here to learn more about pipelines.

Jenkinsfile (Declarative Pipeline)

pipeline {
    agent any

    stages {
        stage(‘Initialize’) {
            steps {
                echo ‘Initialize your project here.’
            }
        }
        stage(‘Build’) {
            steps {
                echo ‘Build the code here.’
            }
        }
        stage(‘Test’) {
            steps {
                echo ‘Test the code here.’
            }
        }
        stage(‘Deploy’) {
            steps {
                echo ‘Deploy the code here.’
            }
        }
    }
}

Tip 3: As a best CI/CD practice, always store your CI/CD pipeline configuration as code. When working with Jenkins, declarative pipelines must be your first choice over scripted pipelines. 

CI/CD Best Practice 4: Establish Access Controls for CI/CD Tools

It’s crucial to safeguard the CI/CD system seeing as it has full access to your codebase and credentials in order to deploy in various environments. Attackers that succeed in breaching one environment may be able to breach other environments because of loopholes in your CI/CD servers if they are not properly isolated or secured. 

The important point to keep in mind is that your CI/CD systems are highly valuable targets and in many cases, they allow for a broad degree of access to your other vital systems. Shielding all external access to the servers and tightly controlling the types of internal access allowed will help reduce the risk of your CI/CD system being compromised.

Additionally, not every employee in the business should have access to your CI/CD, and even if they do, they shouldn’t have unrestricted access to all pipelines and opportunities.

E.g., Developers should only have access to the pipelines they require. Having access to the pipelines of other teams serves no purpose. Although they shouldn’t necessarily be able to design pipelines, managers or team leaders may need access to CI/CD for reporting needs. When it comes to Jenkins, Jenkins’ Authorization capability is your friend. Jenkins offers other options as well. Click here to learn more about the various security options available with Jenkins.

Tip 4: The next CI/CD Best Practice restricts access to your CI/CD Environment and CI/CD Pipelines. Make sure that only authorized persons can access the designated pipelines. 

CI/CD Best Practice 5: Secure Your Secrets and Don’t Store Them in Source Code

Access to applications and services requires authentication credentials such as usernames, passwords, API tokens, SSH keys, and encryption keys. These credentials have the potential to lead to a significant data breach if they are not properly protected and used. Moreover, they physically hold the keys to the information and materials relating to a project.

Both your application and your CI/CD tool may require certain secrets in order to deploy applications. You shouldn’t send any secrets via the pipeline in plain text. Furthermore, do not hard-code secret data directly into configuration files or source code. 

The majority of modern CI/CD platforms have a secret management solution, allowing you to safely store your secrets there and feed them to your pipelines as environment variables.

With Jenkins, you can leverage the credentials functionality. Credentials stored in Jenkins can be used anywhere throughout Jenkins by a specific Pipeline project or a specific Jenkins user. Jenkins can store Secret text, Username and password, Secret file, SSH Username with the private key, Certificate, and Docker Host Certificate Authentication credentials. Click here to learn more about Jenkins’s credentials functionality.

Depending on your requirements, you can also use any of the Secret Management Tools available. These include HashiCorp Vault, AWS Secrets Manager, Cloud KMS, Microsoft Azure Key Vault, 1password, and LastPass, to name just a few.

Tip 5: Always keep your secrets as secrets and never store them in your source code repositories. Never make the mistake of storing or passing credentials in a plain text format. 

CI/CD Best Practice 6: Build Only Once, Reuse the Artifacts, and Promote the Same To Higher Environments

Building trust in your modifications and lowering the possibility of unanticipated consequences are two of the main objectives of a CI/CD pipeline. If building, packaging, or bundling is necessary for your application, it should only be done once, and the output should then be utilized throughout the pipeline.

Any methods that require building source code more than once should be avoided. You should only carry out the procedure once and then promote your binaries or artifacts. This prevents oversights and reduces the likelihood of mistakes being made or overlooked in the future. 

The same applies when using artifacts in different environments. You should reuse the same artifact for all environments and should avoid building separately for QA, Staging, UAT, or Production. This enables the deployment of the same build for testing in each environment and builds confidence among team members. 

E.g.

Consider that each step in your pipeline creates a docker image for your application. On some base images, your docker image will be created. There is always a chance that the base image will be updated in the meantime if you don’t lock the original version of that image. Any software you put in the container is subject to the same rules hence why it’s necessary to use the artifact from the initial build throughout the entire pipeline and run and promote it to higher environments.

It’s important to note that the build scripts, configuration files, and deployment scripts are stored in the source control system, like the application code. In contrast, artifacts or docker images generated as part of the build stage must be stored in an artifact repository.

Tip 6: Build packages, docker images, and artifacts only once and reuse them in other pipeline stages and different environments to build confidence among team members in each artifact.

CI/CD Best Practice 7: Define a Deployment and Rollback Strategy

Creating a deployment and rollback strategy is the next CI/CD Best Practice that we have on our list for you. Distributing software to consumers is known as a software release or deployment. Every time you deploy your software, there is a chance that it will have bugs, vulnerabilities, and other problems. There are a number of reasons why you may want to roll back your deployment. Hence it’s important to define a deployment and rollback strategy that works for your software release process in order to reduce the risk of deployments.

You should be able to quickly roll back to the prior stable version of any new code should the newer version break a feature or general application. Also, to prevent production pauses, you should be able to deploy the most recent successful build right away. This is when the deployment and rollback strategy will help you.

Here are various deployment strategies that can help with release and rollback:

1. In-Place Deployments
   The application is deployed using this technique once it has been stopped, the most recent revision of the application has been installed, and the new version has been started and validated.
2. Blue/Green Deployments
   Blue/Green Deployments involve moving traffic between two identical environments running different versions of the application. With Blue/Green Deployments, you may simultaneously launch a new version of your application (green) alongside an existing one (blue), monitor and test the new version before redirecting traffic to it, and roll it back if problems arise.
3. Canary Deployments
   This process can be two-step or linear, with new application code being deployed and made available for testing before being rolled out to the rest of the environment or in a linear form after being approved.
4. Linear Deployments
   When traffic is shifted via a linear deployment, there is always an equal number of minutes between each increment.
5. All-At-Once Deployments
   With this deployment, all traffic is transferred simultaneously from the old environment to the new one.

In order to achieve rollback, you can also try any of the following strategies:

1. Re-running an old pipeline
   As soon as the deployment is made to production, your CI/CD pipeline shouldn’t declare it “done.” Instead, a stage should be included in the CI/CD pipeline to monitor the newly deployed application for a short period of time after deployment and check for defaults. If issues are found, redeploy the previously running version of the application by manually executing the previous run.
2. Use a “rollback” stage with a manual gate
   You can also have an additional stage if you want an automated rollback. Basically, instead of executing the previous run manually, you can automate the rollback.

Tip 7: Have a release and rollback strategy in your CI/CD pipelines so that you can revert back to the stable version of the deployment in case any new code changes break a feature or the application.

CI/CD Best Practice 8: Adopt Parallel Testing

QAs used to manually test each step of the application and assert behavior in the early days of testing. The process of manual testing is laborious and time-consuming. Additionally, it’s vulnerable to human error. Therefore, test automation addressed every drawback of manual testing. More frameworks were developed to speed up testing as Test Automation suites were more widely adopted.

That said, sequential testing becomes time-consuming when there are many test cases and extensive device coverage. Running separate test cases simultaneously is one of the most practical solutions to this problem and also one of our CI/CD Best Practices.

In light of the above, you should consider simultaneously running several independent test cases using parallel testing, seeing as running lengthy test cases sequentially consumes a significant amount of time. Parallel execution of these test cases can shorten the run’s overall completion time. As costs significantly increase when high-performance servers are used to conduct tests in series for longer, you should set up tests to run concurrently so that the underlying hardware utilizes its full potential and runs quicker.

You can try Unit Tests, Code Linting, Integration Tests, and Security Tests in parallel to improve the speed and make the CI/CD Pipeline fail if any of these tests fail to save you time.

Tip 8: As a best practice for CI/CD Pipelines, your CI/CD Pipelines should include parallel tests, seeing as parallel testing allows you to do more in a shorter period of time. 

Jenkinsfile (Declarative Pipeline)

pipeline {
    agent none
    stages {
        stage(‘Initialize’) {
            steps {
                echo ‘Initialize your project here.’
            }
        }
        stage(‘Build’) {
            steps {
                echo ‘Build the code here.’
            }
        }

        stage(‘Tests’) {
            parallel {
                stage(‘Tests On Windows’) {
                    agent {
                        label “windows-node”
                    }
                    steps {
                        bat “run-tests-on-windows-machine.bat”
                    }
                    post {
                        always {
                            junit “**/TEST-*.xml”
                        }
                    }
                }
                stage(‘Tests On Linux’) {
                    agent {
                        label “linux-node”
                    }
                    steps {
                        sh “run-tests-on-linux-machine.sh”
                    }
                    post {
                        always {
                            junit “**/TEST-*.xml”
                        }
                    }
                }
            }
        }
    }
}

CI/CD Best Practice 9: Make Your CI/CD Pipeline the Only Way To Deploy to Production

The fact that tooling frequently helps with enforcing recommended practices for testing and deployment is part of what makes it possible for CI/CD to improve your development processes and code quality. Each change must demonstrate that it complies with your business’s defined rules and processes for it to advance through your CI/CD pipelines. However, in order to take advantage of these CI/CD benefits, you must make sure that every modification to your production environment passes through your pipeline. Your code should only enter the production environment via the CI/CD pipeline. With continuous deployment techniques, this can happen automatically at the completion of successful testing or manually through the manual promotion of tested changes that have been approved and made available by your CI/CD system.

Sometimes, teams frequently use pipelines for deployment, but as issues arise, they add exceptions. It’s crucial to realize that your CI/CD system is useful for ensuring that your modifications don’t introduce new issues or worsen the system. 

Tip 9: You should never manually fix issues on your Production environment or any lower environments; everything must go through your CI/CD Pipelines.

CI/CD Best Practice 10: Involve the Whole Team in the CI/CD Implementation To Create a DevOps Culture

Although CI/CD tools can serve as a starting point, you need more than just tools for software development optimization. To make the most of CI/CD, you must modify your organizational culture using DevOps concepts. The team environment and company culture are just as important to the success of your CI/CD pipeline as the procedures and tools in place. 

For CI/CD to be successfully implemented, all team members must share the CI/CD process. CI/CD is cultural, not technological. In order to encourage active engagement in the process, make it apparent that CI/CD is more than a technical step but a crucial component of your company’s culture. You must encourage everyone on the team to contribute by developing a sense of shared responsibility for delivering your software, whether that means pitching in to fix the build, taking the time to containerize environments, or automating a manual activity.

Keep in mind that your engineers are professionals in software development, not necessarily DevOps. Therefore, they could initially be resistant to implementing CI/CD. Showing them how it will complement their workflow and make their life simpler, not harder, is the best way to win them over.

Tip 10: Build a DevOps or CI/CD culture rather than just CI/CD Pipelines. Encourage a culture of trust among team members so that they can feel comfortable taking risks and exchanging ideas.

Bonus CI/CD Best Practices

Bonus CI/CD Best Practice 1: Save Infrastructure as Code Templates Inside the Automated Process of CI/CD

In our first CI/CD best practice, we talked about “Strict Version Control,” where we recommend storing every aspect of the application life cycle involving source control should be checked in. In our third CI/CD best practice, we suggested “Save Your CI/CD Pipeline Configuration as Code,” which allows developers to access, update, and check your pipelines at all times. 

In one of our blogs that talks about Terraform Best Practices, we recommend “Host Terraform Code in the Git Repository,” i.e., save your Infrastructure as Code in Version Control System.

Now, looking at these three recommendations from a broader perspective, what we suggest is to have your Infrastructure as Code Templates(e.g., Terraform Templates) stored in Version Control System(e.g., GitHub) and integrate the same with the CI/CD process(e.g., Jenkins CI/CD Pipelines). In this way, as part of a deployment, new resources can be provisioned instantly. They don’t need to be made before your pipeline runs, and you can get rid of them as soon as your testing is done. You’ll save time and money this way.

The way your CI/CD pipelines keep an eye on modifications to your code, they can also handle changes to your infrastructure. Changes to system configurations can now be tested, deployed, and integrated just like code following this CI/CD Best Practice.

Bonus Tip 1: Don’t only include your application code in your CI/CD pipeline. Deploy Infrastructure as a part of your CI/CD process and manage it as Code in the Version Control System.

Bonus CI/CD Best Practice 2: Monitor and Gauge Your CI/CD Pipeline

One of the tenets of the DevOps concept is continuous improvement. It encompasses all aspects of software development, including the service or product you’re creating and the organizational culture and procedures. The building, testing, and releasing of your software can be improved by using the same techniques on the CI/CD pipeline itself. This can increase the feedback loops you utilize to enhance your product.

Here are a few operational measures you may use to gauge how well your pipeline is operating and spot potential opportunities to streamline your procedure.

1. Code coverage: As you develop more code, it’s essential to keep an eye on this statistic to ensure you’re keeping a sufficient level of test coverage.
2. Build time: The amount of time needed to finish each stage of the automated pipeline is measured as build time.
3. Test pass rate: The proportion of test cases for a particular build that passed with a successful result is known as the test pass rate.
4. Time to fix tests: The period of time between a build reporting a failed test and the same test passing on a subsequent build is known as the “time to fix tests.”
5. Failed deployments: The count of unsuccessful deployments that cause unwanted downtime need to be rolled back or urgently need a remedy.
6. Defect count: The number of open tickets in your backlog that have been labeled as bugs is referred to as your defect count. Issues discovered in testing or staging and issues discovered in production can be further divided.

Monitoring these metrics helps you to determine whether you see an increasing or decreasing trend and how well your CI/CD pipeline functions. These metrics can help you find areas of your process that need more attention.

Bonus Tip 2: As a CI/CD best practice, monitor your CI/CD pipelines for different metrics, and analyze the metrics to identify potential issues and areas for improvement that can help improve your product.

Conclusion

Our DevOps lifecycle is defined and automated by CI/CD pipelines, which determine when, where, and how software is delivered to clients. Even though every CI/CD implementation will be unique, adhering to the fundamental guidelines will help you stay clear of common problems and improve your testing and development procedures.

In this article, we’ve given you an overview of 10 CI/CD Best Practices that can assist you in creating a strong CI/CD pipeline. These CI/CD Best Practices will help you improve code quality, deliver software updates quickly, store your secrets safely, and foster a DevOps culture within your organization.

Summary