Continuous Test Automation Using CI/CD: How CI/CD Has Revolutionized Automated Testing

This is an article from DZone's 2022 DevOps Trend Report.

For more:

Read the Report

There have been a few breakthroughs throughout the short history of software development that have completely revolutionized the way we write and release code. From Object-Oriented Programming to web-based languages like JavaScript and TypeScript, these innovations have moved software engineering by leaps and bounds. 

One of the more recent groundbreaking innovations is automated testing. Prior to automated testing, a large portion of the test cases for our software were executed manually. This painstaking process has many flaws, including: 

• Inconsistent execution of test cases
• Manualized setup of testing environments
• Tediousness and slowness
• Inconsistent format of test results

Automated testing — and the introduction of Continuous Integration (CI) and Continuous Delivery (CD) — has completely transformed the quality and the cadence with which we release our software. In this article, we will delve into the CI/CD pipeline and see how automated testing can be used to dramatically improve the quality and swiftness of our software releases. We will also look at some of the most popular and practical tools that we can use to create our CI/CD pipelines. 

The CI/CD Pipeline

In order to release software, we must fulfill a set of business needs. In some cases, these business needs include a quick set of system tests and a suite of User Interface (UI) tests, while other releases may require more involved needs. Regardless of the complexity, these business needs can be conceptualized as a set of steps that are executed in serial and in parallel. In the CI/CD vernacular, each step is called a stage, and the set of ordered stages is called a pipeline. Below is an example pipeline:

The particular stages in the pipeline will vary based on the business needs of the project, but all pipelines will be executed when a trigger (such as a commit) is activated. Once the execution of the pipeline starts, each stage is executed one-by-one; when one stage successfully completes, the next stage is executed. 

When a set of parallel stages is reached, such as the User Acceptance Testing, Capacity Testing, and Staging, stages in the example above, all of the stages are executed at the same time. The pipeline proceeds when all of the parallel stages are successfully completed. For example, execution of the Deploying stage will not start until User Acceptance Testing, Capacity Testing, and Staging successfully complete.

There is no requirement that all stages of a CI/CD pipeline must be automated, and in some cases, it can be difficult to introduce automated test cases into a CI/CD pipeline. For example: 

• Unclear business needs and specifications – In most cases, the difficulty in defining automated tests stems from a lack of clarity about the business needs of our project (which defines our CI/CD pipeline) and the specifications of our software under test. Before we create stages in our CI/CD pipelines, we must understand what we need to test and why we are testing it.
• UI tests – UI tests can be difficult to automate due to the visual and fluctuating nature of UIs. We can overcome this by using a UI test framework, such as Selenium.
• Inconsistent reporting – Many CI/CD pipeline tools include a test summary that displays the number of executed and successful tests completed in a stage. This summary requires a consistent, well-known report to be generated by our automated tests. We can meet this requirement by using an automated testing tool whose reporting format is widely known, such as JUnit (or any of the xUnit frameworks) and Cucumber.

While there may be instances when manual testing is required, the greatest benefit of CI/CD pipelines is achieved when all tests, including UI tests, are automated. 

Automated Testing in the CI/CD Pipeline

 The major advantage to utilizing automated testing in a CI/CD pipeline is that a single commit can be tested against a gauntlet of tests — including unit, integration, system, performance, and acceptance tests — and then be deployed to a production system without having any human interaction. For example, even on a large-scale project, it is possible to have a single engineer make a commit that will automatically result in a feature being deployed to production in a few minutes or hours. 

Conversely, an automated pipeline ensures that failed tests prohibit a feature from being deployed to production. For example, if a developer adds a new feature, and a unit or integration test fails, the execution of the pipeline immediately stops, and the feature is not deployed. The developer is then notified of the test failure and can track down the bug to the commit that triggered the failed execution of the pipeline. 

In addition to the benefits reaped for deployment and release, there are a host of benefits that automated testing brings to the quality of the code itself: 

• Documentation of its intended behavior
• Reduction in the number of regressions
• Decoupling into smaller, more independent components
• Reduction in test execution time
• Involvement of stakeholders in the generation of test specifications (i.e., acceptance tests)

Although it may not be possible for all tests in a CI/CD pipeline to be automated, in order to garner the greatest benefit from our pipelines, we should strive to maximize the number of automated stages and, if possible, completely automate our pipelines. 

Popular CI/CD Tools

There are numerous tools and frameworks that can be used to create automated CI/CD pipelines. The list below is not comprehensive and represents only a small selection of the many excellent tools that can be used to facilitate CI/CD pipelines. Generally, these tools can be divided into two categories: native tools and third-party tools. 

Native Tools

Native tools are CI/CD tools that are integrated directly into our repositories. For these tools, we create a configuration file that resides alongside our source code, and when we make a commit, the repository consumes our configuration file and executes the stages that we define. 

The two most popular native tools available today are: 

1. GitHub Actions – An automated workflow tool that integrates directly with GitHub repositories. New pipelines, called workflows in the GitHub Action lexicon, can be constructed by creating a new Yet Another Markup Language (YAML) workflow file in the .github/workflows/ directory of a GitHub repository. More information about GitHub Actions can be found in the official GitHub Action Documentation and the Getting Started with GitHub Actions Refcard. 
2. GitLab CI/CD – Similar to GitHub Actions, GitLab CI/CD is integrated directly with GitLab repositories and allows developers to create new workflows by creating a .gitlab-ci.yml file in the root of the GitLab repository. More information about GitLab CI/CD can be found in the official GitLab CI/CD documentation. 

When a native tool is available, it is best to use it because it affords the greatest level of integration with a repository and the source code managed by the repository. For example, if our code is stored in a GitHub or GitLab repository, we should use GitHub Actions and GitLab CI/CD, respectively, by default unless we have a pressing need to use a third-party tool. 

Third-Party Tools

Third-party tools are CI/CD tools that reside outside of our repository. For many of these tools, we create a hook in our repository that notifies that third-party tool when a commit has been made. The tools then check out our code from our repository and execute the configured pipeline. The two most popular third-party tools available today are: 

1. Jenkins – An open-source automation server that allows developers to automate the build, test, and deployment of their projects. Jenkins is commonly used as a standalone service, deployed by a development team. Pipelines are either configured directly through the Jenkins UI or through the creation of a Jenkinsfile within a source code repository. More information about Jenkins can be found in the official Jenkins Handbook.  
2. CircleCI – A hosted automation service that integrates with GitHub, GitHub Enterprise, and Bitbucket. The advantage of CircleCI is that a team does not have to deploy and maintain a CircleCI instance but, instead, can access CircleCI through circleci.com. What it gains in convenience, though, it loses in its narrow repository support and lack of flexibility. More information about CircleCI can be found in the official CircleCI documentation.

While using a native tool should be our default option, there are some instances when a third-party tool may be a better choice, such as when: 

• A native tool does not provide the functionality we need
• A third-party tool allows us to utilize more computing power (i.e., a native tool may only allow us to use the resources of a single machine or the resources associated with our repository to execute our pipeline)
• A standalone option is needed so that we can manage the CI/CD pipeline directly (i.e., we wish to manage a CI/CD server within a firewall or company subnet)

Conclusion

Test automation and the introduction of CI/CD into software development have irrevocably changed the way that we create, test, and release our software. While the CI/CD space continues to grow and advance, it is essential that we learn the fundamentals of automated testing in CI/CD and select tools that best enable the time savings and quality improvements that CI/CD offers.  

This is an article from DZone's 2022 DevOps Trend Report.

For more:

Read the Report