Test Automation vs. Automated Testing

When we talk about continuous testing in terms of Continuous Delivery and DevOps, the term "automation" gets thrown around a lot. In a basic sense, we all understand what automation means: The use of some technology to complete a task. However, when we talk about automation in terms of continuous testing, there are some nuances that we need to take into account.

Two Types of Automation in Testing

In the world of testing in general — and continuous testing, in particular — there are two types of automation: automated testing and test automation.

While it might just seem like two different ways to say the same thing, these terms actually have very different meanings.

Automated testing is the act of conducting specific tests via automation (i.e., a set of regression tests) as opposed to conducting them manually, while test automation refers to automating the process of tracking and managing the different tests.

Both automated testing and test automation are important to continuous testing, but it’s the latter that really takes the cake on this one.

Why Test Automation Is Critical to Continuous Testing

To fully understand why test automation is so critical to continuous testing, it’s important to make clear what exactly continuous testing entails and why it came about.

Continuous testing is a relatively new approach to software testing that aims to ensure quality at all times.

In a traditional environment, testing gets completed at the end of a development cycle. However, as more and more companies move toward a DevOps and Continuous Delivery model in which software is constantly in development and must always be deployment-ready, leaving testing until the end no longer works. That’s where continuous testing comes in to ensure quality at every stage of development.

So, with continuous testing, rather than testing happening in a big bang at the end of a cycle, it happens in small pieces at all times as soon as the need arises.

While ensuring quality at all times is of utmost importance to this model, it’s not all that counts. The speed at which all of the development and the testing occurs also matters quite a lot. That’s because if something in the pipeline stalls or breaks down, it holds up everything else and slows down the release of new developments. Given that the need to deliver new releases faster and on a more regular basis paved the way for this Continuous Delivery and testing model, that roadblock defeats the purpose of taking this approach.

This “how” and “why” make organization, consistency, and speed imperative to supporting a continuous testing model, and that’s where test automation can help. Managing all of the testing needs in a continuous testing environment is a massive undertaking. It requires tremendous communication efforts to keep track of which environments have deployed new code, when each piece needs testing, and how those requirements integrate back into the moving process of continuously delivering software.

Test automation eases this burden by automating the tracking and managing of all those testing needs, including how much of the system different tests cover and what other types of testing might be required to cover all the moving parts. In doing so, test automation goes a long way toward helping ensure that teams maintain a high standard of quality at all points along the pipeline. Additionally, it allows testers to focus more time and effort on creating effective test cases to ensure the quality of the software since they’re no longer bogged down in managing all the minutia of testing needs.

Making Test Automation a Reality

In theory, the concept of test automation is a perfect fit for testers operating in a continuous testing environment — but what happens when reality strikes?

In a typical real world scenario, when testers need to schedule and verify test cases, they:

• Communicate with the Product Owner to gather product requirements and distil the essence of the problem for which the product owner is trying to solve.

• Break the product requirements down into user stories and then incremental units of work to create the functioning software. This often requires working with the team of developers, analysts, and operators.

• Write a combination of test cases (automated, exploratory, regression, etc.) to fulfill the contract of those requirements. 

• Track the progress of each step to completion, running the appropriate test cases for each phase:

  • Developer branches that compose components of the feature.

  • Deployment artifacts that compose parts of the systems and services needed to support the feature.

  • Regressions so that changes or additions to the component don’t affect other aspects of the working system.

  • Functional verifications to ensure the product does what the product owner intended it to do and solves the problem correctly.

With this type of scenario in mind, what testers really need to make test automation a reality is a solution that can help automate the process of creating test cases for specific work items and scheduling test runs to execute those test cases.

Ideally, this solution should allow for test automation by:

• Flagging a list of work itemsfor which test cases need to be created to bring new needs to testers’ attention automatically.

• Integrating with the ALM so that when a particular type of task gets created in the ALM, a copy of that task also gets created in the test automation tool and presented to the user.

• Allowing users to separate and categorize work items by logical containers (such as by feature, component, or Sprint) to make it easy for users to ensure proper coverage for each pipeline.

The Move to Embrace Continuous Testing

As DevOps and continuous delivery models become the norm, so too will continuous testing. In order to make continuous testing efforts successful, testers need to start thinking now about what it will take to manage the change that comes with injecting testing throughout the entire development pipeline.

Clearly, test automation will be a critical weapon in the continuous tester’s arsenal given its ability to help keep track of all of the different tests that need to take place at various points in the pipeline.