3 Keys to a Successful Continuous Testing Implementation

Unless you’ve been hibernating for the past decade or two, I’m sure you have already become well aware of the benefits of continuous testing: reduced cost of development, less waste, improved system reliability, reduced risk upon release, and so on. Of course, you believe putting it into practice in the real world is not as simple as some vendors may have. It’s become clear that having a continuous integration tool configured will not mean that a team successfully achieves continuous testing.

Fortunately, we now have more resources than ever to overcome this organizational challenge, such as the ACT Framework. Not long ago, my team and I also released our Ultimate Guide to Continuous Testing, which is very much in line with the philosophy of the ACT framework. In this post, I will share some key considerations and tips from the guide to successfully implement continuous testing. 

Assess Your Current Team, Technology, and Processes

Any time you are looking to improve the results of software development, it’s necessary to consider three fundamental pillars that are closely linked together: processes, tools, and people. Teams can learn continuously and improve by bettering their processes. Still, the tools they use and the team members themselves must also adapt in order for the new and improved processes to be successful in the long run.

A few of the questions to ask here:

• Do we have a common understanding of our delivery pipeline? Don’t think of the development pipeline only but also about how you deliver value to your business. 
• How much of it is automated? Do we have a CI/CD engine in place?
• Do our current testing tools and processes integrate with the said pipeline?
• Are we missing the expertise required to automate? Should we invest in training? If so, in what? 
• What are our biggest constraints: time, budget, resources? How can we offset any of these?

Know Which Quality Factors Matter Most

For every team, some quality factors will be more important than others, which is why it is not always necessary or smart to focus on continuous testing for every factor of software quality. For example, if you’re working on a system for a very diverse user group, usability testing may take priority over performance testing. For example, a government system used by citizens to file taxes might require a higher degree of usability and security than performance.

According to the ISO 25010 standards for software product quality, there are eight software quality properties: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. For each, you have the corresponding set of tests to assess its level of quality.

Evaluate with your team which is the biggest areas of concern regarding software quality for users and stakeholders. What are the worst-case scenarios in which, if something were to break, it would cause the greatest damage? These are the risks that must be addressed first. With a limited number of weeks, days, or maybe even hours for testing between releases, following a risk-based approach will help to make it clear what is worth prioritizing. 

Map Your Route to Continuous Testing

There are several quality areas to focus on that will help you in your efforts to reach an efficient continuous testing program, encompassing shift-left and shift-right testing.

In the chart below describing three testing maturity levels (basic, efficient, and continuous), you can see how the management of different quality aspects from the source code, environments, test automation, etc., need to be set up in order to pull off continuous testing. For example, before you have CI/CD in place, you must first implement source code versioning.

I encourage you to use this chart as a reference to review where your team stands in the nine different quality areas. Going across the rows from top to bottom, contemplate with your team members if you are handling each given area in a way that is characterized by basic, efficient, or continuous testing.

From there, you can make a plan of action to progress upwards in these maturity levels, achieving continuous testing. They tackle which areas need to be improved first, the most critical ones that underpin the next steps toward continuous testing, and also the properties of software quality that your team has determined to be most critical. 

Planning for Success

As the old saying goes, “failing to plan is planning to fail.” Before going gung-ho with continuous testing, automating everything that is capable of being automated, it’s wise first to have things clear about what success should look like, given your team’s context and business goals. It’s okay if your team makes some mistakes along the way, and it won’t happen overnight that your continuous testing is a well-oiled machine!

Remember first to understand your organization’s goals, people, processes, and tools. Know the most important risks to mitigate with continuous testing and build a concrete, realistic plan of action from there.