The Four Steps of Regression Testing

This article provides a structured approach to create and update a regression test suite. What kinds of tests should be in a regression test suite? Which regression tests should be run, how do you respond to regression tests that fail, and how does a regression test suite evolve? These questions and other considerations are explored in a step-by-step manner. I will first explore the basic dynamics and considerations of regression testing. Then, I will provide a set of steps that can help bring long-term software stability from regression testing.

Nuts and Bolts of Regression Testing

Let's assume that we made a couple of changes in our software code, any kind of changes. How can we be confident that these changes will not negatively affect our code overall? One way to achieve confidence is to perform thorough regression testing. Write and execute tests to check and explore how our code behaves after our changes. So, the more tests we write and execute, the more confident we will be. Yes, but there are practical costs to be considered as well — the time, effort, and money required to write, execute, and maintain a regression test suite.

Including every test possible results in a regression test suite that is too large to manage — and it’s challenging to run, as often as changes are made to the software. If the regression tests do not finish in a timely way, the development process is disrupted. It is well worth throwing money at this problem in terms of additional computational resources and/or new hires to execute tests. At some point, however, the added value of adding a test may not be worth the added expenditure of the resources needed to execute it.

On the other hand, a test suite that is too small will not cover the functionality of the software sufficiently well, and too many bugs will be passed on to the users.

Adding a small number of tests to a regression test suite is usually simple. Even if the marginal cost of each additional test is quite small, cumulatively, the test suite can become unwieldy. Removing tests from a regression test suite may create problems. What if a customer reports a bug that one of the removed tests would have found?

Test Case Selection Techniques

Selecting the right test cases involves identifying directly and indirectly affected test cases. We should know at least which features are used the most from our customers, which tests cover important functionality and which tests fail often. Other selection techniques include linear equations, symbolic execution, path analysis, data flow analysis, program dependence graphs, system dependence graphs, modification analysis, cluster identification, slicing, graph walks, and modified entity analysis. When it comes to choosing a selection technique, it is useful to think in terms of the following criteria:

Inclusiveness

Inclusiveness refers to the extent to which a regression test selection technique includes tests that are likely to expose faults introduced by recent changes to the software. A technique is considered more inclusive if it effectively identifies and selects tests that cover modified or affected parts of the code. Inclusiveness is vital to ensure that the selected tests provide thorough coverage of the changes made since the last test cycle. Unsafe techniques have inclusiveness of less than 100%.

Precision

Precision measures the ability of a regression test selection technique to exclude tests that are unnecessary for the current testing objectives. A precise technique should minimize the inclusion of tests that do not contribute to detecting faults related to recent modifications. This criterion aims to prevent over-testing, which can lead to longer test execution times and resource inefficiency.

Efficiency

Efficiency evaluates the computational and time resources required to perform regression testing using a particular technique. An efficient technique should be able to quickly identify and select the relevant subset of tests while minimizing the overall testing time. This is especially crucial for large software systems with extensive test suites where faster testing cycles are desirable to support agile development practices.

Generality

Generality assesses the applicability of a regression test selection technique across various software testing scenarios and domains. A more general technique can be used in a wide range of practical situations without significant customization. It should not be overly specialized for a specific type of software or testing context, making it adaptable to different development projects.

In what follows, the four steps of regression testing are explored. We start by identifying the modified code under test. The tests that need to be executed are identified, followed by a step for balancing the test suite’s size. It’s all about our test execution results. How extensive did we test, how fast did our tests run, and how confident are we that our test results provide a true picture of the system under test? Software stability in the long term can follow as we get better at each of the four steps.

Step 1: Identify Modified Code

Determine the specific parts of the software that have been modified since the last regression test cycle. This can be achieved through version control systems and change tracking mechanisms. This step is the foundation for the subsequent regression testing steps.

Version Control and Change Tracking

To identify modified code, we can use version control systems and change tracking mechanisms. Version control systems like Git, SVN, or Mercurial keep a historical record of changes made to the software's source code. Developers use these systems to commit changes along with descriptive commit messages. Change tracking mechanisms can also include issue tracking systems like JIRA or bug databases.

Analyze Commit History

In a version control system, we can examine the commit history to see what changes have been made to the codebase. Each commit typically includes information about which files were modified, what lines of code were added, deleted, or modified, and a description of the changes made. By analyzing this commit history, we can pinpoint the specific code that has been altered.

Identify Modified Files and Code Sections

Based on the commit history, we can identify the modified files and the sections of code within those files that have undergone changes. This may include functions, classes, methods, or even individual lines of code. It's essential to be as granular as possible in identifying the modified code.

Document Changes

It's helpful to document the nature of the changes. Are these modifications, bug fixes, new features, enhancements, or other types of changes? Understanding the nature of the changes can guide our regression testing strategy.

Collaboration With the Development Team

Collaboration between the testing and development teams is crucial during this step. Testers should communicate with developers to get a clear understanding of the changes and their impact on the software's functionality.

Traceability 

Establish traceability between the identified modified code and the corresponding requirements or user stories. This helps ensure that the modifications align with the intended functionality and that our regression tests adequately cover these changes.

By the end of Step 1, we should have a comprehensive list of the code that has been modified, along with details about the changes. This information serves as the basis for selecting the relevant tests in Step 2, ensuring that we focus our regression testing efforts on the areas of the software that are most likely to be affected by recent modifications. This targeted approach is the backbone of our regression testing structure. It is essential for efficient and effective regression testing.

Step 2: Select Relevant Tests

Once we have identified the modified code, the next step is to select the relevant tests to include in our regression test suite. This step is critical to ensure that we thoroughly test the changes made to the software.

Coverage Criteria 

The first part of this step involves evaluating coverage criteria to determine which types of tests should be included in our regression test suite. Coverage criteria help us define the scope of our testing efforts. Two common coverage criteria are:

Node Coverage (or Method Call Coverage)

Node coverage focuses on identifying methods or functions that are never invoked in the modified code. This criterion is essential for ensuring that all parts of our codebase are exercised, which can help uncover dead code or unused functionality.

Structural Coverage

Structural coverage goes a step further by analyzing which code paths are affected by the modifications. This criterion considers if methods are called and the specific execution paths within those methods. Techniques like statement coverage, branch coverage, and path coverage fall under this category. It helps ensure that every method is invoked and that different execution branches and scenarios are tested.

Selection of Tests

For each modification identified in Step 1, we need to select tests that directly or indirectly exercise the modified code.

Directly Affected Tests

Identify the tests that directly cover the modified code. These are the tests that specifically target the functions or methods that have been changed. Running these tests helps ensure that the modifications are working as intended and haven't introduced new bugs.

Indirectly Affected Tests

Some changes may have ripple effects on other parts of the software. Indirectly affected tests are those that may not directly exercise the modified code but interact with it in some way. For example, if a change in one module affects the output of another module, tests for the latter module should also be considered.

Test Adequacy

It's crucial to assess the adequacy of our selected tests. Ask yourself if these tests provide sufficient coverage of the modified code. Consider the complexity of the changes and the potential impact on the software's behavior. In some cases, we may need to create new tests specifically tailored to the changes.

Documentation

Keep thorough documentation of which tests are selected for each modification. This documentation ensures transparency and allows for easy tracking of test coverage for different code changes.

By the end of Step 2, we should have a well-defined regression test suite that includes the necessary tests to validate the modified code effectively. This focused approach to test selection ensures that our testing efforts are comprehensive, helping us catch regressions and defects early in the development cycle.

Step 3: Balance Test Suite Size

While it's essential to select tests that adequately cover the modified code, it's equally important to avoid including every possible test in the regression test suite. Managing a massive test suite can become time-consuming and resource-intensive. The third step in regression testing can focus on managing the size of our regression test suite effectively. It's essential to strike a balance between thorough testing and practicality.

Avoid Including Every Possible Test 

Including every conceivable test in our regression test suite is generally not feasible. As our software evolves, the number of tests can grow exponentially, making it impractical to execute them all within a reasonable timeframe. Running an exhaustive set of tests could significantly slow down the testing process, making it difficult to keep up with the pace of development. The optimal size of our regression test suite should be determined. This size can be based on factors like resources availability, time constraints, risks, development process and prioritization.

Available Resources 

Consider the hardware, software, and team members available for testing. Limited resources may restrict the size of our test suite.

Time Constraints

Be aware of project deadlines and release schedules. We should aim to complete regression testing within the available time while ensuring adequate coverage.

Risk Assessment

Evaluate the criticality of the modified code and the potential impact of defects. Highly critical code changes may require more extensive testing, while less critical changes can be covered with a smaller test suite.

Development Process Considerations

The choice of test suite size should align with the development process.

In Agile methodologies, where changes are frequent, regression tests are typically executed more often (e.g., after each sprint or iteration). Therefore, the test suite size for each regression cycle may be smaller to keep testing agile and responsive to changes.

In more traditional development processes, where changes are less frequent, and releases occur less often, regression tests may be conducted less frequently. In such cases, we might have larger test suites that cover a broader range of functionality.

Prioritization

Consider prioritizing tests based on factors such as critical business functionality, frequently used features, or areas with a history of defects. This can help ensure that the most critical parts of the software are thoroughly tested, even if we have limitations on the test suite size.

Documentation

Our decisions regarding test suite size should be documented. This documentation will serve as a guideline for our testing strategy and provide transparency to all stakeholders.

Balancing the size of our regression test suite is essential for efficient testing. It allows us to focus our testing efforts where they matter most while ensuring that we can complete regression testing within our project's constraints. By tailoring our test suite size to our specific context, we can strike the right balance between thoroughness and practicality in regression testing.

Step 4: Execute Tests and Handle Results

With a balanced regression test suite at hand, we can now execute it and evaluate our test results. 

Tests that Fail 

If one or more regression tests fail, investigate whether the failure is due to a fault in the software modification or an issue within the regression test itself. Did the test fail for the right reasons or for the wrong reasons? The right reason for a test to fail is that it found a bug. One wrong reason is that there is no bug, and the test fails because of how it is written or executed. Additional work is required in either case. 

Tests that Pass

If no regression tests fail, then we should be able to answer the following question confidently. Do our tests pass for the right reasons or for the wrong reasons? One right reason for this to happen is that tests exercise parts of the code that function properly. However, a test could pass testing because it may actually test nothing. It is an old test that was not maintained properly and it currently doesn't test what it was intended to test. It happens to pass testing accidentally.

Test Automation

Regression testing is an area that test automation gives its most benefit. Ideally, we would like to ensure that all regression tests are automated. If this was always feasible and practical, then regression testing execution would take no manual effort and would be repeatable at any time. Unfortunately, there may be automated regression tests that fail for unknown reasons, while some of them fail regularly and others irregularly. Some tests will execute fast, others will be slow, and others may execute fast in some runs and slow in other runs. Problems like these may be solved, but if we don’t solve them, as the number of regression tests increases, they can only get worse.

Test automation is most valuable when it is used in a continuous integration/continuous delivery (CI/CD) pipeline. If our project follows a CI/CD pipeline, we may have the opportunity to automate and streamline regression testing. Smaller, focused test suites can be run more frequently as part of the CI/CD process, catching regressions early in the development cycle. 

Keep in mind that automation testing has the same goal as manual testing: To give us a clear picture of how the system under test behaves as expected. We should be confident from our manual testing results, and we should also be confident from our automation testing results. When an automated regression test suite finishes execution, we should be confident that the test results depict the true picture of the system under test. The more confident we are, the less time we will spend debugging the results of our automated tests and identifying real bugs or fixing tests that are useless.

Wrapping Up

To adapt to software changes, we must first recognize that a regression test suite suitable for one version of the software may not suffice for subsequent versions. The first step is to identify what changed in our code. We must account for different types of code changes like creating new features, improving existing features, fixing bugs and glitches, refactorings, and performance improvements. Even when functionality remains unchanged, we must reassess the regression test suite's adequacy, especially if there's code restructuring.

The second step is to identify what tests to include in our regression test suite. We can select all relevant tests that cover the code changes identified from step 1. Regression test suites should be created or modified as needed to incorporate new tests that cover altered functionality or code paths. Bear in mind that there are no perfect metrics. For example, node coverage can be a useful metric, but it is possible to have a high node coverage and still have gaps in the test suite. For example, a test suite may cover all the nodes in a program, but it may not test important input values or execution paths. 

The third step is to balance our test suite’s size. This is very important as the number of tests grows and/or the time it takes for test execution to finish becomes an obstacle. Obsolete tests that are no longer relevant due to software changes can be removed.

Once test execution has finished, it is important to scrutinize our test results. Once we are confident that our test results are trustworthy, we can share our findings with the appropriate stakeholders.