8 Test Automation Anti-Patterns (And How to Avoid Them)

No matter how well-versed you are with testing or how advanced your automation technology is, it’s essential to start automation with a strong test design. Without one, teams can encounter a range of problems that compound to create ineffective, incomplete, and hard-to-maintain tests. To ensure quality, cost efficiency, and on-time delivery, it’s important to recognize the signs that your tests aren’t performing optimally. Below are eight “anti-patterns,” or habits that I believe should be avoided if possible, for the best testing experience and end-product.

#1: Sequence Length Is Too Long

Tests are often designed as long sequences of small steps, making it difficult to manage and maintain them. For example, when an application you are testing changes, it is hard to factor these changes into other relevant tests.

Rather than a bottom-up approach, create a high-level test design first. Depending on the method, this can include elements like definitions of test products and their scopes, further detailed down in test objectives and main objectives for the various tests. For example, a test product could be a set of test cases testing the calculation of home loan mortgage premiums. This will allow steps (actions) to be high level, like “enter mortgage,” and hide the navigational details in the implementation of those actions.

#2: Too Much Interaction Tests, Not Enough Business Tests

Testers that focus primarily – and sometimes only – on interaction are at risk of developing shallow tests that miss key business-level problems, such as interacting business processes or application responses to unexpected situations.

A main distinction that testers should make is between “business tests” that focus on business objects, rules and processes, and “interaction tests” that focus on the interaction with the application. For example, in a business test a user might log in, enter some orders and check a financial balance using high-level actions that hide the interaction details. In an interaction test, one might enter a user name and password in a login dialog and check whether a “submit” button is enabled, a test that could occur in any kind of business environment.

#3: Interaction and Business Test Lines Are Blurred

Just as it’s critical to run both tests (as described above), it’s also critical to run them separately, even if tests are for business functionalities. For example, business object life cycles and business rules, calculations, and processes should not be mixed with tests on interaction details such as checking the “submit” button in a login dialog or checking every time that a welcome message is displayed after login. This can result in a convoluted and hard to maintain test mix. For example, if no welcome message is displayed in a new version of the application, all tests checking for it will have to be maintained.

A good, modular test design can provide the direction you need to avoid blurred lines to ensure manageability and maintainability. In a good test design, test modules have a clear scope. They avoid checks that don’t fit into that scope and hide detailed steps to interact with the UI (or API) if they’re not needed to understand the test in relation to its purpose.

#4: Missing Life Cycle Steps of Business Objects

Most applications work on “business objects,” like orders, invoices, products, customers and more. These objects have their life cycles in the application, like create, retrieve, update, and close (known as “CRUD”). However, the test for such life cycles is often scattered, hard to find, and incomplete. This lack of testing can cause gaps in the scope of the tests, in particular, if business objects have variations. In a car rental, we might test for cars and vans, but have less coverage for motorcycles and buses.

Life cycle tests are usually not hard to design. Start by identifying the business objects and for each of them identify their operations. This could include variations, such as canceling or updating an order. Be sure to design life cycle tests as business tests, not interaction tests, as that is their primary function.

#5: One-Dimensional, Poorly-Designed Tests

Time pressure and other factors often result in shallow test cases that don’t challenge the application. This can result in quality issues, like missed situations that the application then doesn’t respond to well. And it can make it harder to maintain the tests, costing more time down the line than what was “gained” by the initial time-pressure.

In particular, it is important to get testing and test automation in sync with the other activities in a Scrum sprint. Both the tests and their automation need a lot of cooperation, and that becomes significantly harder if they are still working after a sprint has ended. If better test design and automation architecture alone are not sufficient to maintain the velocity, consider outsourcing some of the tasks so the QAs in the team can continue to keep up.

Try to think as a true “tester,” somebody who has a talent to “break” things. For example, applying various testing techniques such as decision tables and error guessing can help you identify situations that need to be addressed with the test cases, while techniques like state-transition diagrams and equivalence partitioning can give ideas for the design of the test cases. If you’re looking for more creativity in test case design, also take a look at exploratory approaches and my own Soap Opera Testing.

#6: Lack of Scope

A very common situation is a lack of scope for tests, such as whether a test is meant to test an order entry dialog or a set of financial transactions. The tests are then hard to find and update if there are application changes, and may duplicate work done in other tests. It might also be hard to determine in which test a given condition should be verified.

Designing clear scopes and objectives for tests (especially test modules) will ensure that you can focus your checks and choose actions that best represent the flow of the test. Understand and refer to the scope often and try not to deviate from it.

#7: Duplicate Checks With Tests

Test designers often approach steps with an expected result for each step, as various test management tools encourage you to do. This means separate tests can be doing the same exact check. For example, the earlier mentioned test checked whether a welcome screen is shown after a successful login.

Start with a focused test design effort to avoid this, making sure all test modules have clear and well-differentiated scopes. Then, when developing the tests, resist the temptation to check after every step. Only include checks that fit the scope.

#8: Checks Are Hidden Within Actions

As described above it is good practice to use business-level actions that hide details that do not contribute to the scope of a test case or test module. However, be moderate about hiding checks. People should be able to understand a test by just looking at the test module, including the checks the test is performing to meet the objectives of the test. In my view, checks that are included to verify the successful navigation (like the earlier mentioned check on a welcome message) are probably okay, but I would prefer having good low levels that make sure the interaction (UI or non-UI) is working properly before higher level tests are executed.

Test design anti-patterns are not meant as absolute rules, but more as warning signs. They encourage testers to spend the time and resources to obtain a thoughtful test design and test organization from the beginning. Issues can still creep into the best-laid plans. When noticing a potentially bad habit, take another look at the test design. Reviewing your tests and updating the test modules so each test fits in well can make them more effortless and successful.