Going Lean on Your Testing Approach

When teams are looking to transform, optimize, or cut costs in testing, where do they first look? More often than not, they follow the advice given by outside consultancies: “You haven’t got enough test automation,” or “Your peers have achieved xx% tests executed through automation.”

There are a number of problems with this advice:

1. It focuses solely on test execution.
2. It usually comes with zero context as to what that figure even means or how testing approaches at one organization are comparable to another.
3. It assumes that the way you do testing today is effective and simply needs automation. Sadly, this is rarely the case, and you likely have a ton of waste, duplication, and inefficiency in there.

Don't Automate Broken Approaches

So, what are some of the telltale signs that your testing efforts today aren’t as effective and efficient as they could be? Let’s explore some of the warning signs that the way you test today could do with some changes and a fresh outlook on your test approach:

• You have a large team of testers doing mainly e2e tests (even if they call it something else).
• You have a test pack largely made up of UI-based e2e tests, which always grows after each release — you suspect there’s lots of duplicated effort in there.
• Testing is delayed because you are waiting for e2e environments to be built or waiting for your slot in the few e2e environments that persist in your organization.
• Test coverage relies on traceability back to requirements that you know are incomplete or out of date.
• Test design is largely an activity that involves testers creating test cases without input from the wider development team. Alternatively, you don’t design tests at all; you just write test cases/automation.
• Despite trying many different tools, you are struggling to increase the rate of tests executed through automation, as the tests are just too flaky!
• A large % of your defects are environment/config-related, causing large delays in test phases.
• You are finding fundamental design issues in the late e2e test phases.

These antipatterns lead to a poor return on your testing efforts. Testing becomes too costly, too time-consuming, and leads to questionable outcomes in terms of quality and your ability to implement change.

You probably recognize some of these challenges. Yet, if you are suffering from one of them, you likely have them all but haven’t looked from the right angle. This is because all of these challenges stem from one common cause: Not setting testing up properly. They are all signs that you don’t have solid foundations in your test approach and strategy.

First, Fix Your Test Approach and Strategy

If you don’t consciously have a test approach today, then the chances are that you have unwittingly implemented the “upside-down test pyramid” in some form:

With this, all of your testing is likely more expensive, slower, and less reliable than it needs to be. Yet, this is unfortunately common in test teams across organizations today, regardless of their delivery approach or technology being tested.

Lean Approaches for Moving Beyond the Upside-Down Test Pyramid

The upside-down test pyramid will slow you down and cost you more the longer you persist with it.

So, what can you do? Take the first step in consciously implementing a test approach that matches the IT vision, technology, and change methodology of your organization. And be lean about it.

A Mismatch Between “Agile” and Test?

Many organizations have been adopting smaller, iterative ways of working. Common goals include boosting agility and speed to market while lowering cost and improving feedback (quality).

While these organizations have introduced plenty of new technologies and design patterns, testing has often remained large, cumbersome, and poorly suited to automation. As dev teams work so hard to keep design and build small, they are frustrated as the test doesn’t (or won't) follow those same principles. This is fundamentally and evidently wrong.

Uncoupling Monolithic Architecture

If your team is trying to adopt Agile, but your wider team struggles to carve out small iterations of delivery, you have more than a test problem. You have a “big ball of mud.” This will either be a tightly coupled architecture or a distributed monolith if you are in the cloud. You need to start decoupling.

Decoupling means establishing boundaries within your architecture. A good place to start is by understanding your interface-building models by analyzing how interfaces work today. BDD-style collaboration and monitoring production are both examples of methods of understanding how an interface works. Then, you should validate that these interfaces work in the way they were intended.

Once you know that interfaces are working as intended, you have an opportunity to rearchitect your big ball of mud. More relevant to this article, you also have an opportunity to change your inefficient test approach. You can start targeting the newly understood, observable, and testable interfaces through API testing. And then, you are starting to turn that testing pyramid the right way up.

Isolating Blast Radiuses for Testing

Yet, a good test approach involves more than doing different types of tests. Just as we want small, interactive delivery, we also want fast feedback. For that, we need to have a full stack test approach:

Full stack testing should not test everything every time. The architecture should act like a Lego kit for constructing test approaches and assembling “blast radiuses” that we need to test. Testing then becomes appropriately lean and targeted at the things we are changing.

In other words, we want to follow the architectural best practices that are used around loose coupling and isolation so that we don’t end up unnecessarily testing interfaces that aren’t changing and pose zero risk. Targeted tests should live and run alongside the components they test. They then provide immediate and specifically directed feedback on what’s changed and what might have gone wrong. If those best practices don’t exist, we can establish them through our test approach.

The diagram and table below indicate the broadened scope that testing should consider during a more full-stack approach. You could easily go further than this to make testing more agile. Have a think for yourself about where you might add further separation:

From Testing Phases To Continuous Testing

Being able to isolate creases in your system architecture and use different types of testing, in turn, replaces the traditional phases of testing from waterfalled projects of old and those that carried into newly formed agile teams.

An effective test approach in this context no longer talks about System Tests or System Integration Tests as phases. Instead, you have types of tests that you execute continually as part of your pipeline, and these tests are harmonized and targeted to the area of the system you are changing, including testing platforms.

Reaching the Whole System

Testing has been traditionally focused on functionality, but being able to isolate change involves getting closer to the system stack where the change is happening. This is not just API testing; it involves testing things like the platform and moving toward compliance as code.

Testing continuously at the platform level will be especially important as organizations transition to the cloud, risking challenges in test environment builds and production outages due to manual and ambiguous config changes. We’ve all been there, yet many test approaches still don’t include such test types.

The problem is that a fully automated test pack of your user interface is going to fail as a non-starter if the environment you run it in is incorrectly configured. Check out the bugs you raised in your last release: Are environment problems ranking as a top cause of defects? Are you looking for these problems in your testing? Or are you finding these things far too late in your testing efforts, leading to substantial delays as your teams must spend time triaging the problems?

Plan a Lean Test Approach, Then Automate

If this sounds familiar, your testing isn’t targeted sufficiently. Taking a lean approach to your test approach will do more than just improve the effectiveness and efficiency of your testing; it will make your whole development effort more effective. It will also create a change in mindset to help your team collaborate and strive for continual improvement.

Fast feedback is the key thing your test approach must provide within newer development practices. Automating test execution alone misses the mark of providing this critical capability. Before you automate, take a step back to revise your test approach. Go slower to go faster.