Defining Test Boundaries — An Example

I think that everybody agrees that testing is required in order to build a quality product. But there’s also a lot of confusion about the boundaries of each test type. What’s the scope of a unit test? What’s the difference between an integration test, an integrated test and a contract test? If you ask three developers about test boundaries, you’ll most likely get three different answers. For example, I still talk to people who consider that a unit test should test a single class/method.

What’s clear is that most teams don’t have a consensus on what’s the scope of the different types of automated tests and the differences between them. Getting to a universal consensus might be hard, but getting to a consensus inside the team should be easy enough. In this blog post, we’ll see an example of how to do that.

Start With The System’s Architecture

I’m a fan of lightweight documentation. So I think that the best way to define the different test types is to start with the high-level architecture of your system. Here is an example:

On the front end, we have a single page application. It’s composed of different components. (As a note, I’m more of a back end developer, that’s why I only mention the front end in this post, without going into too many details.)

If we zoom into the back end part, it looks like this:

This example uses the Clean Architecture style with three layers:

• The Domain layer contains the domain model.
• The Application layer abstracts the low-level details of the domain model behind more coarse-grained application use cases.
• The Infrastructure layer contains the low-level details of the infrastructure. Here you’ll find the presentation layer (e.g. Controllers), repository implementations, gateways to third parties, queues, access to the file system, etc.

I find it useful to put example of classes from the actual system, just so it’s easier for the team members to know what we’re talking about.

Defining Test Types

With a visualization of our architecture at hand, we can start defining test boundaries and scope. This should be as easy as drawing boundaries around different elements from our architecture.

Unit Tests

I like Ian Cooper's definition of a unit test – a test that doesn’t cross a port. For our example, if it’s not touching the network, the file system, etc., then it’s a unit test:

This diagram clearly shows some example of unit tests:

• It can test a single class
• It can test a group of classes
• It can span the application and the domain layer

It also shows that unit tests don’t cross a port and don’t touch external dependencies (e.g. databases, file systems).

Component Tests

A component test exercises the component through its public interface. In our case this means you might send a JSON request:

Component tests still run in process. As you can see, we’re not testing external systems. For example, we’ll use a fake for the Product Catalog, instead of the SQL implementation.

Integration Tests

Integration tests check the integration between two components. They are very focused and their scope is very narrow. They will test against the real dependencies. In our case, this means testing the real database and a sandbox version of the Payment Provider.

HTTP API Tests

At the next level, we have HTTP API Tests. This is what Martin Fowler calls Subcutaneous tests – tests that run at the layer immediately below the UI. These tests need the service to be up and running, because they test the external components (e.g. databases and third-party services).

End-to-End

Of course, end-to-end tests exercise the system through the UI. This can be done using UI automation tools like Cypress or Selenium WebDriver.

Conclusion

I find it useful to get a consensus inside the team on the scope and purpose of different types of tests. You might not agree with the definitions from this article and that’s OK. What’s important is to speak the same language inside the team. And this should be easy to do. Get the team together, draw the architecture of your system and then start defining the scope of each test type. After getting to a common understanding, save the output of the discussion in your team’s wiki. It might also be useful to link to a couple of examples for each test type. This can help while on-boarding new team members.

If you want to see more example of testing strategies, have a look at these resources:

• "Testing Strategies in a Microservice Architecture" by Toby Clemson
• "Testing Strategies in Interaction-Driven Design" by Sandro Mancuso (video here). (As a note, I’ve been on his Crafting Code and Crafted Design courses and they’re really good. You can read my notes here.)
• A simple test taxonomy from J. B. Rainsberger

Also, if you’re looking for ideas of how to improve your tests, check the Fifty Quick Ideas to Improve Your Tests book – it’s good.

If you know other good examples of test taxonomies for different types of system, please leave a comment.