Organizing and Improving Development Teams: The Testing Matrix

Introduction

The testing matrix is a framework for improvement. It can help plan our testing before we start developing. During development, it can help as a single source of truth for what has been done daily. When development and testing are finished, it may be the big picture of our testing choices and decisions. Did we make the right decisions? Sometimes the answer to this question can only come in the future. But in the future, we must make sure that we can recall our decisions in some detail, otherwise, no lessons will be learned. In retrospect, it may be beneficial if the big picture for our testing activities can be found by looking at a single matrix.

The Matrix

On the vertical axis, we can display the levels that we've tested or that we plan to test. Did we do performance testing at a unit level? What about integration testing and regression tests?

On the horizontal axis, we may use the testing types. Did we do performance testing at a system level for feature A? During development we decided that there was no reason to do so. Did we perform regression testing before releasing feature B? At what level? Are we planning for any security tests? What about functional testing?

The Level of Details for the Matrix

I've seen teams using colored ticks and crosses. They decided that they mainly needed qualitative information. Quantitative information was depicted on a t-shirt size basis. For example, a green tick on the functional unit testing box would indicate that enough functional unit tests have been performed. A grey tick would indicate "some tests" and a black tick "a few tests". A cross would indicate no tests. If no integration testing was done at all then the corresponding row was deleted. If no security testing was done then the corresponding column was deleted.

Other teams have used this matrix for test planning. Before development started they broke their tasks and indicated on the matrix who would test what. On a daily basis, the matrix helped them align what tests have been done and what remained. The matrix raised discussions when facing problems or when clarifications were necessary.

Another team decided that they needed a clear picture of what they tested. They wanted to see it visually on the board by the time a new feature was ready to be released. All they needed for each and every mission was two things: The Jira board with their tasks and the testing matrix. These two combined, have helped for clarity and alignment. If they decided that no performance testing on the UI was applicable, they deleted the corresponding box. If it was applicable but for some reason, it was skipped then that reason was briefly described on the box.

Whole-Team Testing

One of the main assumptions in this article is that development teams test. There could be testers within the team or software developers in a test or just software developers. Irrespective of what software engineering roles make up the team, it is assumed that testing is a whole-team activity. 

Developers and testers usually perform different types of testing at different levels. The testing matrix could help to avoid misunderstandings, blaming, and frustration. There could also be pair-testing performed between different roles. Functional testing at a system level could be a good candidate for pair-testing as well as bug triaging and exploratory testing. A testing matrix could be the scope map for such activities.

Mix and Match as You Like

The types and levels depicted on the matrix above are indicative. They are not carved in stone. Feel free to mix and match as you may find appropriate. Perhaps you need the unit, API, and UI levels or unit, integration, and UI levels. Perhaps you need positive and negative testing on the horizontal axis. This article is not about what types of testing are important at what level. Consequently, that choice is left to the reader. I am just elaborating on the ways that you can mix and match a testing matrix to your abilities and needs.

Retrospective Discussions

Above all, during our retrospectives, the testing matrix can facilitate discussions based on testing. What kind of testing are we not confident with and at what level? Are there any testing results that we do not fully trust? Are there any testing results that do not make sense? Was there any testing effort that took much more time than expected? Can we understand why this happened? Are all our decisions about what we tested, when, and how we tested, clear and legitimate? Questions like these can help the team focus on testing and improvement.

But all that can be done with other tools!

Maybe so, but the matrix is free of charge. You can just draw it on a piece of paper and stick it on your board. You need no licenses and no money and you've got all that information on a single matrix. No need to jump from one tool to another.

Evolution

As we evolve as a group or team, the way that we develop and test will evolve too. The testing matrix is also expected to evolve. When we use it, how we use it and at what level of detail, will all probably change. That’s a good thing provided that we change our matrix only to get more value.

Improvements From the Matrix

1. Awareness and alignment on decisions. When all the testing options are there, the team can make informed decisions. Do they have a week to release the new feature? In that time frame, what testing activities should be done? What are the most important testing activities? Is one week enough? Decisions on such questions could be available for anyone on the matrix.
2. A simple way of viewing the big picture. The team doesn't have to switch between viewing code commits or Jira boards each time they want to see the decisions taken. They must be looking for fine-grained information to do that. All the basics of the testing strategy and all the high-level details can be captured in the matrix. Looking for more fine-grained information on testing activities? The matrix could be a good starting point to navigate to the right tasks or code commits.
3. Discussions and improvements. A core value of the matrix is to facilitate discussions about testing. Implying that we test is different from explicitly specifying our testing activities. So, how did our last milestone go? What testing can we improve? In what new ways could we test to get more value? Are our customers happy? How many bugs did we catch and solve before releasing them to the customer?

Wrapping Up

The testing matrix can be helpful to organize and improve our testing efforts. I’ve seen startups as well as established companies making value from it. If whole-team testing is an approach that your team follows then a testing matrix tailored to your needs could be beneficial.