Testing Iceberg

Anyone who has done some test automation or has searched for information about the subject will have come across the famous "Testing Pyramid" more than once.

This figure represents the "ideal" way to distribute the automatic tests of a system in its different layers. Where the width, indicates the number of tests that should exist about the different levels.

Therefore, according to this distribution, it would be desirable that the number of unit tests is proportionally greater than the rest of the levels, and that as we go up the layer, these decrease.

The reason behind this recommendation is that unit tests are very fast to execute and "easy" to develop. In contrast, integration and e2e tests (typically through a web interface) are slower, more expensive to maintain, and require some additional infrastructure to run.

This diagram by Martin Fowler shows the relationship between all the variables:

This pattern became popular when tools like Selenium began to be used regularly since in many cases the test automation process was focused on trying to mimic the actions of users on a web page. Following this approach, in many situations, we arrive at an "inverted pyramid" or "ice cream cone" model, which is considered an anti-pattern, because it is not the most effective way to validate the quality of the systems in terms of cost-benefit.

At this point, we must understand that the "testing pyramid" is only a model that represents the "ideal" distribution of automated tests according to the layer on which they act. Therefore, it only refers to the number of them, but it does not indicate how to achieve this in terms of techniques, tools, frameworks, languages, roles, etc.

The pyramid, therefore, represents the final RESULT of a process. That is, the output of our testing strategy and implementation of the same will determine the distribution of tests and the resulting shape. That ultimately, it could be a pyramid, a diamond, a cone, or a square for instance.

I insist on the idea that the pyramid only represents the "ideal" model of automated test implementation, and we must be clear that we will not be able to reach it if the appropriate conditions are not met, that is if the context is not the right one.

So, I mean that the pyramid, what you see, is the tip of an Iceberg, where, without a good foundation, which is not seen, it will be complicated that we can implement an effective automated test suite.

Therefore, to reach a suitable automated test suite (in any shape), we must start from a context that facilitates these tasks. Among other things, conditions must be given such as:

• Both the company and, especially, the team, must favor and be involved with a quality culture and understand its value contribution. The tests mustn't be treated as something optional, second-rate, or something that obstructs development. On the contrary, good automation is only possible if the whole team has it in mind from the start and works on the testability of the systems

• First, the objective and then the testing strategy must be elements that are identified and known by the whole team. We mustn't lose sight of what problems we want to solve with automation (objective) and how we are going to do it (strategy). The effectiveness of our test suite will be determined by these points.

• Just as the development team must have the right knowledge about programming, they must also have it for testing, among knowing the right tools and techniques. Many times this knowledge is simply assumed, but reality indicates that few teams have the right foundation to develop an effective and robust automated test suite. Therefore, the context must favor that the team can be properly trained and acquire this knowledge.

• It must be taken into account that the tasks associated with test automation require time, have a cost, and need the appropriate resources and environments to be executed. These issues should not be overlooked when establishing the automation strategy, as they can mark the success or failure of the same.

In summary, when dealing with a test automation project, it is much more important to work on the right conditions than to focus on reaching a certain "distribution" of tests. 

However, as I said before since the context is "the part you don't see", it will be what is more complicated to modify but also what is more challenging.

Final Notes

At this time, since many of the solutions that are built are based on APIs and their connections between them, it is very possible that the pyramid model is not the most appropriate one. For this type of system, it makes much more sense to focus on integration testing, since that is where the business logic is encoded and where most interactions are established.

This is why other models such as "diamond", "honeycomb" or "testing trophy" are more effective.