How to Make Testing Progress Visible

Testing efforts can easily go under the radar or be hard to grasp. It’s important to remember that software testing is not only about running tests: it involves multiple crucial facets that contribute to creating better and better software. In this article, we share with you all what we do in order to help the team and the different stakeholders to be informed of the testing progress. 

Testing is aimed to provide information about the quality of our system and the risks that might affect the user experience and our business so that better decisions can be taken. All this in order to mitigate those risks and improve the quality and the experience of our users.

We are a quality partner. We know that effective communication is essential to optimize our work and achieve our goals. That’s why we focus on building relationships with our clients, through transparent processes that make testing progress visible on a daily basis. Find out how we do it in this article!

Making Goals and Setting Expectations

Testing must be aligned with the business goals. This is a common pitfall that many testers and testing providers make. The challenge with testing is not only how to test, how to design test cases, or how to test a particular feature. One of the most challenging things in software testing is deciding what not to test.

We must understand the business goals and risks in order to be thoughtful and mindful of prioritizing. With that context in mind, we design the best testing strategy aligned to the business needs, and this will set the expectation with the team for the progress they will want to see from us.

The creation of quality software is crucial for society’s development and the improvement of its quality of life. We live in a globalized and almost wholly interconnected world, in which permanent digital transformation is a fact and good quality technology makes a real difference in people’s daily lives.

Our greatest impact lies in what the technology we create enables, and that this technology is of good quality,” recently said my friend and Abstracta’s CEO Matias Reina.

“We empower technologies that are enabling software to be made faster and with quality, we help create more accessible technology, and more people have access to communications infrastructures. When we analyze what we do in each project and with each client, we don’t just think about the products themselves but about what they make possible, how their technology extends and improves the quality of life of many people,” he continued.

All in all, as testers, the first step to give visibility on our work and progress is to understand and align our test strategy to the one from the business. 

Communication Strategy

Once we are clear with the goals, we start with the testing activities. As the rest of the team members are typically busy with their different responsibilities, the challenge on the day-to-day is to keep everyone informed. That doesn’t mean that we must tell everyone about every little detail, about every test, or every result we get after each test we do. Different people have different needs for different levels of detail, which is crucial for an effective communication strategy. 

We are convinced that communication is essential for a smooth, effective workflow. So, we pay great attention to communication with our customers and seek to deeply understand their needs.

Our experience of almost 15 years in testing reconfirms every day that testing isn’t only about designing tests! It’s giving our best so that we can better the experience of the user. And, of course, it’s also about enjoying the path together collaborating with the rest of the team.

We have different levels of communication pre-established to keep stakeholders on the same page. Let’s explore some of them. 

Day-to-Day Communication

The testers are embedded as much as our customers allow us to be. The most common scenario is that the testers become part of the development team, so they participate in all the ceremonies, meetings, slack channels, etc. That way there’s almost no difference between the communication with testers (in this case, Abstracta’s team members), and other team members, whether they are from the client’s team, other providers, etc.

In order to make communication more effective and reduce the chance of having blind spots, all our testers are guided and mentored by technical experts that support them, so there’s always someone with more experience reviewing their work, helping them brainstorm, design tests, plan, communicate results, etc.

There’s one extra layer of communication which is the customer success team, that helps with the business perspective, and they are a good escalation point if anything needs to be adjusted, to scale up the team, etc. At this level of communication, we try to gather the information that might impact our testing strategy, such as business decisions, changes in the global strategy, etc. This way we can align and strategize with the technical team, making the required adjustments, and making sure that our progress is in the right direction. 

There are different things to share at each level, with different frequencies, and with different people. Just to give you some ideas:

• On a daily basis, you could discuss the weekly goals or the sprint goals, and how you are focusing, prioritizing, and progressing, so that you and the team can reprioritize if needed.
• On a monthly basis you could discuss the test strategy, how the different risks are covered with your current testing strategy, or how the different quality factors are being prioritized (you could be delaying your performance tests to face them after you finish solving your security issues.)
• On a quarterly basis, you could do a recap of what you have been doing and analyze how your work is aligned with the business strategy, if your team is getting better or how you are going to step up the game.

Of course, the frequency and content will be different for each team according to their context. It is important to point out that these are just examples; there is no such thing as one size fits all. We make everything very specific to what is most useful for each client.

What About Metrics?

A good way to give visibility on our testing is by managing and sharing key metrics, intelligently selected to be useful in our goal of aligning the team. Testing always looks to provide more information to have less uncertainty and better control over risk, but that information must be analyzed carefully. 

We typically insist on this last idea because metrics can promote unwanted behaviors, so we have to be very careful when choosing them. For instance, if we merely measure the number of bugs, then we could be motivating our team to focus on the areas where they can find more bugs easily, instead of focusing on the most important parts of the system. We don’t want to focus either on the number of test cases, because it doesn’t tell you anything about the effectiveness of your testing. 

Our focus is always on the metrics that are important for the business, something that if improved, would lead to a real improvement for the business. It’s also important to note that one metric only tells part of the story and what’s fundamentally important is the team’s commitment to quality and not only metrics.

We are usually the ones that push to provide metrics. For us, it is very important that everyone understands the impact of the activity of testing. This way, we can help improve the quality, velocity, agility, performance, or whatever reason motivated them to hire us. 

In order to have useful metrics, it’s important to be clear about the goals that we are pursuing, and then, select the metrics that will help us to see if we are getting closer to success or not to that goal. 

Let’s see a couple of examples:

• If we want to improve the quality of our product, let’s focus on the bugs we find and solve before releasing the software, not only the plain number (to avoid the bad behavior we mentioned before) but along with their severity. We should also measure the bugs or complaints reported by users, and measure user satisfaction with our products.
• If we are automating test cases, in order to have an efficient way to support the regression testing efforts, it does make sense to measure progress with the number of test cases. We have a goal of automating all the smoke tests, or all the regression suites, and that number will help us to measure progress over time, if we are delivering it according to the plan or not. 
• If we talk about other quality factors such as performance, it’s really important to have an eye on the response times of the most important features, as well as on the required resources to be able to respond to the current load at a proper time. This way we will ensure quality and control costs. 

In sum, testing always looks to provide more information to have less uncertainty and better control over risk. It’s important to choose metrics that align with business goals and not to rely solely on metrics, as a team’s commitment to quality is crucial. Furthermore, at Abstracta we like to talk about metrics but also about milestones that the team achieved together.

Learn more about this topic in this article.

Reporting

We test software in order to gather information about the quality and risks, the product, its users, and the conditions of its use. This information helps our clients to make empirically informed decisions about the product, project, or business. One way of giving visibility to our test results is to share them through test reports. 

Of course, the purpose of our work is not on documenting, but on delivering working software as the Agile Manifesto says, we want to get the most out of our time spent in reporting.

We are aware that it is not enough to plan a testing strategy and to carry out performance, exploratory, manual, or automated tests carefully; sharing all this with our customers is a must for our team. So we (Abstracta and its customer, as a team, as a collaboration between partners) can be informed and act accordingly. 

In this path, we typically agree on different types of reports according to what’s best for the collaboration between the different parts involved, which could be weekly, quarterly, or with different frequencies and different levels of detail.

We see a test report as an opportunity to stop to reflect, recap on our achievements, celebrate them, and also learn about the process, and the difficulties we had to overcome, in order to continue improving day after day. 

When preparing a report you gather more information, from a different perspective, by doing a retrospective analysis. That can lead you to find different opportunities for improvement and discuss them with the team. It’s another opportunity to learn and share these learnings with the rest.