Sad Truth Is, Bad Tests Are the Norm!

This article is a high-level overview introducing some often overlooked concepts and the need for productive functional and or integration tests.

Modern development should be founded on the following:

• Iterating with fast feedback
• Eliminating waste
• Amplifying value

Based on Lean_software_development.

Allen Holub tweeted:

"Agile:

• Work small
• Talk to each other
• Make people’s lives better

That’s it. All that other junk is a distraction."

Here’s a link to the original tweet. Here’s more information.

What’s the Problem?

How can we effectively fulfill the above, if we don’t test productively? We should be focusing on delivering value, however, most companies and projects don’t focus on this. In the ideal world, your whole company would be aligned with Agile to facilitate fast feedback and quick iterations. This very concept is hard for some to imagine, maybe you think that you are already doing this? To paraphrase Allen Holub — we have world views; one person’s world view could be an agile one, as they have worked in a startup. Another, if they have worked in a non-tech savvy organization, couldn’t imagine what agile looks like.

When it comes to testing, many people seem to have the world view that hard-to-maintain tests are the norm and acceptable. In my experience, the major culprits are BDD frameworks that are based on text feature files. This is amplifying waste. The extra feature file layer in theory allows;

• The user to swap out the language at a later date
• Allows a business person to write user stories and or acceptance criteria
• Allows a business person to read the user stories and or acceptance criteria
• Collaboration
• Etc…

You have actually added more complexity than you think, for little benefit. I am explicitly critiquing the approach of writing the extra feature file layer first, not the benefits of BDD as a concept.

You test more efficiently, with better results not writing the feature file layer, such as with Smart BDD, where it’s generated by code. Here I compare the complexities and differences between Cucumber and Smart BDD. In summary and reality the feature file layer:

• Doesn’t help with swapping out the language. Both language implementations and maintenance would have diminishing returns.
• Doesn’t help with a business person writing user stories and or acceptance criteria, because as the article above shows in more detail, the language you use is actually determined by the framework and limitations thereof. Therefore, it would need to be re-worded to some degree, hence a feature file isn’t helping.
• Doesn’t help with a business person reading the user stories and or acceptance criteria. Usually, feature files are badly written due to the complexity of the BDD framework, therefore a business person in reality is more likely to ask a developer how the system behaves. Smart BDD generates more concise and consistent documentation.
• Does help with collaboration, which is a major benefit

What’s the Best Tool for the Job?

If you want to be productive, you might seek the best tool for the job, but what is “the best tool for the job”? It’s retrospective; you don’t know upfront. Using an extra framework alongside your existing unit test framework could be wasteful. It’s better to learn one testing framework well. For example, Cucumber is evidently hard to master leading to poor quality tests that take longer and longer to maintain.

You can easily underestimate the future complexities, especially with frameworks that make you write feature files first. You don’t plan to write poor-quality tests, it’s something that could happen over time. You don’t want to spend extra time battling over a framework.

You could have the worst of all worlds and use a feature file-driven framework with language you’re not proficient in. If you’re a Java developer using Ruby or Typescript etc. for tests, this could also lead to poor quality and less productivity. I’m suggesting, that if you’re a Java developer, Smart BDD would be the closest to your main skill-set (the least friction) and it tries its hardest to promote good practices. You do less and get more out!

Test Based on Your Project's Needs

If you’ve heard of the testing pyramid, you can use it as a reference, but do what works for you. You don’t have to do more unit tests and less functional testing because you think that’s the shape of a triangle. You need to align your culture by thinking what value does something provide? Aim for what works for your team, not something designed by a committee. The number of unit tests or the coverage is an output, not an outcome. Aiming for some % test coverage is an amplifying waste. With unit tests, TDD is about code quality, it drives the architecture, and coverage is a side effect. Higher quality code is better for agility.

Where does Smart BDD and or another productive testing framework fit in? If you are going to test, it’s best to test first, as it’s more work to test later, and you’ll miss many of the benefits of testing first. With any new feature and or requirement you should generally start outside in. If you start with the functional tests, you start to understand better the requirements and the features that you are delivering. A lot of software development is about learning and communication. Once you have validated that you are working on the right thing, and you’ve increased your understanding of the problem you’re solving, you can work on the code, ideally using TDD. Next is to get the feature in the hands of your client and or the next best available person for feedback. Obviously, feedback is feedback and not future requirements. Feedback from a client is not a silver bullet. It could be used with metrics, for example, was the new feature used as expected? Think of this process as learning, or even better, validated learning if you can prove something.

You should strive to solve the problem; how can you get meaningful feedback, as soon as possible? It’s a red flag when you spend too long writing and maintaining functional tests. For example, if you use well-designed defaults and builders you can really reduce the amount of work required in creating and maintaining functional tests. You also want the test framework to be smart, for example;

• Create interaction diagrams
• Test boundary cases
• Produce metrics like timers
• Give insights
• And many more

At the heart of this is specifying intent and not implementing it many times over. I think the industry in general is moving in the direction of declaring intent over implementing details.

By using best practices you’ll get better at testing/documenting behavior at an appropriate level and not making the tests/documentation obfuscated with irrelevant data.

Conclusion

Culture is hugely important, I’m sure we and our bosses and senior leaders would all ultimately agree with the following:

• For more value, you need more feedback and less waste
• For more feedback, you need more value and less waste
• For less waste, you need more value and more feedback

However, most work culture is not aligned with this. Agreeing with something and having the culture are very different, it’s the difference between agreeing that eating healthily and exercising is good for you and actually doing something about it. The next level in the healthy analogy is, having friends or a partner that are similarly minded. Is your company metaphorically encouraging a healthy lifestyle or just agreeing that being healthy makes sense?

Culture drives your mindset, behavior, and processes.

This has been a very brief introduction, hopefully, you’ll think about amplifying value, thanks for reading please do have the courage to do more or less of something and introduce change where needed.