Requirements, Code, and Tests: How Venn Diagrams Can Explain It All

In software development, requirements, code, and tests may form the backbone of our activities. Requirements, specifications, user stories, and the like are essentially a way to depict what we want to develop. The implemented code represents what we’ve actually developed. Tests are a measure of how confident we are that we’ve built the right features in the right way. These elements, intertwined yet distinct, represent the essential building blocks that drive the creation of robust and reliable software systems. However, navigating the relationships between requirements, code implementation, and testing can often prove challenging, with complexities arising from varying perspectives, evolving priorities, and resource constraints.

In this article, we delve into the symbiotic relationship between requirements, code, and tests, exploring how Venn diagrams serve as a powerful visual aid to showcase their interconnectedness. From missed requirements to untested code, we uncover many scenarios that can arise throughout the SDLC. We also highlight questions that may arise and how Venn diagrams offer clarity and insight into these dynamics.

The General Case

A new feature is coming up for a software development team. To make things simple, we will concentrate on software behavior. We will consider software code as a means to implement behavior. Within the universe of all possible behaviors (designated as the rectangle in Figure 1), we can identify three main sets. Set S is the universe of all specified behaviors. Set C is the universe of all implemented behaviors. Set T is the universe of all tested behaviors.

There are also areas that are numbered in Figure 1. Let’s explore each one of them in detail.

• In area 1, we find all behaviors that have been specified, coded, and tested. This is the happy path area where we’ve tested all behaviors that have been specified and implemented. 
• In area 2, we find all behaviors that have been specified and coded but not tested. This is the classic area where we may expect bugs to be undetected and released. 
• In area 3, we find all behaviors that are coded and tested but have not been specified. Is this code justified? Why does it exist? Testing it could be a good idea as it may be buggy, or it could negatively affect other areas of the code.
• In area 4, we find all behaviors that are specified and tested but have not been implemented in the code. So, we’ve found requirements that are not implemented. Is there a legitimate reason why the behavior is not implemented?
• In area 5, we find all specified behaviors that have not been implemented in the code or tested. If they should have been implemented and tested, then we need to allocate the appropriate resources. This is an area that may be unnoticed by the development team since it is untested.
• In area 6, we find all behaviors that are implemented in the code but have not been specified or tested. Is this code justified? Why does it exist? This is another area that may be unnoticed by the development team since it involves untested behavior.
• In area 7, we find all behaviors that are not specified; they are not implemented in the code but are tested. For testing to be legitimate in this area, we need to justify our testing activities. Could this be emergent behavior from other features? Otherwise, we do not expect much value from testing. However, it may be a good idea to test emergent behavior or to make sure that unwanted behavior is not there.
• Finally, in area 8, we find all behaviors that are not specified, not implemented in the code, and not tested. This is another area that rarely gets spotted in practice. We need to be aware of our blind spots, and minimizing this area as much as possible is a good idea. Maybe there are ways of using our software that we’ve never thought of before. Perhaps there are environments where our software can be used that we’ve never tested.

In short, we can expect that there may be specified behaviors that are not tested (areas 2 and 5), specified behaviors that are tested (areas 1 and 4), and test cases that correspond to unspecified behaviors (areas 3 and 7). Similarly, there may be implemented behaviors that are not tested (areas 2 and 6), implemented behaviors that are tested (areas 1 and 3), and test cases that correspond to behaviors that were not implemented (areas 4 and 7). All areas have their own significance. If specified behaviors exist for which no test cases are available, then our testing is incomplete. If certain test cases correspond to unspecified behaviors, several scenarios may unfold: either the inclusion of such test cases lacks justification, indicating potential deficiencies in the specification, or the tester aims to verify the absence of certain unspecified behaviors. Usually, proficient testers frequently propose test cases of this nature. This underscores the importance of involving skilled testers in specification and design reviews, as they bring valuable insights to the process.

This covers the basic cases that we usually encounter in software development. The beauty of Venn diagrams lies in the simplicity that they depict what may happen. But let's dive into more details.

The Confidence Case

This is where sets S and C are one and the same, implying that what we want to develop is exactly what we’ve developed. Assuming that we haven’t missed or misinterpreted anything in our requirements, this is a very good start. This is also an idealistic scenario for various reasons. However, it can serve as a compass for directing our efforts. For a confident release that no major bugs have slipped to production and that all is developed as expected, set T could be a superset of S and C as shown in Figure 2.

Area 1 contains all specified behaviors that are implemented and tested. Area 2 contains unspecified and unimplemented behaviors that have been tested. Area 3 contains unspecified, unimplemented, and untested behaviors. Minimizing Area 3 will also boost the development team’s confidence for optimal results.

Reasons For Undocumented or Wrong Requirements

Wrong requirements are those that are documented when they should not exist. They exist for the wrong reasons, and they should be avoided. Wrong and undocumented requirements can arise due to a variety of factors, including assumptions, changing priorities, lack of clarity, time constraints, scope creep, legacy systems, communication issues, and insufficient stakeholder involvement. To mitigate the risk of such problems, development teams need to prioritize thorough requirement gathering, effective communication, and stakeholder engagement throughout the software development process.

• Assumed Knowledge: Sometimes, stakeholders or developers may assume that certain requirements are so obvious or fundamental that they do not need to be explicitly documented. This assumption can lead to important requirements being overlooked or neglected in the documentation process.
• Changing Priorities: In dynamic project environments, priorities and requirements can shift rapidly. As a result, certain requirements may be added, removed, or modified throughout the development process. If not properly managed, these changes can result in undocumented requirements and wrong requirements.
• Lack of Clarity: In some cases, stakeholders may have vague or ambiguous expectations, making it challenging to document requirements effectively. Without clear communication and understanding between stakeholders and development teams, crucial requirements may go undocumented or may be misinterpreted and documented wrongly.
• Time Constraints: Development projects often operate under tight deadlines, leaving limited time for comprehensive requirement gathering and documentation. In such cases, teams may prioritize implementing functionality over documenting requirements, leading to gaps in documentation.
• Scope Creep: Scope creep occurs when additional features or functionalities are added to a project without proper documentation or approval. As new requirements emerge, existing documentation may become outdated or incomplete, resulting in undocumented requirements or wrong requirements.
• Legacy Systems: When working with legacy systems or codebases, it's not uncommon for undocumented requirements to exist. Over time, changes and updates may be made without proper documentation, leaving behind undocumented functionality or dependencies.
• Communication Issues: Poor communication between stakeholders, developers, and other project team members can lead to misunderstandings, missed requirements, and wrong requirements. Without effective communication channels in place, important requirements may not be properly documented or conveyed to the development team.
• Lack of Stakeholder Involvement: If key stakeholders are not actively involved in the requirement gathering and documentation process, important requirements may be overlooked or omitted. Engaging stakeholders throughout the development lifecycle is essential for capturing and documenting all relevant requirements.

Reasons for Unimplemented Requirements

The non-implementation of requirements can be attributed to various factors, including technical constraints, scope changes, resource limitations, misinterpretation or miscommunication, changing priorities, budget constraints, quality assurance concerns, and legal or regulatory compliance requirements. Effective project management, clear communication, and proactive risk mitigation strategies can help minimize the impact of these factors and ensure that essential requirements are implemented successfully.

• Technical Constraints: Certain requirements may be technically challenging or infeasible to implement, given the current technology stack, infrastructure limitations, or resource constraints. In such cases, developers may need to find alternative solutions or workarounds, leading to the non-implementation of specific requirements.
• Scope Changes: As project priorities and objectives evolve, there may be changes to the project scope that result in certain requirements being deprioritized or deemed unnecessary. These changes could be driven by shifting business needs, budget constraints, or changes in market conditions, leading to the non-implementation of some requirements.
• Resource Limitations: Development projects often operate under constraints such as time, budget, and personnel. If resources are limited or allocated inefficiently, certain requirements may not receive the attention or resources needed for implementation, resulting in their non-implementation.
• Misinterpretation or Miscommunication: Requirements that are unclear or ambiguously communicated may lead to misunderstandings between stakeholders and development teams. If requirements are not properly understood or documented, there is a risk that they may not be implemented as intended, leading to their non-implementation.
• Budget Constraints: Budgetary considerations play a significant role in determining which requirements are implemented during the software development process. If a project exceeds its budget or funding is limited, there may not be sufficient resources available to implement all requirements, leading to the non-implementation of some features or functionalities.
• Quality Assurance Concerns: During the development process, certain requirements may be identified as posing risks to the overall quality or stability of the software. In such cases, development teams may choose to defer or exclude the implementation of these requirements until quality concerns can be addressed, resulting in their non-implementation.
• Legal or Regulatory Compliance: Requirements related to legal or regulatory compliance may introduce additional complexity or constraints that impact their implementation. If compliance requirements are particularly stringent or challenging to meet, developers may need to prioritize other requirements, leading to the non-implementation of certain features or functionalities.

Reasons for Untested Requirements

These are requirements that are not covered by any test cases. While the code may technically fulfill these requirements, their absence from the test suite leaves them vulnerable to potential bugs or regressions. Resource constraints, incomplete test coverage, requirement bugs, insufficiency of testing types and testing levels, and complexity that leads to untestability are some of the reasons that may lead to untested requirements.

• Resource Constraints: Limited testing resources, including time, budget, and personnel, may restrict the ability to test all requirements thoroughly. Testing teams may need to prioritize their efforts based on available resources and project constraints.
• Incomplete Test Coverage: Incomplete coverage criteria may limit the extent of test coverage, resulting in certain requirements being left untested. Inadequate test planning or prioritization may also contribute to gaps in test coverage.
• Bugs in Requirements: Requirements themselves may contain bugs or inconsistencies that go unnoticed until testing begins. Identifying and rectifying such problems may require additional time and effort, delaying testing activities for affected requirements.
• Insufficient testing levels and types: Different types of testing and levels of testing address different testing needs and requirements. Unit-level testing, integration-level testing, API-level testing, and UI-level testing, focusing on positive and negative tests, may be necessary. Both functional and non-functional testing may also be appropriate. manual or automated, or both.
• Complexity: Some requirements may be inherently complex or involve intricate business logic, making them challenging to test thoroughly or making them completely untestable. Testing such requirements may require specialized skills, tools, or resources that are not readily available.

Reasons for Testing the Unexpected

Software behavior that is unspecified and unimplemented in the code but has been tested can be important for several reasons.

• Discovering Hidden Functionality: Sometimes, during testing, unexpected behaviors or functionalities that are not explicitly documented or implemented may be uncovered. These behaviors could indicate hidden features or functionalities that may have been overlooked during the requirements gathering or implementation phase. For example, a tester might stumble upon a shortcut or workaround in the software that was not part of the original requirements but provides added value to users.
• Identifying Bugs: Testing behaviors that are unspecified and unimplemented can help uncover defects or bugs in the software. These defects may arise due to incomplete or inaccurate requirements documentation, misinterpretation of requirements, or errors in the implementation process. By testing these behaviors, testers can identify areas where the software deviates from expected behavior and flag potential issues for resolution.
• Enhancing User Experience: Testing unspecified and unimplemented behaviors can provide valuable insights into user experience and usability aspects of the software. By exploring how users interact with the software under different scenarios, testers can identify usability issues, pain points, or areas for improvement that may not have been addressed in the original requirements. For example, testers may discover navigation challenges or inconsistencies in the user interface that impact user satisfaction.
• Validating Assumptions: Testing behaviors that are unspecified and unimplemented can help validate assumptions made during the development process. These assumptions may relate to how users will interact with the software, how certain features will perform under specific conditions, or how the software will integrate with other systems or platforms. By testing these assumptions, testers can validate their accuracy and identify any discrepancies that may require further investigation or adjustment.
• Mitigating Risks: By testing unspecified and unimplemented behaviors, testers can help mitigate risks associated with incomplete or inaccurate requirements documentation, potential defects or bugs in the software, and usability issues that may impact user satisfaction. Identifying and addressing these risks early in the development process can help prevent costly rework, delays, or customer dissatisfaction down the line.

Wrapping Up

Venn diagrams, an indispensable tool of set theory, serve as a powerful mechanism for showcasing the intricate relationships between requirements, code, and tests in software development. By visualizing these relationships, testers can better navigate the complexities of software testing, ultimately contributing to the delivery of high-quality and reliable software systems.