Along Came a Bug

A software bug or defect exists when actual behavior deviates from expected behavior. This is when customers are unhappy and the development team’s morale is hit.

If the expected behavior is not clear, deviation from what’s intuitively expected raises the question: Is this a bug or a feature? In any case, from a customer’s point of view, bugs are undesirable. Customer-driven development teams make their best efforts to identify and fix bugs.

There are two complementary ways of viewing software bugs. One way to view them involves the metaphor of harmful insects that can be eliminated using pesticides. Software bugs are the insects and software testing is the pesticide. The second way is to view them as a learning opportunity that calls for the right learning processes.

Faults, Errors, and Failures

But what exactly is a bug? What can make a software product deviate from expected behavior? In terms of programming code, there can be faults, errors, and failures. A fault in the code may result in an error. An error may result in a failure. 

Another metaphor could help us clarify these concepts. We wake up in the morning and are in pain or feeling dizzy. Think of these symptoms as failures. We decided to go to the doctor. The doctor measures our pressure which is found to be very low. Think of this as an error since our pressure should not be that low! Then we do a blood test which shows a high cholesterol level. The high cholesterol level is another error like any deviation from acceptable levels. The doctor analyses all the failures and errors. She tries to find out the fault (or the faults) that resulted in the errors which led to the failures.

Errors could add up resulting in bigger failures (more pain). Errors could be irrelevant to each other, some causing failures while others causing no failures. Errors could also be canceling each other resulting in little or no failures (little or no pain). They could be due to the same fault, due to different faults, or due to the interaction between faults.

Here is an example of a simple method that accepts arrays of numbers and counts the number of zeros.

Method numZero has a fault in the initialization expression of variable i. Due to this fault, numZero doesn't take into account the first element of the input array. This is an error. As a result, if the first element is zero then numZero will fail in the calculation. However, if the first element is non-zero then numZero will not fail. For example, numZero ([1, 9, 0, 7]) correctly evaluates to 1 while numZero ([0, 1, 2, 6]) results in a failure since it incorrectly evaluates to zero.

There is a tendency in the software industry to use the term bug or defect informally to refer to any of the three:  Faults, errors, and failures.

Types of Bugs

To find bugs we need to test. Consequently, bug types are in accordance with the types of testing we perform to catch them. There exist functional bugs where functionality is broken. There also exist non-functional bugs like security bugs, reliability bugs, availability bugs, usability bugs, and performance bugs. Input validation bugs, happy-path bugs, negative testing bugs, regression bugs, smoke testing bugs, and bugs caught via manual testing or via automated testing. The list of bug types is long but most importantly, testing cannot prove that there are no bugs. It can help to gain confidence that we’ve found most of them or the most important ones. Be aware though to avoid the trap of constantly seeking to achieve zero bugs after each and every testing activity. Is it always worth the effort and the costs?

Bugs Should Be Detected Early

A bug could be created during many stages in the software development life cycle (SDLC). During requirements analysis, architectural design, development, or deployment. The earlier we detect and fix a bug the better. This is because the cost of fixing a bug increases considerably as we move from requirements to design, to development, and then to QA testing. As an English proverb says: A stitch in time saves nine. 

To detect bugs, testing at any level is key: at a unit level, at an integration level, at an API level, at a system level, or at an acceptance level.

One software group used testers throughout the SDLC. Bugs detected during requirements analysis were documented and solved with high priority. 

Another group has trained business analysts and product owners to have a testing mindset. They would look for positive and negative cases and problematic edge cases. Having found such problems they would directly notify testers to incorporate specific scenarios during their test-case design.

Another group used to employ product owners that were good exploratory testers. People who had the knowledge to explore and the ability to constructively critique requirements. People who also had business domain knowledge and a willingness to expand and focus on that. Product owners with such a background would often catch bugs early. They would also take the necessary steps to avoid complex requirements. They generated requirements that were clear, unambiguous, complete, and testable.

As Insects Get Immune to Pesticides

Following the insect-pesticide metaphor, in time, insects in biology will get immune to pesticides and we must use new pesticides to kill them. Similarly, software bugs in our code will get immune to our tests and we will need new tests to find them and fix them.

Tests are focused on specific areas of the code and in time they will find fewer bugs. This is because the code tested will eventually be improved and few (or no) defects will be caught. If we want our tests to always catch important bugs, we must test in new ways, new testing levels, and try different types of testing and our test cases must evolve as our code evolves.

Defect Tracking Systems

A defect tracking system is a tool for documenting problems and fixes. Information like defect priority and severity, reproduction steps, possible workarounds, and issuer and assignee, are some of the details included.

One team viewed documenting defects as rework and waste. They fixed bugs as soon as they were discovered. Unit tests were written to reproduce defects and the code was fixed so that unit tests passed. The tests and the fix were then checked-in and work would continue as normal. Testers would collaborate with developers as soon as a problem was discovered. If a developer could provide a fix immediately then the bug was not documented. If no developer was immediately available or if the fix required difficult decision making then the bug was documented.

Another team would create a list of bugs in a Google document shared between testers, developers, and product owners. These were bugs found during feature testing and before the feature release. At least all the important bugs were fixed before release. If the decision was to release candidate features with some low-priority bugs then they would document them in a defect tracking system. They always documented defects detected after feature releases. 

There was a group that viewed defect-tracking systems as a database of knowledge. All bugs were reported to the system irrespective of whether they were found before or after feature releases. They were looking for patterns in the bugs. They were asking questions like are there interrelated bugs? Why do some bugs that get fixed keep coming back again? Why are some specific bugs sporadic? Root cause analysis was conducted in a quest to find answers and learn how to prevent similar issues from reoccurring.

Detecting Bugs Is One Thing, Fixing Them Is Another

Although preventing, detecting, and fixing bugs early in the SDLC is clearly a best practice, fixing bugs can be challenging. If the costs of fixing are greater than the costs of not fixing then maybe a workaround could make customers happy (if available). This is perhaps when bug tracking systems could be very useful since all known issues that need to be tackled can be clearly documented and communicated to all stakeholders. Important bugs that require across-team collaboration, which may also require difficult decisions across teams and may take time to fix should be documented. Maybe we’ve found an important bug but there are currently more important ones that need to be fixed. We should fix the most important first whilst keeping the less important ones filtered per priority.

Artificially Introduced Bugs as a Learning Process

Bugs may also be artificially introduced into the system to check if the existing processes are good enough to detect them. This is a policy that has been popular in a number of software groups.

One group introduced UI bugs to check if their overnight test automation runs would catch them. Another group had a regular bug training session for developers. Bugs were introduced by the DevOps team and hints were given to developers in order to find and fix them.

There was a group that introduced bugs at a microservice level to check microservice behavior. They were also interested to see how such carefully crafted bugs would propagate across microservices.

A number of groups were inciting faults to ensure their systems were tolerant. They were switching systems on/off and deliberately running software that caused failures. They learned about how tolerant their systems were, what kind of failures can emerge, how to improve such problems, and how to respond to a number of unforeseen circumstances.

Wrapping Up

Software bugs exist in one form or another in almost all software products. Their presence makes everyone unhappy, including customers and software development teams. Maybe, one of the prerequisites for a software engineer to flourish is to keep a balance between faults, errors, and failures in code. Since they are inextricably linked to our work we need to be clear on how to prevent them as much as we can, how to spot them when they are introduced, and how to fix and handle them under any circumstances.