Security Bugs Are Fundamentally Different Than Quality Bugs

This topic has come up a few times this year in question period: arguments that quality bugs and security bugs "have equal value," that security testing and QA are "the same thing," that security testing should "just be performed by QA" and that "there’s no specific skillset" required to do security testing versus QA. This article will explain why I fundamentally disagree with all of those statements.

First, some definitions.

A software bug is an error, flaw, failure, or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.

A security bug is specifically a bug that causes a vulnerability. A vulnerability is a weakness which can be exploited by a threat actor, such as an attacker, to perform unauthorized actions within a computer system.

QA looks for software bugs (any kind); security testers look for vulnerabilities. This is the main difference — their goals.

Just as all women are human beings, but not all human beings are women; while all security bugs are defects, not all defects are security bugs.

Now let’s dissect each of the claims above.

1. Quality Bugs and Security Bugs "Have Equal Value"

If a security bug can lead to a low risk vulnerability, it does not have "the same value" as a non-security-related bug that is making the system crash over and over. The same as if a security bug is creating a situation of a potential data breach, or worse, it’s not equivalent to the fonts not matching from page to page. I am of the opinion that security bugs are more likely to be able to cause catastrophic business harm than a regular bug, due to the fact that if your system has fallen under the control of a malicious actor, creativity is the only limit. Malicious actors never cease to amaze me with the damage they can do.

2. Security Testing and QA Are "The Same Thing"

The goals of security testing and quality assurance testing are different, which I feel makes them obviously different (if they were the same, why would they not be called the same thing?). However, I want to get deeper into this idea.

Security is a part of quality.

I often say “Security is a part of quality” because I believe this to be true. You cannot have a high-quality product that is insecure; it is an oxymoron. If an application is fast, beautiful, and does everything the client asked for, but someone breaks into it the first day that it is released, I don’t think you will find anyone willing to call it a high-quality application.

There are many different types of testing:

• Unit Testing — small automate-able tests to verify small units of code (functions/subroutines) to verify it does the one thing it is supposed to do.
• Integration Testing — test between different components to ensure they work well together. Larger than unit tests, but less intense than end-to-end tests.
• End-To-End Testing — ensuring the flow of your application from start to finish is as expected.
• User Acceptance Testing (UAT) — manual and/or automated testing of client requirements (often used interchangeably with "QA").
• User Experience Testing (UX) — verifying that the application or product is easy to use and understand from a user perspective).
• Regression Testing — verifying that new changes have not broken anything that was already tested, a "retesting" of all previously released functionality.
• Stress/Performance/Load Testing — verifying your application can handle large amounts of usage/traffic while continuing to perform well, generally performed using software tools (although each of these three have slight differences, they are all generally lumped in together).
• Security Testing — a mix of manual and automated testing, using one or more tools, with the aim to find vulnerabilities within applications.

There are more types of testing, but I think you get the point.

Some or all of these types of testing can be used to verify that a product is of high quality, and security is just one part. Therefore, security testing and QA are not "the same thing."

3. Security Testing "Should Be Performed by QA"

For each one of the types of testing listed above, a different skillset is required. All of them require patience, attention to detail, basic technical skills, and the ability to document what you have found in a way that the software developers will understand and be able to fix the issue(s). That is where the similarities end. Each one of these types of testing requires different experience, knowledge, and tools, often meaning you need to hire different resources to perform the different tasks. Also, we can’t concentrate on everything at once and still do a great job at each one of them.

Although theoretically you could find one person who is both skilled and experienced in all of these areas, it is rare, and that person would likely be costly to employ as a full-time resource. This is one reason that people hired for general software testing are not often also tasked with security testing.

Another reason is that people who have the experience and skills to perform thorough and complete security testing are currently a rarity. There is a skill shortage, while as an industry we are lucky to have quite a number of skilled QA professionals, making them easier to hire and staff. Lastly, the amount of time, training, and experience that it takes to become a security tester versus a general software tester is more difficult to acquire.

Training on how to perform security testing is extremely expensive and difficult to find, it generally takes longer to learn it as a skill than other types of testing, and there are fewer opportunities to get into that industry, when compared to QA. Thus, it is more difficult to become a security tester, when compared to general testing. Scarce resources, high demand, and expensive training means it costs significantly more to hire security testers than it does to hire general software testers.

All of these facts lead up to the reality that it is cost-prohibitive to have staff your QA team with professionals who are skilled and experienced in both QA and security testing. This also means that you are creating a single point of failure for testing in your organization, which will not save you money in the long run.

Another point on this topic: those who work in the security industry are likely to have a preference for their area of focus, security, and may be unwilling to perform other types of work outside their area of concentration. People who specialize generally want to work within their area of specialization, whenever possible, and security testing is a specialization.

4. "There’s No Specific Skillset" Required to Do Security Testing Versus QA

First of all, I feel this statement is insulting to QA testers, as though they do not have a specific skillset that makes them good at what they do. I don’t believe that to be true. I suspect that when people make this argument that it is out of frustration with our industry, because I honestly cannot fathom someone thinking that security testing does not require specific experience, training, or skills; otherwise, there would be no skills shortage and it would not be a high-paying job. Security testing is a specialization within the field of testing, just as there are specializations within any field, and by definition it requires more knowledge and training to form the skillset in order to do the job.

I do not intend to downplay the value of QA testing; I only explain that quality assurance is different from ensuring that a product is secure. I should also say that I feel that hacking is sometimes glorified in television, the media, and our industry as a whole, in a way that isn’t logical to me. Security testing is very important, but I do not believe that hackers are superior to other professionals who work in IT. In fact, I choose to focus my career on AppSec, DevSecOps, and other types of defense because I truly believe that it is more important that we write secure code than we "hack all the things." Security is so much more than just security testing (ethical hacking); it is secure design, secure coding, threat modelling, and so on.

I feel comments like this (#4) are not based on facts, but feelings, and it’s difficult to debate with someone when that is the case.

It is okay if we disagree on this topic. Debate is good and healthy, and I would love to hear your feelings, thoughts and ideas in the comments.

At this point I’d like to remind you all that security is everybody’s job. Not only is it everyone’s responsibility to do their job in the most secure way they know how, but having many different people look at something with security in mind can help us find new and different problems that may have otherwise been missed.