Code Coverage vs Test Coverage — Which Is Better?

Test Coverage and Code coverage are the most popular methodologies for measuring the effectiveness of the code. Though these terms are sometimes used interchangeably since their underlying principles are the same. But they are not as similar as you may think.

Many times, I have noticed the testing team and development team being confused over the use of these two terminologies. This is why I thought of coming up with an article to talk about the differences between code coverage and test coverage in detail.

You may also like: JaCoCo End-to-End Code Coverage at Runtime

Code Coverage vs Test Coverage: How Do They Differ?

Code coverage: indicates the percentage of code that is covered by the test cases through both manual testing and automation testing with Selenium or any other test automation framework. For example, if your source code has a simple if…else loop, the code coverage would be 100% if your test code would cover both the scenarios i.e. if and else.

Test coverage: includes testing the features implemented as a part of the Functional requirements specification, software requirements specification, and other required documents. For example, if you are to perform cross-browser testing of your web application to ensure whether your application is rendering well from different browsers or not? Your test coverage would be around the number of browsers + OS combinations over which you have validated browser compatibility of your web application.

With the understanding of the basic difference between code coverage vs test coverage, let us jump into further details around code coverage and test coverage.

Understanding Code Coverage In-Depth

Code coverage is performed by developers during unit testing to verify the code implementation in such a manner that almost all the statements of code are executed. Most of the code coverage tools make use of static instrumentation where statements that monitor the execution are inserted at necessary locations in the code.

Though the addition of instrumentation code increases overall application size and execution time, the overhead is minimal when compared to the information that is generated via the execution of the instrumented code. The output consists of a report that details the test suite’s test coverage.

Why You Should Perform Code Coverage?

Unit tests are primarily used to test the code at an individual unit level. Since unit tests are written by the developer himself, he has better visibility of the tests that should be included as a part of unit testing. Unit testing helps improve the overall quality of the software, but there will always be a question about the number of tests that comprise unit testing. Are there enough test scenarios in the test suite? Should we add more tests? Code coverage is the answer to all these questions.

As the product development progresses, new features, as well as fixes (to the bugs raised during testing) are added to the release cycle. This means that the test code may also require changes to keep it updated with the software changes made during development.

The testing standards which were set during the start of the project must be maintained with subsequent release cycles. Code coverage can be used to make sure that your tests are meeting those standards and the best quality code goes into the production phase.

Higher the percentage of code coverage; the lower are the chances of having undetected bugs. In some organizations, the quality team sets the minimum amount of code coverage that needs to be achieved before the software is pushed to the production phase. The primary reason for the same is to reduce the probability of bugs being detected at a later stage of product development.

How To Perform Code Coverage?

There are different levels at which the code coverage is possible, some of the common subtypes of code coverage are:

• Branch coverage – Branch coverage also called decision coverage is used to ensure that every possible branch used in a decision-making process is executed. For example, if you are incorporating a fall back for cross-browser compatibility using an If…An else conditional statement or by a Do…While statement in your code, as a part of coverage; you have to ensure that all the branches i.e. If, Else, Do, and While are tested by providing appropriate inputs to make a cross-browser compatible website.
• Function coverage – Function coverage ensures that necessary functions (especially exported functions/APIs) are tested. This should also include testing the functions with different types of input parameters as that would also test the logic used in the functions. Once all the functions in the code are tested, function coverage would be 100%.
• Statement Coverage – This is an important code coverage methodology in which test code has to be written in a manner that every executable statement in the source code is executed at least once. This also includes corner cases or boundary cases.
• Loop Coverage – This approach is to ensure that every loop in the source is executed at least once. There might be some loops that may execute based on results that you achieved at runtime, it is important to test such loops as well to make the code foolproof.

For checking the code coverage, an approach called instrumentation is used. Instrumentation can be used to monitor the performance, insert trace information, and diagnose any kind of error in the source code. There are different kinds of instrumentation and there could be minimal performance (and timing) overhead depending on the instrumentation approach being used.

Types of Instrumentation

There are three major types of instrumentations

• Code instrumentation – Here the source code is compiled after the addition of instrumentation statements. The compilation should be done using the normal toolchain, successful compilation results in the generation of instrumented assembly. For example, to check the time taken to execute a particular function in your code, you can add instrumentation statements in the Start and End of the function.
• Runtime instrumentation – Contrary to the code instrumentation approach, where the information is collected from the runtime environment i.e. when the code is under execution.
• Intermediate code instrumentation – In this type of instrumentation, an instrumented class is generated by the addition of byte codes to the compiled class files.

Depending on your test requirement, you should choose the right code coverage tool and the best instrumentation approach supported by the tool.

Tools for Code Coverage

Several code coverage tools are supporting different programming languages and many of them double up as QA tools as well. Many of the tools can be integrated with build tools and project management tools which make them much more powerful and useful.

While choosing an open-source code coverage tool, you should check the features supported by the tool and whether there is an active development of the tool. Based on those factors, below are some of the popular open-source code coverage tools.

1. Coverage.py – It is a code coverage tool for Python. As the name suggests, it analyzes your source code and identifies the percentage of code that was executed. It is developed in Python. It is free of cost and you can refer https://coverage.readthedocs.io/en/coverage-4.4.1/index.html for more information.
2. Serenity BDD –Supporting Java and Groovy programming languages, Serenity BDD is a popular open-source library that is primarily used for writing excellent quality acceptance tests faster. You can make use of stories and epics for the tests and the code coverage is computed for these stories and epics. Due to this, the test reports that are generated are more illustrative and narrative. You can map those automated tests back in your requirements.

   It can be used with JUnit, Cucumber, and JBehave. Serenity BDD can be easily integrated with Maven, Cradle, JIRA, and Ant. If you are using the Selenium WebDriver or Selenium Grid framework for automation testing, choosing Serenity BDD can be a huge advantage due to its compatibility with Selenium. You can refer to https://www.thucydides.info/#/documentation for further information.

3. JaCoCo – JaCoco is a code coverage tool for Java. Though there are other options like Cobertura and EMMA, these tools were deprecated since there was no update for a long time. The JaCoCo tool is a part of the Eclipse Foundation and it replaced the EMMA code coverage tool in Eclipse. Apart from the active development of JaCoCo, another advantage of using it is seamless integration with CI/CD and project management tools like Maven, Jenkins, Gradle, etc. Please visit the official site for more information.
4. JCov – JCov is a test framework agnostic code coverage tool. It can be effortlessly integrated with Oracle’s test infrastructure – JavaTest and JTReg. Though it is not in active development, support for on-the-fly instrumentation and offline instrumentation are the major advantages of using JCov. Please visit https://wiki.openjdk.java.net/display/CodeTools/jcov for further information.
5. PITest –Most of the code coverage tools check the code for branch coverage, statement coverage, loop coverage, etc. and give out the coverage results. Though that information is useful to improve the quality of your test code, the test does not tell you how good the automation tests were in discovering bugs. This is where mutation testing can be very helpful.

   PITest is a very popular code coverage tool that is used for mutation testing for Java and JVM. It does the job of mutation testing by modifying your test code and the unit tests are now performed on this modified code. If bugs are discovered using this code i.e. after extra code is added by PITest, the unit test is foolproof; else it needs changes as the issues were left undiscovered. PITest is easy to use, fast and is under active development. It also integrates with popular CI/CD tools which makes it much more useful. Please refer to http://pitest.org/ for more information.

Understanding Test Coverage In-Depth

Unlike code coverage which is a white-box testing methodology, test coverage is a black-box testing methodology. Test cases are written in a manner that there is maximum coverage of requirements mentioned in FRS (Functional Requirements Specification), SRS (Software Requirements Specification), URS (User Requirement Specification), etc. Since the tests are derived from these documents, there are minimal/no chances of automation.

How To Perform Test Coverage?

Like Code coverage, Test coverage can also be evaluated through different types of testing. However, which type of testing should you follow, depends entirely on your business proposition For example – In user-centric web applications, there might be a case where the UI/UX tests are of higher priority over functional tests whereas in another type of applications (e.g. banking, finance); usability tests, security tests, etc. might be more important.

Some of the test coverage mechanisms are below:

• Unit Testing – This type of testing is performed at a unit level/module level. Bugs encountered at a unit level can be different from the issues encountered at the integration stage.
• Functional Testing – In functional testing, the functions/features are tested against the requirements mentioned in the Functional Requirement Specification (FRS).
• Integration Testing – It is also called system testing since the software is tested on a system level. This type of testing is performed once all the required modules are integrated.
• Acceptance Testing – It all depends on the result of acceptance testing whether the product would be released to the end consumer/customer. This is where the developers get a sign-off as a green pass from the testers and SMEs of the web-application, right before pushing the code changes from the Staging environment to the Production.

The other important point to note is that the purpose and meaning of test coverage can differ depending on the level at which testing is performed. It also depends on the type of product on which black-box testing is performed. The test coverage metrics for testing mobile phones would differ from metrics for e-commerce website testing. Some classifications are below:

• Features coverage – Here the test cases are developed in a manner that there is maximum coverage of product features. For example, if a tester is assigned the job of testing a phone dialer application; he should make sure that the number being dialed is proper in length. If the testing is done in India, the mobile number should be a maximum of 10 digits; else it should flash an error. Similar to this, all the mandatory and optional features have to be tested according to a priority set by the product team.
• Risks coverage –Every product/project requirement document has a section that mentions the Risks and Mitigation associated with the project. Though some risks (e.g. changes in business dynamics) are beyond the scope of the planning/development/test team, some risks need to be addressed during the testing phase.

  For example, while developing a business website, the server infrastructure should be set up in a manner that page access is very fast. Depending on the location (i.e. country, city, etc.) from where the website is accessed, the closest server should be chosen for loading the website; else the entire experience would be hampered. The test team should also perform a load test where a performance test is done when multiple numbers of users are trying to access the website at the same time i.e. scenario where there is huge traffic on the website. If the results of these tests are not good, it can result in a below-average user experience (which can be a huge risk).

• Requirements coverage –Here tests are defined in a manner that there is maximum coverage of the product requirements mentioned in various Requirement Specification documents.

  For example, if you are testing a pre-installed SMS application, you need to make sure that the default language is set properly e.g. If the mobile is marketed in a country where English is not the primary language e.g. China; the default SMS language should be Chinese and for other customers e.g. India; the default language should be English.

Tools for Test Coverage

In the case of code coverage, the measurement metric is the percentage of code that is tested through test cases/test suites. Hence, the test results can be quantified i.e. Out of 100 LOC (Lines of Code), the code coverage was 80 lines. This means that the code coverage is 80%. The results of test coverage cannot be quantified since the tests are performed to verify the functional requirements.

You can also come up with black-box tests that can test more than one requirement in a single test e.g. To test the failure scenario in a simple email login-page, you can write a test case that enters an email address without @ symbol than try to proceed with the login. This would test the functionality of the login page and also check if the logic for verifying the format of the email address is working as per the requirement.

Though there are few instances where you have to write test code for achieving the test coverage requirement, you may still need to use some of the popular test frameworks in some cases. Two of the most popular test frameworks are:

• JUnit – JUnit is the unit testing framework for Java. It can also be used for UI testing. It is open-source and considered important in the development of TDD (Test Driven Development). Developers and testers make use of JUnit to write and execute repetitive tests. This also makes it a popular framework for regressive testing. Please refer to https://junit.org/junit5/ for more information.
• PyUnit – PyUnit (also called Python Unit Testing Framework) is a widely used testing framework primarily used for unit testing. It is the Python port of JUnit and is used by Python developers who follow the TDD approach. PyUnit supports the development of test cases, test suites, test fixtures, etc. The unit test module is the core of the PyUnit framework. Please refer to https://docs.python.org/2/library/unittest.html for more information.

Though many other tools/test frameworks are using which developers/testers can write test code, JUnit and PyUnit are the most popular testing frameworks for their respective programming languages.

Code Coverage vs Test Coverage — Which One Should You Follow?

The bases on which the impact of code coverage and test coverage are measured are different. Code coverage is measured by the percentage of code that is covered during testing, whereas test coverage is measured by the features that are covered via tests.

The million-dollar question is “Which one of these is best suited for your project”?. There is no definite answer to this question as the solution depends on the type and complexity of the project. In most cases, test coverage and code coverage are used since they are equally important in software projects.

Advantages of Test Coverage

• A good approach to test the software features and compare the results across different specification documents (requirements, feature, product, UI/UX, etc.)
• As the tests performed as a part of the coverage are black-box in nature, executing those tests might not require much expertise.

Shortcomings Of Depending On Test Coverage

• There is no scope of automation as the tests are predominantly black-box tests. Manual comparison of the test results has to be done with the expected output since these tests are performed at a ‘feature level’ and not ‘code level’.
• No concrete way of measuring test coverage. Hence, the coverage results largely depend on the domain competence of the tester who is performing the tests and may vary from one tester to another.

Advantages of Code Coverage

• Provides the effectiveness of your test code and how you can improve the coverage
• Irrespective of the type of tool being used (open-source, premium), setting up a code coverage tool should not take much time.
• It helps in improving the code quality by capturing bugs in the code.

Shortcomings of Code Coverage

• The majority of code coverage tools are limited to unit testing.
• The methodology used by tools could be different hence; you may not be able to compare code coverage results of one tool to another.
• Searching for the best-suited tool could be a big task as you need to compare and try features from those tools before selecting the best one that suits your project requirements.
• There are very few tools that provide support for different programming languages e.g. Java, Python, C, etc. Hence, you may need to have more than one tool in case your team is using multiple programming languages (for test code development).

How Much Coverage Is Enough?

More the number of tests; the better would be the quality of the product. There should be frequent communication between the design, development, and test teams so that every stakeholder has detailed information about the activities involved in the project.

The test team should spend a good amount of time understanding the overall requirements and prioritize the test activities. To keep track of the progress, they should have a checklist that should be updated periodically (at least after every release).

It is important that the test team also has frequent communication with the Quality Assurance (QA) team since they have details about the target (test/code) coverage that has to be achieved for the product/project to be released to the client/customer.

To summarize, there is no specific thumb rule which mentions the minimum percentage of code coverage or test coverage that needs to be achieved while testing the product.

How Do I Ensure Maximum Code Coverage And Test Coverage?

One way in which you can attain better results through testing is by incorporating automation into the test plan. There is no way that the complete testing process can be automated, so you have to come up with a plan. The plan should highlight test activities that have to be performed manually and via automation.

Coverage is a useful metric but its scope is limited. There is no ideal way in which you can measure the effectiveness of the efforts spent on testing. It is a known fact that 100% coverage is only a myth.

A holistic approach that encompasses different types of testing e.g. automation testing, integration testing, manual testing, cross-browser testing, etc. would be extremely useful. It helps you measure the effectiveness of different tests and test systems in a single place.

Don’t Aim For A 100% Code Coverage

One thing you would want to make a note of is to never assume that your code changes are bug-free. The reason being, a 100% code coverage represents that every line of code is covered through your automated test scripts. However, we need to take into account the false negatives and false positives, that may sway us from the reality of our added code. Also, there would be some of the codes in your web-application that may be untested. You need to ponder around the untested code. Either remove it if it is from an archaic release or have it tested if it is from a recent hotfix that missed the documentation.

Conclusion

Code coverage is a white-box approach, whereas test coverage is a black-box approach to testing. Depending on the requirements and scope of the project, you should choose test coverage/code coverage/test coverage and code coverage. You should prioritize the test activities and assign a tentative deadline with each activity. Irrespective of the methodology being used, it is important to achieve a higher level of coverage since that would indicate that the code and product features are tested to a good extent.

Cheers!