How Do You Choose the Best Test Cases to Automate?

According to the 2021 Test automation report, more than 40% of companies are looking to expand and invest their resources in test automation. While this doesn’t mean manual testing is going away, there is an increased interest in automation from an ROI perspective - both in terms of money and time. 

After all, we can agree that writing and running those unit test cases is boring. A good automation strategy can free up the tester’s time to tackle some of the more complex problems and help with the early detection of bugs.

However, teams often rush to automate tests without a proper testing strategy, which causes things to break when there is an overhaul. Maximize your energy and return on investment by choosing the right tests to automate.

What Should You Consider Before Automating Any Test Case?

1. Test Frequency

It is usually more efficient to write that one manual test for an edge-case component. In fact, testing a new functionality allows you to learn more about the application quickly. However, that’s not effective as the number of functionalities increase.

Split your test scenarios into two sections: Ones that are repetitive and ones that are one-off or pretty complex.

Automate the ones that repeat the most. You can even set a threshold for test frequency above which you will consider automating.

For example, application login or alert system tests are ideal candidates for test automation since they need to run after every application build. 

There are a couple of exceptions to this rule, too - say, the volume of data entry a single test needs to execute is pretty large. In that case, it makes sense to automate that specific test since it would save a lot of time.

The only caveat here is automating a series of repetitive tests that depend on each other. In case of a failure, it can be hard to pinpoint the exact test that is the main culprit. That’s where logs come in handy and help you detect these long-term pattern failures efficiently.

2. Test Coverage

Test coverage is crucial for software quality and ensuring the stability of your software build. Automating the right kind of tests can aid your goal of achieving high test coverage with almost the same amount of time invested. 

For example, if your application has many components, running automated tests is a great idea. This bypasses the off-chance of a manual error of missing a specific test and ensures that the most critical parts of your application run smoothly. You can also run those lengthy overnight tests unattended and wake up to detailed logs of why your test failed (or worked!).

3. The Results

How predictable are the results? Automation requires pre-defined inputs and outputs to produce pass and fail conditions otherwise they might lead to false outcomes.

If you’re in the exploratory stage of testing, and your tests are ad-hoc or require very specific domain knowledge, automating them is not the best idea.

4. Feature Importance

If a project is an important feature that could cause a disruption in the user experience if it fails, you should write an automated test suite. This way, you can safeguard against any human errors messing up your release.

Ideally, the tests should be run on a continuous cycle, so that relevant teams can be alerted as soon as possible.

5. Time-Reward Ratio

While automation frees up the tester’s time, organizations and individuals often overlook a crucial aspect of testing - the cost and time required to maintain the automated tests. If there are significant changes to the backend of your application, often writing and rewriting the code for automated tests is just as cumbersome as manual testing.

One interesting way to tackle this is for test engineers to automate just enough to understand which part of the program is failing. You can do this by automating the broader application tests so that if something does break, you know exactly where to look. Smart test execution, one of the top trends in the test automation space, does exactly this by identifying the specific tests that need to be executed.

6. Human Involvement

How complex is the test suite you’re trying to automate? If the test results need to be rechecked with a human eye or need to have actual user interaction, automating it probably won’t help a lot.

For example, user experience tests are best left unautomated because a testing software can never mimic human emotion while using a product. However, if you need visual confirmation on the test output, automated tests that take screenshots can be run, and later be manually verified.

7. Priority

When do you need the test results? If automating the tests helps you achieve a faster time to market the product, you should go ahead with it. However, don’t let writing and running automated tests be a bottleneck when you need the results immediately.

Also, you should remember that ‘tests’ are not the only thing that can be automated to make your application more efficient. Tasks such as manual data gathering, or setting up data inputs also make great candidates for automation. So, if there’s a large dataset but you’re running low on time, automating it might be your savior!

Frequently Automated Test Cases

1. Performance Tests (Load, Stress Tests)

Load tests are almost notorious for being the “overnight” tests. By definition, load tests require a lot of resources simply because they identify system lag and performance issues that arise when a company is scaling up. 

That’s why tools that do automated testing make a lot of sense - because they can effectively simulate users and resources at a fraction of the cost. I mean, try finding 1000 users to do bug testing on a product that hasn’t been released yet - yikes! 

While you definitely can’t hire 1000 QA specialists to do automation testing, a testing automation framework can set up dummy users and have them interact with your product just as a real user would. This will enable you to scale and avoid outages by identifying them early in the process. Your team can then look into the performance metrics and determine the exact cause for the loss of speed or outage.

Similarly, if you need to do cross-browser testing, automated testing can help you gather your application’s performance across multiple configurations with just a couple of steps.

Automate your performance tests to see where things are breaking, and whether your application can handle those breaking points.

2. Unit Tests

If you are developing the codebase of a large application, automating the unit tests will save you time. Automating testing for unit tests will help you find errors in real-time by giving you a continuous overview of whether the individual components are working or not. 

Automation is especially useful when you refactor your code since as long as the unit tests are green, you can safely assume that the behavior of individual units of code hasn’t changed. Moreover, the reports of these tests can be made available to the entire team instantly.

3. Regression Tests (Smoke, Sanity Tests)

Regression tests ensure that the application runs smoothly even if there are a lot of changes made. That means several application components need to be re-tested repeatedly. Due to this repetition, regression tests are an ideal candidate for test automation.

Automating regression tests will help you save manual resources and time, and scale much faster. Although regression tests are usually performed at the end of a software release, automating them also gives you an option to run them iteratively and continuously. This helps identify bugs in the program faster and creates rapid feedback loops, thus leading to a quicker resolution.

4. Functional Testing

Functional testing essentially verifies if an application is functioning the way it should on the front-end. Although some aspects of functional testing are manual, a lot of it should be automated to ensure bug-free product delivery. 

For example, end-to-end testing automation ensures key predefined user experience flows are running smoothly for products that have daily releases.

Using selenium to automate functional testing is a popular choice. You can even tweak the tests with a slightly different data set or user behavior to cover multiple use-cases.

Which Tests Should You Definitely Not Automate?

1. Exploratory Testing

Exploratory tests include a wider range of unscripted tests that are essential but are done on the fly. Typically, these tests require some domain knowledge and familiarity with the application to figure out unexpected behavior. Since they are not very well defined, they cannot be automated.

Once a tester discovers a defect through exploratory testing though, those test actions can be documented and automated for future builds. 

2. Usability Tests

As mentioned earlier, usability tests shouldn’t be automated because it’s hard to predict human behavior. This could range from wrong fonts, colors, or a UI that confuses people. You’ll only know these when you either perform beta or QA testing. Although there are tools that try to automate this, it is more effective (and less expensive) to get a human to look at it.

To Automate or Not To Automate?

Test automation is crucial for an efficient CI/CD pipeline. A lot of innovation has been going on in the test automation space such as parallel test execution, DevTestOps, IoT test automation, and so on. These automation frameworks have helped drastically reduce time to market for products and increase the quality of the build.  

Choosing the right test to automate is just the first step to achieve that for your organization, so test faster, fail faster, and fix faster!