How to Shift-Left Performance Testing Across Your Organization

Today's agile testing teams must build and execute test cases in the earlier stages of the software delivery lifecycle, a strategy often referred to as "shifting left." In this post, I'll give you everything you need to shift-left performance testing across your organization.

Every sprint is critical, and the decisions made moving forward are lightning fast. In order to facilitate the quick feedback process, testing teams must thoroughly validate their applications, end-to-end, in a very short timeframe.

To maximize this effort, testing teams can modernize their approach to testing, to get the best return on investment possible in the earliest stages possible. Focusing at the API layer of an application can help enable a scalable testing practice that can be efficiently executed as part of an accelerated delivery process, and is a practice that can be adopted and enabled at the earliest possible stages of development -- truly shifting left functional testing. But what about performance testing? How do we enable the shift left of nonfunctional testing? Here, we explore what this means and how to enable it in your organization.

What Is Shift Left Performance Testing?

Shifting performance testing left means enabling developers and testers to conduct performance testing in the early stages of development cycles. Traditionally, performance testing is a task performed at the end of development cycles because it requires a set of specialized tools and skills, i.e. expensive hardware in dedicated environments by trained performance testing engineers. A shift-left performance testing strategy instead allows smaller, ad hoc performance tests to be performed by testers against individual components as they are developed.

To accomplish this, teams need to begin creating performance tests along with unit and functional tests, when functionality is implemented and configure those performance tests to run automatically and report in a way that alerts to you to decreases in performance. To execute the tests automatically, performance test execution must be tightly integrated as a part of the CI/CD process, in which after each code check-in, performance tests are executed in local environments along with functional and unit tests.

This process enables organizations to understand the subtle impact of new components being added into the aggregate performance of their application, and ultimately enables the discovery of performance-related defects much earlier in the delivery lifecycle. From a company culture perspective, shifting left performance testing also means getting developers more involved. In most cases, development teams can make optimization enhancements within a day of discovering performance degradation, as opposed to waiting until the entire application is built.

What Does the Organization Have to Do to Make Shift Left Performance Happen?

First and foremost, you need well-established organizational buy-in. Addressing quality as a process and not as a reaction is essential to shifting performance testing left across the enterprise. The key players in this process are the product managers because performance testing and the associated development time come at a cost of implementation, which may slow down the development cycle. PM teams must understand why this process is taking place and understand that the value comes in reduced hot fixes and performance optimizations.

Next, defining SLAs at the component level in addition to the application level enables earlier-stage feedback, and helps developers understand the impact of code modification to the individual components they are developing. This granular performance testing makes it easier for stakeholders to learn where hotspots are occurring.

It's important to migrate as much of your testing practice away from UI centric testing into automated tests such as API and database tests. These types of testing practices, aside from being more maintainable and scalable, can be immediately leveraged in performance testing, can pinpoint the root cause of performance issues, and are highly resilient to change.

Finally, organizations must integrate performance testing into the build process so that basic smoke test performance tests get executed upon code check-in, and a full set of performance tests are run nightly. To do this, you need to consider hardware. Automated performance tests do require more computing resources than functional tests do, so development teams need to be prepared for that. Reviewing whether the existing performance infrastructure fits the shift-left approach or requires modification (i.e cloud agents) would also be one of the key considerations in transitioning to performance test automation.

Developer and Tester Roles in Shift-Left Performance

Developers OWN the performance of their applications. Developers must create applications that are ready for performance testing by using microservices, REST/SOAP APIs, and modular design architectures such that individual pieces can be load tested as they are developed.

Testers can align their test cases to key workflows in the application so that they can be leveraged in the performance testing process. Focusing on the API layers of the application make this more resilient to change and manageable. Both teams consume reports that have fallen outside of the SLAs for the application to identify problem areas based on recent code check-in to help them identify which components need to be optimized.

How Do You Integrate Tools to Enable Shift-Left Performance?

Selecting the right tools for a shift left performance testing process is important, but notas important as using them together in automated workflows. Early stage performance testing often happens in pockets, where individual savvy testers and developers devise techniques using a variety of open source and commercially-available tools, but this ultimately gets overlooked because it's not integrated as a part of the entire automated process.

Instead, testers should be using specialized commercial tools that enable them to create performance tests in an automated way. Developers can use similar tools to optimize their efforts or to create low-level scripts to drive automation and load. So what tools do you need?

The following tools simplify maintenance, can be managed in a centralized way, and provide an easy-to-use UI to comprehend results.

A Functional Testing Tool

Functional testing should already be a part of your continuous testing strategy. The tool you select for functional test automation should focus at both the API layer of the application (to simplify the test case execute action and maintenance) as well as the UI layer (for end-to-end and user experience testing). Functional testing tools are used to create baseline (re-use) execution paths, whether at the UI level or at the API level. These execution paths match up to user stories, so there will be a correlation between the performance test's result and the user story that is impacted.

A Performance Testing Tool

Specifically, you need a performance testing tool that can consume the functional testing artifacts and run them under load. These tools should have a variety of load control parameters such as number of virtual users or transactions over time. These tools should then report into a centralized dashboard for aggregating results.

A Service Virtualization Tool

Service virtualization tools address the missing components of monolithic applications in the early stages of shift-left performance testing. One of the primary challenges you will face in early stage performance testing is a lack of supporting infrastructure, by parallel development efforts or third-party components. By establishing the baseline of those dependent systems and modeling them in virtual services, you can create similar application baseline conditions to production and laser focus on individual component performance during your test.

A Continuous Integration Tool

Shift-left performance testing works best when it's an automated process. If automation is deployed, "performance testing" means simply the review/maintenance of the automated performance tests, reducing time to execute tests over the long run since the process is automated and not manual.

By aligning your performance testing strategy with your continuous testing strategy and integrating with tools such as Jenkins, Bamboo, Microsoft VSTS, etc, you can create a fully automated process. Your CI tooling should enable you to execute the performance tests as a function of code check in so that consistent performance tests can run nightly.

Additionally, your CI tooling should integrate with your reporting and analytics dashboard, and automatically publish results so you can quickly understand trending data.

A Centralized Dashboard for Aggregating Results

Speaking of your reporting and analytics dashboard... A centralized dashboard is important because it enables users to understand the incremental impact of component performance tests by displaying trending information by project, component, API etc.

Your centralized dashboard should provide the ability to automate performance tests, define SLAs that turn the performance tests into pass/fail indicators, and see historical trends. Additionally, the reporting dashboard should provide details that link performance tests to their initial requirements so the business can properly prioritize issues that arise, as well as the high-level pass/fail view and, at the same time, every little detail, so you can determine the causes of failures after they have been detected.

The shift-left approach adds developers as dashboard users (in addition to managers and testers), so the dashboard has to have the low-level details that the developers are looking for to effectively investigate and establish the causes for SLA failures or historic trends.

Summary

Consumers are burnt out with constant hot patches and performance optimization updates. They hunger for new features and functionality. Since performance testing is traditionally done at the end of the cycle, it inevitably impacts delivery deadlines, and as such, it is looked at through a negative lens. By federating out the performance testing process and enabling agile teams to adopt a "shift left," iterative approach to performance testing, issues can be identified early. Not only does this ensure that technology decisions made can be easily assessed for performance degradation, but ultimately provides a more performant product overall by optimizing each individual area and laser focusing on performance.