How to Run JMeter in a Continuous Integration Environment With Bamboo

We all know that periodic load tests are vital if you want to avoid unpleasant surprises and regressions. However, in the fast-paced agile age, it’s becoming increasingly important to integrate load testing into your continuous integration process. Including load testing in your application build process will allow you to: 

1. Catch performance regressions early
2. Understand the cost of adding new features on the performance
3. Know the side effects of dependencies when you add or upgrade
4. Feel confident about your application performance

About Atlassian Bamboo

Atlassian Bamboo is one of the most popular continuous integration, release and build servers around today, trusted by more than 50,000 software development companies. Due to its widespread popularity, today I’m going to focus on integrating load testing into your Continuous Integration process with Bamboo.

So, if your application build uses Bamboo, keep on reading! 

The Challenges of Using Bamboo With JMeter

The biggest problem here is that Bamboo doesn’t have built-in support of JMeter. There is a workaround. This involves:

1. Adding a build step to trigger a JMeter test
2. Fetching the results from the Bamboo server of the build agent
3. Analyzing them to decide whether the build is good to go or not

This is a feasible approach - but far from ideal! It’s much better to automate the analysis step so you can get immediate notifications whenever a threshold has been exceeded or a SLA violated.

Therefore, if you’re using JMeter with Bamboo, it’s crucial to install the Bamboo JMeter Aggregator plugin. It’s not officially supported but it was originally authored by James Roper (an Atlassian employee) and it’s up-to-date with the latest Bamboo Version.

How to Install the Bamboo JMeter Aggregator Plugin in 6 Steps

1. Download the latest version of the JMeter Aggregator plugin from the Atlassian Marketplace.
2. Stop the Bamboo server.
3. Copy the atlassian-bamboo-jmeter-aggregator-x.x.x.jar to the /atlassian-bamboo/WEB-INF/lib folder of your Bamboo installation.
4. Start the Bamboo server.
5. Validate that the plugin has been successfully installed with the Bamboo Plugin manager. Just click the “Administration” dropdown and select “Add-ons”.
   

How to Run a JMeter Test With Bamboo

It’s pretty easy to add a JMeter test run to Bamboo.

Like any efficient continuous integration system, Bamboo executes tasks and checks their status (Note: a task is a small discrete unit of work which is usually run in Bamboo’s job context). Bamboo supports various task types, from single commands or shell scripts to much more complex activities like performing a Visual Studio or Xcode build. 

The most popular way to launch JMeter is through the command-line non-GUI mode - and this is the approach I’ll be taking today.

First of all, let’s add an empty Command Line task to run a JMeter test in your Bamboo project.

1. Click the “Edit” icon (the one with the pencil) on the right-hand side of your project
2. Click the Job that you want to add JMeter tests to
3. Click the “Add Task” button
4. Choose a “Script” from the list (or enter the word “script” in the Search input)

Before we continue with populating the script body, it’s important to remember the following:   

1. The test result file should be located in the working directory of the Bamboo build agent. So you need to “tell” JMeter to store the results in Bamboo’s Job working folder.
2. The test result file needs to be in an XML format. By default, JMeter stores its results in a CSV format so you’ll need to change it.

Therefore, put the following lines into the “Script body” area: 

/bin/bash -c "/tmp/jmeter/bin/jmeter.sh \
-Jjmeter.save.saveservice.output_format=xml \
-n -t /tmp/tests/example.jmx \
-l ${bamboo.build.working.directory}/example.jtl"

Code Breakdown 

1. /bin/bash = the system command interpreter
2. /tmp/jmeter/bin/jmeter.sh = the path to the JMeter executable script
3. -Jjmeter.save.saveservice.output_format=xml = the property which changes the JMeter result file format to XML. See the Apache JMeter Properties Customization Guide for more information on working with JMeter properties
4. /tmp/tests/example.jmx - the path to the JMeter test script
5. ${bamboo.build.working.directory} - the Bamboo variable which stands for the current build working directory

So your task configuration should look like the image below (just make sure you change the settings above to match your JMeter and test script locations) 

At this stage, it’s a good idea to run the first build to ensure that it passes successfully. Ideally, you should see something like this:

Now let’s configure the Bamboo JMeter Aggregator plugin so it can track performance trends and conditionally fail the build based on key metrics and KPIs. The plugin configuration is performed on the “Miscellaneous” tab of the Bamboo Job which contains the JMeter tests. Here  you can specify the location to look for the JMeter result files and set arbitrary assertions:

I recommend setting the “Build Log File” value to: **/*.jtl. When combined with the ${bamboo.build.working.directory} variable, it will configure Bamboo to store the JMeter test results along with the build working folder and you’ll be able to monitor performance trends across consecutive builds of your application.

 Once you’re all set, run your build plan a few more times to get initial statistics. The results of the Bamboo JMeter Aggregator plugin can be found in the “Load Test Reports” tab on your build plan dashboard.

If you click the “Load Test Reports” link, you’ll get to the JMeter Load Test Reports page. Here  you’ll be able to see graphs on metrics and KPIs of your choice and monitor how these values change from build to build.

Now you should be ready to integrate JMeter load tests to your Continuous Integration process with Bamboo.

If you’re using BlazeMeter, you can just install the BlazeMeter Bamboo Plugin. This will allow you to kick off tests on BlazeMeter’s distributed infrastructure and report the results directly back to Bamboo.