Getting Rid of Performance Testing Outliers With a New JMeter Plugin

Introduction

Whether we like it or not, outliers happen in our daily lives! For instance, it can be a mansion that sells in your neighbourhood for a price three to four times higher than the average house price. In the workforce, it can also be a few professionals earning very high salaries when compared to the rest of the employees. Or if you were to look at the arrival times of planes at your local airport, you would see that most planes land in a [0..10min] interval around the scheduled arrival time, but then, from time to time, a few planes may be late by 1hr or more...

But back to performance testing where we measure the response times of our applications.

My experience has shown that, here as well, we may have a number of data points of which the elapsed times are significantly higher than our normal observations. The reasons as to why some data points are much slower than others are usually independent from the application under test and due to environmental issues such as ad-hoc network issues, jobs or tasks kicking off on the server under test in the middle of your test such as IIS Application Pool Recycles, etc.

Why Should We Care About Outliers?

If we use the original results for our performance analysis and to generate the statistics, then we will get statistical values (average, median, max, 90th percentile, etc.) "skewed to the right". In essence, these values are not giving a real picture of the application just tested. Furthermore, they make it difficult to compare one version to the next as the statistics can be more or less distorted by the number of outliers and by their respective values.

For example, if you see an increase of the average value, is it because of a genuine degradation of the application, or is it because of the presence of some outliers in the data set?

If Outliers in the Far End Are an Issue, What Can We Do?

Performance Testers are using a couple of workarounds to address the issue of outliers. For example, they can repeat a performance test two or three times and check if the statistics are consistent across all tests. They can also look at the statistical values up to the 90% percentile and ignore the last 10% (i.e., 95 and 99 percentiles) as that is where the outliers lie most of the time. 

Despite their merits, these techniques can still be impacted by the outliers!

What if we could remove the outliers from our test results and then run our statistic tools only on a "cleansed" data set? Wouldn't that resolve the problem once and for all?

Well, for those of you using JMeter, there is a new plugin that does this very work of removing all outliers in a data set.

JMeter Right Tail Outlier Detector Plugin

This new plugin acts as a "low-pass filter": it will parse the results of your test and discard the data at the top end so that you can use the "cleansed" data for your statistics. The plugin does not filter the values in the lower end (i.e., in the left tail).

More Info About the Plugin

The plugin will find the upper bounds of the samples by itself using Tukey's technique and provides you with two options for the Tukey K parameter: you can either be very severe and remove all outliers by selecting K = 1.5, or be less stringent and remove only the far out values with K = 3.

The plugin will save the trimmed results and the outliers in two separate files. You can then use the trimmed results file for your performance analysis; you may also review the list of outliers to check for any potential issues.

Finally, the upper bounds and the number of omitted samplers are displayed in a table which you can export should you wish to document the truncation process in your test report. In this table, when the number of samples left after the truncation is less than 100, the plugin will mark them as "Small Group" as a precaution.

For further information on setup and usage of the plugin, please see the RightTail Outlier Detector plugin wiki.