Java Microbenchmark Harness (JMH)
A quick hands-on lesson to learn about Java Microbenchmark Harness (JMH). The article helps you get started and configure JMH project.
Join the DZone community and get the full member experience.
Join For FreeIntroduction
In my previous article, we established that microbenchmarking is hard with jvm
. It is not enough to surround the code in a loop with System.out.println()
and gather the time measurements. While benchmarking, a developer should consider warm-up cycles, JIT compilations, JVM optimizations, avoiding usual pitfalls, and even more.
Thankfully, OpenJDK has a great tool Java Microbenchmark Harness (JMH) that can help us generated benchmarking stats. In this article, we will discuss how JMH can help us avoid the pitfalls that we have discussed earlier.
Getting Started With JMH
A quick way to start with JMH is to use the Maven archetype. The command below will generate a new Java project benchmark
. The project will have com/gaurav/MyBenchmark.java
class and pom.xml
. The Maven pom.xml
includes all the required dependencies to support JMH.
mvn archetype:generate -DarchetypeGroupId=org.openjdk.jmh -DarchetypeArtifactId=jmh-java-benchmark-archetype -DinteractiveMode=false -DgroupId=com.gaurav -DartifactId=benchmark -Dversion=1.0
Good Benchmarks With JMH
Let us discuss how JMH can help us to write better microbenchmarks.
- JMH by default makes several warm-up cycles before collecting the stats. Thus, it makes sure that the results are not completely random and
jvm
has performed optimizations. @benchmark
runs iteration over the code, and then collects the average. The more runs you make through the code, the better stats you will collect.- Use
Blackhole
class of JMH can avoid deal code elimination byjvm
. If you pass the calculated results toblackhole.consume()
, it would trick thejvm
.jvm
will never drop the code, thinking thatconsume()
method uses the result.
Writing First Benchmark
Maven has already provided with us a template in MyBenchmark
class to fill in. I am going to utilize the same class.
package com.gaurav;
import org.openjdk.jmh.annotations.Benchmark;
public class MyBenchmark {
public void testMethod() {
// This is a demo/sample template for building your JMH benchmarks. Edit as needed.
// Put your benchmark code here.
}
}
I would like to keep my first benchmark simple. Let us start by iterating over all the elements of a list and sum them up using a conventional for
loop. As discussed, I will use Blackhole
to fool the compiler and return the result. Here, I am asking JMH to calculate the average time, using @BenchmarkMode
, it takes to run the testMethod()
.
x
Mode.AverageTime) (
public static double testMethod(Blackhole blackhole) {
double sum = 0;
for(int i=0; i<list.size(); i++) {
sum += list.get(i);
}
blackhole.consume(sum);
return sum;
}
Compiling the JMH Project
You can compile and build the project like any other Maven project, using the below Maven command:
mvn clean install
The command will create a fully executable jar
file under benchmark/target
the directory. Please note that Maven will always generate a jar
file named benchmarks.jar
, regardless of the project name.
Next step is to execute the jar
using the below command:
java -jar target/benchmarks.jar
It produced the below result for me. It means that test operation is taking approx. 0.053 seconds on the current hardware.
xxxxxxxxxx
# Run progress: 80.00% complete, ETA 00:01:41
# Fork: 5 of 5
# Warmup Iteration 1: 0.052 s/op
# Warmup Iteration 2: 0.051 s/op
# Warmup Iteration 3: 0.053 s/op
# Warmup Iteration 4: 0.056 s/op
# Warmup Iteration 5: 0.055 s/op
Iteration 1: 0.054 s/op
Iteration 2: 0.053 s/op
Iteration 3: 0.053 s/op
Iteration 4: 0.054 s/op
Iteration 5: 0.059 s/op
Result "com.example.MyBenchmark.testMethod":
0.053 ±(99.9%) 0.002 s/op [Average]
(min, avg, max) = (0.052, 0.053, 0.061), stdev = 0.002
CI (99.9%): [0.051, 0.055] (assumes normal distribution)
# Run complete. Total time: 00:08:27
Benchmark Modes
In the previous example, I used @BenchmarkMode(Mode.AverageTime)
. If you try to decompile the JMH jar, you will find enum Mode
with the below values:
Modes | |
---|---|
Throughput("thrpt", "Throughput, ops/time") |
It will calculate the number of times your method can be executed with in a second |
AverageTime("avgt", "Average time, time/op") |
It will calculate the average time in seconds to execute the test method |
SampleTime("sample", "Sampling time") |
It randomly samples the time spent in the test method calls |
SingleShotTime("ss", "Single shot invocation time") |
It works on single invocation of the method and is useful in calculating cold performance |
All("all", "All benchmark modes") |
Calculates all the above |
The default Mode is Throughput
.
Time Measurement
It is evident from the console output above that calculations are in seconds. But, JMH allows you to configure the time units, using @OutputTimeUnit
annotation. The @OutputTimeUnit
accepts java.util.concurrent.TimeUnit
, as shown below:
@OutputTimeUnit(TimeUnit.SECONDS)
The TimeUnit
enum has the following values:
NANOSECONDS
MICROSECONDS
MILLISECONDS
SECONDS
MINUTES
HOURS
DAYS
The default TimeUnit
is SECONDS
.
Configure Fork, Warmup, and Iterations
The benchmark is currently executing 5 times, with 5 warmup iterations and 5 measurement iterations. JMH even allows to configure these values using @Fork
, @Warmup
and @Measurement
annotations. The code snippet below would execute the test method twice, with a couple of warmup iterations and 3 measurement iterations.
xxxxxxxxxx
value = 2) (
iterations = 2) (
iterations = 3) (
@Warmup
and @Measurement
annotations also accepts parameters:
batchSize
- configures the number of test method calls to be performed per operation.time
- time spent for each iteration.
Practice
You can play around to compare execution times of different for
loops i.e. a conventional for
loop, a forEach
loop and an stream
iterator. Something like:
xxxxxxxxxx
private static final List<Integer> list = IntStream.rangeClosed(1, Integer.MAX_VALUE/100)
.boxed().collect(Collectors.toList());
Mode.AverageTime) (
public static double conventionalLoop(Blackhole blackhole) {
double sum = 0;
for(int i=0; i<list.size(); i++) {
sum += list.get(i);
}
blackhole.consume(sum);
return sum;
}
Mode.AverageTime) (
public static double enhancedForLoop(Blackhole blackhole) throws InterruptedException {
double sum = 0;
for (int integer : list) {
sum += integer;
}
blackhole.consume(sum);
return sum;
}
Mode.AverageTime) (
public static double streamMap(Blackhole blackhole) {
double sum = list.stream().mapToDouble(Integer::doubleValue).sum();
blackhole.consume(sum);
return sum;
}
Conclusion
In this article, we have gone through a hands-on example of creating a JMH project. We have seen how can we configure our JMH project to suit our needs. You can refer to JMH Github Samples for more in-depth examples.
We have seen that JMH is a jvm
tool. In the next article, we will try to explore if it can help us with other jvm
based languages.
Published at DZone with permission of Gaurav Gaur. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments