Class Sharing in Eclipse OpenJ9: How to Improve Memory, Performance (Part 1)

Memory footprint and startup time are important performance metrics for a Java virtual machine (JVM). The memory footprint becomes especially important in the cloud environment since you pay for the memory that your application uses. In this tutorial, we will show you how to use the shared classes feature in Eclipse OpenJ9 to reduce the memory footprint and improve your JVM startup time.

In 2017, IBM open sourced the J9 JVM and contributed it to the Eclipse foundation, where it became the Eclipse OpenJ9 project. The J9 JVM has supported class sharing from system classes to application classes for over 10 years, beginning in Java 5.

In the OpenJ9 implementation, all systems, application classes, and ahead-of-time (AOT) compiled code can be stored in a dynamic class cache in shared memory. These shared classes feature are implemented on all platforms that OpenJ9 supports. The feature even supports integration with runtime bytecode modification, which we will discuss later in Part 2 of this article.

The shared classes feature is one that you don’t have to think about once it’s started, but it provides a powerful scope for reducing memory footprint and improving JVM startup time. For this reason, it is best suited to environments where more than one JVM is running similar code or where a JVM is regularly restarted.

In addition to the runtime class-sharing support in the JVM and its class loaders, there is also a public Helper API provided for integrating class sharing support into custom class loaders.

You can download the JDK with OpenJ9 from the Adopt OpenJDK project or pull it from the docker image if you'd like to follow along with the example.

How it Works

Let's start by exploring the technical details of how the shared classes feature operates.

Enabling Class Sharing

To enable class sharing, add -Xshareclasses[:name=] to an existing Java command line. When the JVM starts up, it looks for a shared cache of the name given (if no name is provided, it uses the current username). It either connects to an existing shared cache or creates a new one.

You can specify the shared cache size using the parameter -Xscmx[k|m|g]. This parameter only applies when a new shared cache is created. If this option is omitted, a platform-dependent default value is used. Note that there are operating system settings that limit the amount of shared memory you can allocate. For instance, SHMMAX on Linux is typically set to about 32MB. To learn more about the details of these settings, see the Shared Classes section of this user guide.

Shared Classes Cache

A shared classes cache consists of a shared memory of a fixed size that persists beyond the lifetime of the JVM or a system reboot unless a non-persistent shared cache is used. Any number of shared caches can exist on a system, and all are subject to operating system settings and restrictions.

No JVM owns the shared cache, and there is no master/slave JVM concept. Instead, any number of JVMs can read and write to the shared cache concurrently.

A shared cache cannot grow in size. When it becomes full, JVMs can still load classes from it, but it can no longer store any data into it. You can create a large shared classes cache up front while setting a soft maximum limit on how much shared cache space can be used. You can increase this limit when you want to store more data into the shared cache without shutting down the JVMs that are connected to it. Check out the OpenJ9 documentation for more details about the soft maximum limit.

In addition, there are several JVM utilities to manage actively shared caches. We will discuss these in the Shared Classes Utilities section below.

A shared cache is deleted when it is explicitly destroyed using a JVM command line.

How Are Classes Cached?

When a JVM loads a class, it first looks in the class loader cache to see if the class it needs is already present. If yes, it returns the class from the class loader cache. Otherwise, it loads the class from the filesystem and writes it into the cache as part of the defineClass() call. Therefore, a non-shared JVM has the following class loader lookup order:

1. Classloader cache
2. Parent
3. Filesystem

In contrast, a JVM running with the class sharing feature uses the following order:

1. Classloader cache
2. Parent
3. Shared classes cache
4. Filesystem

Classes are read from and written to the shared classes cache using the public Helper API. The Helper API is integrated into java.net.URLClassLoader (and jdk.internal.loader.BuiltinClassLoader in Java 9 and up). Therefore, any class loader that extends java.net.URLClassLoader gets class sharing support for free. For custom class loaders, OpenJ9 has provided Helper APIs so that class sharing can be implemented on custom class loaders.

What Is Cached?

A shared classes cache can contain bootstrap and application classes, metadata that describes the classes, and ahead-of-time (AOT) compiled code.

Inside the OpenJ9 implementation, Java classes are divided into two parts:

• a read-only part called a ROMClass, which contains all of the class's immutable data
• a RAMClass that contains mutable data, such as static class variables

A RAMClass points to data in its ROMClass, but these two are completely separated. So, it is quite safe for a ROMClass to be shared between JVMs and also between RAMClasses in the same JVM.

In the non-shared case, when the JVM loads a class, it creates the ROMClass and the RAMClass separately and stores them both in its local process memory. In the shared case, if the JVM finds a ROMClass in the shared classes cache, it only needs to create the RAMClass in its local memory; the RAMClass then references the shared ROMClass.

Because most of the class data is stored in the ROMClass, this is where the memory savings are made (see a more detailed discussion in the "Memory footprint " sections). JVM startup times are also significantly improved with a populated cache, because some of the work to define each cached class has already been done and the classes are loaded from memory, rather than from the filesystem. Startup time overhead to populate a new shared cache is not significant, as each class simply needs to be relocated into the shared cache as it is defined.

AOT compiled code is also stored into the shared cache. When the shared classes cache is enabled, the AOT compiler is automatically activated. AOT compilation allows the compilation of Java classes into native code for subsequent executions of the same program. The AOT compiler generates native code dynamically while an application runs and caches any generated AOT code in the shared classes cache. Usually, the execution of AOT compiled code is faster than interpreted bytecode but not as fast as JIT’ed code. Subsequent JVMs that execute the method can load and use the AOT code from the shared cache without incurring the performance decrease experienced with generating JIT-compiled code, resulting in a faster startup time. When creating a new shared cache, you can use options -Xscminaot and -Xscmaxaot to set the size of AOT space in the shared cache. If neither -Xscminaot nor - Xscmaxaot is used, the AOT code will be stored to the shared cache as long as there is free space available.

What Happens if a Class Changes on the Filesystem?

Because the share classes cache can persist indefinitely, filesystem updates that invalidate classes and AOT code in the shared cache may occur. If a class loader makes a request for a shared class, then the class returned should always be the same as the one that would have been loaded from the filesystem. This happens transparently when classes are loaded, so users can modify and update as many classes as they like during the lifetime of a shared classes cache, knowing that the correct classes are always loaded.

Pitfalls With Class Changes: Examples

Imagine a class C1 that is stored into the shared cache by a JVM. Then, when the JVM shuts down, C1 is changed and recompiled. When the JVM restarts, it should not load the cached version of C1.

Similarly, imagine a JVM that's running with a classpath of  /mystuff:/mystuff/myClasses.jar . It loads C2 from  myClasses.jar  into the shared cache. Then a different C2.class is added to  /myStuff  and another JVM starts up running the same application. It would be incorrect for the JVM to load the cached version of C2.

The JVM detects filesystem updates by storing timestamp values into the shared cache and comparing the cached values with actual values on each class load. If it detects that a JAR file has been updated, it has no idea which classes have been changed. Because of this, all classes, as well as AOT code from that JAR in the cache, are immediately marked as stale and cannot be loaded from the cache. When the classes from that JAR are loaded from the filesystem and re-added to the cache, only the ones that have changed are added in their entirety; those that haven't changed are effectively made not stale.

Classes cannot be purged from the shared classes cache, but the JVM attempts to make the most efficient use of the space it has. For example, the same class is never added twice, even if it is loaded from many different locations. So, if the same class C3 is loaded from  /A.jar ,  /B.jar , and  /C.jar  by three different JVMs, the class data is only added once. But, there are three pieces of metadata to describe the three locations from which it was loaded.

Shared Classes Utilities

There are several utilities that you can use to manage shared classes caches, all of which are sub-options to  -Xshareclasses  ( you can get a complete list of all sub-options via  java -Xshareclasses:help).

To demonstrate the use of these options, let's walk through some examples.

First, let's create two shared caches by running a Hello class with different cache names, as Listing 1 shows:

Listing 1. Creating two Shared Caches

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp . -Xshareclasses:name=Cache1 Hello
Hello
C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp . -Xshareclasses:name=Cache2 Hello
Hello

Running the  listAllCaches  sub-option lists all caches on a system and determines whether they are in use, as you can see in Listing 2:

Listing 2. Listing all Shared Caches

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:listAllCaches
Listing all caches in cacheDir C:\Users\Hang Shao\AppData\Local\javasharedresources\
Cache name              level         cache-type      feature         last detach time
Compatible shared caches
Cache1                  Java8 64-bit  persistent      cr              Mon Apr 23 15:48:12 2018
Cache2                  Java8 64-bit  persistent      cr              Mon Apr 23 15:49:46 2018

Running the  printStats  option prints summary statistics on the named cache, as Listing 3 shows. For a detailed description of the  printStats  option, see the user guide.

Listing 3. Summary Statistics for a Shared Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,printStats

Current statistics for cache "Cache1":

Cache created with:
        -Xnolinenumbers                      = false
        BCI Enabled                          = true
        Restrict Classpaths                  = false
        Feature                              = cr

Cache contains only classes with line numbers

base address                         = 0x000000001214C000
end address                          = 0x0000000013130000
allocation pointer                   = 0x0000000012297DB8

cache size                           = 16776608
softmx bytes                         = 16776608
free bytes                           = 13049592
ROMClass bytes                       = 1359288
AOT bytes                            = 72
Reserved space for AOT bytes         = -1
Maximum space for AOT bytes          = -1
JIT data bytes                       = 1056
Reserved space for JIT data bytes    = -1
Maximum space for JIT data bytes     = -1
Zip cache bytes                      = 902472
Data bytes                           = 114080
Metadata bytes                       = 18848
Metadata % used                      = 0%
Class debug area size                = 1331200
Class debug area used bytes          = 132152
Class debug area % used              = 9%

# ROMClasses                         = 461
# AOT Methods                        = 0
# Classpaths                         = 2
# URLs                               = 0
# Tokens                             = 0
# Zip caches                         = 5
# Stale classes                      = 0
% Stale classes                      = 0%

Cache is 22% full

Cache is accessible to current user = true

There are other  printStats  sub-options that can be used to print specific data in the shared cache. They can be found in  printStats=help. For example, you can check the classpath data via  printStats=classpath :

Listing 4. Listing the Classpath Contents of a Shared Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,printStats=classpath

Current statistics for cache "Cache1":

1: 0x000000001360E3FC CLASSPATH
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\bin\compressedrefs\jclSC180\vm.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\se-service.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\resources.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\jsse.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\charsets.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\jce.jar
        C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\tools.jar
1: 0x000000001360A144 CLASSPATH
        C:\OpenJ9
…
…

The shared caches are destroyed using the destroy  option, illustrated in Listing 5. Similarly, option  destroyAll destroys all shared caches that are not in use and that the user has permissions to destroy.

Listing 5. Destroying a Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,destroy
JVMSHRC806I Compressed references persistent shared cache "Cache1" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.

The expire option, illustrated in Listing 6, is a housekeeping option that you can add to the command line to automatically destroy caches to which nothing has been attached for a specified number of minutes. Listing 6 looks for caches that have not been used for a week (10,080 minutes) and destroys them before starting the JVM.

The reset  option always creates a new shared cache. If a cache with the same name exists, it is destroyed and a new one is created.

Listing 6. Destroying Caches That Haven't Been Used in a Week

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,expire=10080 Hello
Hello

Verbose Options

Verbose options provide useful feedback on what class sharing is doing. They are all sub-options to  -Xshareclasses. This section offers some examples of how to use those verbose options.

The verbose option, illustrated in Listing 7, gives concise status information on JVM startup and shutdown:

Listing 7. Getting JVM Status Information

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verbose Hello
[-Xshareclasses persistent cache enabled]
[-Xshareclasses verbose output enabled]
JVMSHRC236I Created shared classes persistent cache Cache1
JVMSHRC246I Attached shared classes persistent cache Cache1
JVMSHRC765I Memory page protection on runtime data, string read-write data and partially filled pages is successfully enabled
Hello
JVMSHRC168I Total shared class bytes read=11088. Total bytes stored=2416962
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0. Unstored AOT bytes due to the setting of -Xscmaxaot is 0. Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0. 

The  verboseIO  option prints a status line for every class load request to the shared cache. To understand  verboseIO  output, you should understand the class loader hierarchy. This can be clearly seen for classes that are loaded by any non-bootstrap class loader. In the output, each class loader is assigned a unique ID, but the bootstrap loader is always 0.

Note that it is normal for  verboseIO  to sometimes show classes being loaded from disk and stored in the cache, even if they are already cached. For example, the first class loaded from each JAR on the application classpath is always loaded from disk and stored, regardless of whether it exists in the cache or not. This is to confirm the JAR in the classpath does exist on the file system.

In Listing 8, the first section demonstrates the population of the cache and the second section shows the reading of the cached classes:

Listing 8. Using VerboseIO

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verboseIO Hello
[-Xshareclasses verbose I/O output enabled]
Failed to find class java/lang/Object in shared cache for class-loader id 0.
Stored class java/lang/Object in shared cache for class-loader id 0 with URL C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar (index 2).
Failed to find class java/lang/J9VMInternals in shared cache for class-loader id 0.
Stored class java/lang/J9VMInternals in shared cache for class-loader id 0 with URL C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar (index 2).
Failed to find class com/ibm/oti/vm/VM in shared cache for class-loader id 0.
Stored class com/ibm/oti/vm/VM in shared cache for class-loader id 0 with URL C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar (index 2).
Failed to find class java/lang/J9VMInternals$ClassInitializationLock in shared cache for class-loader id 0.
…
…
C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verboseIO Hello
[-Xshareclasses verbose I/O output enabled]
Found class java/lang/Object in shared cache for class-loader id 0.
Found class java/lang/J9VMInternals in shared cache for class-loader id 0.
Found class com/ibm/oti/vm/VM in shared cache for class-loader id 0.
Found class java/lang/J9VMInternals$ClassInitializationLock in shared cache for class-loader id 0.
…
…

The  verboseHelper  sub-option, illustrated in Listing 9, is an advanced option that gives status output from the Helper API. The  verboseHelper  sub-option helps developers using the Helper API to understand how it is being driven. More details on this output are described in the JVM diagnostics guide.

Listing 9. Status Output From the Helper API

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verboseHelper Hello
[-Xshareclasses Helper API verbose output enabled]
Info for SharedClassURLClasspathHelper id 1: Verbose output enabled for SharedClassURLClasspathHelper id 1
Info for SharedClassURLClasspathHelper id 1: Created SharedClassURLClasspathHelper with id 1
Info for SharedClassURLClasspathHelper id 2: Verbose output enabled for SharedClassURLClasspathHelper id 2
Info for SharedClassURLClasspathHelper id 2: Created SharedClassURLClasspathHelper with id 2
Info for SharedClassURLClasspathHelper id 1: There are no confirmed elements in the classpath. Returning null.
Info for SharedClassURLClasspathHelper id 2: There are no confirmed elements in the classpath. Returning null.
Info for SharedClassURLClasspathHelper id 2: setClasspath() updated classpath. No invalid URLs found
Info for SharedClassURLClasspathHelper id 2: Number of confirmed entries is now 1
Hello

The  verboseAOT  and  -Xjit:verbose  sub-option, illustrated in Listing 10, give you information on AOT loading and storing activities from/into the shared cache.

Listing 10. Verbose Information on AOT Loading and Storing

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=demo,verboseAOT -Xjit:verbose -cp shcdemo.jar ClassLoadStress
…
+ (AOT cold) java/nio/Bits.makeChar(BB)C @ 0x00000000540049E0-0x0000000054004ABF OrdinaryMethod - Q_SZ=2 Q_SZI=2 QW=6 j9m=0000000004A4B690 bcsz=12 GCR compThread=1 CpuLoad=298%(37%avg) JvmCpu=175%
Stored AOT code for ROMMethod 0x00000000123C2168 in shared cache.
…
+ (AOT load) java/lang/String.substring(II)Ljava/lang/String; @ 0x0000000054017728-0x00000000540179DD Q_SZ=0 Q_SZI=0 QW=1 j9m=00000000049D9DF0 bcsz=100 compThread=0
Found AOT code for ROMMethod 0x0000000012375700 in shared cache.
…

That's all for Part 1, be sure to tune in tomorrow when we'll discuss the next steps for class sharing in Eclipse OpenJ9.