Java NIO: OutOfMemoryError

Java NIO (New Input/Output) is a high-performance networking and file-handling API that facilitates you to do non-blocking IO. Non-blocking I/O provides the following advantages:

• Concurrency: NIO enables handling multiple connections simultaneously without blocking threads, leading to better concurrency.
• Asynchronous programming: Asynchronous programming allows the application to perform other tasks while waiting for I/O operations to complete, improving overall efficiency.
• Performance: Non-blocking I/O can manage more connections with fewer threads, reducing the resources required for handling concurrent requests.

One of our applications was leveraging this NIO; however, it suffered from frequent ‘java.lang.OutOfMemoryError: Direct buffer memory’ when we were running in Java 11. However, when we upgraded to Java 17 frequency of the occurrence of ‘java.lang.OutOfMemoryError: Direct buffer memory’ was reduced dramatically. In this post we would like to share our findings and resolution to fix this problem.

Simple Java NIO Client

To demonstrate our case, we have built a simple Spring Boot application that asynchronously uploads images. This application was leveraging Spring WebClient to connect with REST APIs. Spring WebClient underlyingly uses Java NIO technology to handle connections. Below is the source code of this application.

public void webHeavyClientCall(Integer id,String url, String imagePath) {

        // Create a WebClient instance

        WebClient webClient = WebClient.create();

        // Prepare the image file

        File imageFile = new File(imagePath);

        // Perform the POST request with the image as a part of the request body

        MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();

        body.add("file", new FileSystemResource(imageFile));

        System.out.println("Starting to post an image for Id"+id);
       
 webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData
(body))
                .retrieve().bodyToMono(String.class).subscribe(response -> {

                    System.out.println("Response Id"+id+ ":" + response);

        });

}

Java 11 NIO Memory Leak

We executed the above code in Java 11. After around 15 iterations, this simple application started to throw ‘java.lang.OutOfMemoryError: Direct buffer memory’. Below is the output printed in the console.

Starting to post an image for Id0

Starting to post an image for Id1

Starting to post an image for Id2

Starting to post an image for Id3

Starting to post an image for Id4

Starting to post an image for Id5

Starting to post an image for Id6

Starting to post an image for Id7

Starting to post an image for Id8

Starting to post an image for Id9

Starting to post an image for Id10

Starting to post an image for Id11

Starting to post an image for Id12

Starting to post an image for Id13

Starting to post an image for Id14

2023-12-06 17:21:46.730  WARN 13804 --- [tor-http-nio-12] io.netty.util.concurrent.DefaultPromise  : An 
exception was thrown by reactor.ipc.netty.FutureMono$FutureSubscription.operationComplete()

reactor.core.Exceptions$ErrorCallbackNotImplemented: 
io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown

Caused by: java.lang.OutOfMemoryError: Direct buffer memory

         at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]

         at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]

         at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) ~[na:na]

         at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[na:na]

         at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[na:na]

         at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[na:na]

         at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) ~[na:na]

         at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:418) ~[netty-
transport-4.1.23.Final.jar!/:4.1.23.Final]

        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-
transport-4.1.23.Final.jar!/:4.1.23.Final]

    ... 18 common frames omitted

Java 17 NIO Memory Optimization

We executed the same program in Java 17. However, to run this program in Java 17, we had to make some minor modifications. Below is the revised code that runs on Java 17 that simulates the same behavior as above.

public void webHeavyClientCall(Integer id,String url, String imagePath) {

// Create a WebClient instance

WebClient webClient = WebClient.create();

// Prepare the image file

File imageFile = new File(imagePath);

// Perform the POST request with the image as a part of the request body

MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();

body.add("file", new FileSystemResource(imageFile));

System.out.println("Starting to post an image for Id"+id);

webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData
(body))                                                                 

   .retrieve().bodyToMono(String.class).subscribe(response -> {  
           System.out.println("Response Id"+id+ ":" + response);
      });

   }

}

There was an improvement in memory usage after the upgrade. Java 17 was able to handle at least twice as many NIO connections compared to Java 11. Below is the output from the console. You could see the application was able to iterate until 50 connections before it struck with ‘java.lang.OutOfMemoryError’. On the other hand, Java 11 failed  ‘java.lang.OutOfMemoryError’ right after 15 connections. 

Starting to post an image for Id38

Starting to post an image for Id39

Starting to post an image for Id40

Starting to post an image for Id41

Starting to post an image for Id42

Starting to post an image for Id43

Starting to post an image for Id44

Starting to post an image for Id45

Starting to post an image for Id46

Starting to post an image for Id47

Starting to post an image for Id48

Starting to post an image for Id49

2023-12-12 14:49:38.421  WARN 59559 --- [ctor-http-nio-4] r.netty.http.client.HttpClientConnect           : 
[bfc8b2c8, L:/127.0.0.1:57435 ! R:localhost/127.0.0.1:8090] The connection observed an error

reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Cannot reserve 4096 bytes 
of direct buffer memory (allocated: 202956, limit: 204800)

Caused by: java.lang.OutOfMemoryError: Cannot reserve 4096 bytes of direct buffer memory (allocated: 
202956, limit: 204800)

    at java.base/java.nio.Bits.reserveMemory(Bits.java:178) ~[na:na]

    at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[na:na]

    at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[na:na]

    at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-
buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer
-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~
[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:634) ~[netty-
buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:397) ~
[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~
[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~
[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) ~[netty-
buffer-4.1.73.Final.jar!/:4.1.73.Final]

    at org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71)
 ~[spring-core-5.3.15.jar!/:5.3.15]

    at org.springframework.core.io.buffer.DataBufferUtils$ReadCompletionHandler.request(DataBufferUtils.java:945
) ~[spring-core-5.3.15.jar!/:5.3.15]

Troubleshooting ‘OutOfMemoryError: Direct buffer memory’

In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before it surfaces in the production environment. Once it predicts an outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them, and instantly generates a root cause analysis report. Artifacts it captures include Garbage Collection log, Thread Dump, Heap Substitute, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters, and disk usage…

You can register here and start using the free tier of this tool.

The yCrash server analyzed the sample application and provided clear indications of issues with recommendations. Below is the incident summary report that yCrash generated for the SpringBoot WebClient application. You can notice yCrash clearly points out the error with necessary recommendations to remediate the problem.

Fig 1: Incident Summary Report from yCrash

Garbage Collection Analysis Report

yCrash’s Garbage Collection (GC) analysis report revealed that Full GCs were consecutively running (see screenshot below). When GC runs, the entire application pauses, and no transactions will be processed. The entire application would become unresponsive. We observed the unresponsiveness behavior before the SpringBoot WebClient application crashed with OutOfMemoryError.

Fig 2: yCrash report pointing to our Consecutive Full GC problem

Logs Analysis Reporting Outofmemoryerror: Direct Buffer Memory

yCrash’s application log analysis report revealed that the application was suffering from ‘java.lang.OutOfMemoryError: Direct buffer memory’ (see the screenshot below), which caused the application to crash.

Fig 3: yCrash log report pointing java.lang.OutOfMemoryError: Direct buffer memory

Why Is the Java NIO Application Suffering From Outofmemoryerror?

Java NIO objects are stored in the ‘Direct Buffer Memory’ region of JVM’s native memory. (Note: There are different memory regions in JVM. To learn about them, you may watch this video clip). When we executed the above two programs, we had set the Direct Buffer Memory size as 200k (i.e., XX:MaxDirectMemorySize=200k). Under 200k Direct Buffer Memory allocation, Java 11 was able to do only 15 iterations, whereas Java 17 was able to go to 50 iterations. It clearly indicates the optimization the JDK team has done in the Java 17 version. 

Fig 4: WebClient Objects stored in the Direct Memory Region of the Native Region

On Java 11 or Below Versions Increase: Xx:Maxdirectmemorysize

Thus, if your application is leveraging Java NIO and running on Java 11 or below versions and experiencing ‘java.lang.OutOfMemoryError: Direct buffer memory’, there are a couple of solutions in front of you:

1. Consider allocating higher Direct Buffer Memory Size. 
2. Consider upgrading to Java 17 or higher version

Since upgrading Java 17 requires more dependencies, we increased the direct memory size to a higher value using the JVM argument -XX:MaxDirectMemorySize=1000k. After making this change, the Java 11 version of the application was able to run successfully without any errors. 

Conclusion

In this post, we discussed ‘java.lang.OutOfMemoryError: Direct buffer memory’ caused by Java NIO in Java 11 and potential solutions to fix the same. We hope you find it helpful.