Java Thread Dump Analysis

Thread dump analysis is a traditional approach followed to analyze the performance bottlenecks in Java-based applications. In the modern era, we have APM tools that provide various metrics and screens to drill down and identify performance issues, even at the code level. But for some of the performance issues or occasions, thread dump analysis still stands as the best way to identify the bottlenecks. 

When To Use a Thread Dump

To analyze any performance issue, it is good to take a series of thread dumps with a 1 to 2-second time gap. Taking 10-15 thread dumps each with 1-2 second intervals helps to analyze the threads that got stuck or execute the same code across thread dumps.

Thread dumps can be taken in the following scenarios:

1. The application is hung and not responding
2. The application takes time to respond
3. High CPU usage on the server where the application is running
4. Increase in active threads or total number of threads 

Thread dumps are also sometimes automatically generated by the application servers. For example, the WebSphere application server generates a thread dump during the OutOfMemoryError situation, which helps to analyze the various states of the thread at that moment.

For scenarios #1 and 2, focus should be given to the threads that are in blocked, parked/waiting, and runnable states. For scenario #3, the focus should be given to the threads which are in a runnable state. Some threads in infinite loop execution might cause high CPU usage and looking at runnable state might help to find that. For scenario #4, focus should be given to the threads that are in runnable and parked/wait thread states. In all the scenarios, ignore the threads that are in a parked or timed waiting state, which is waiting for the tasks/requests to be executed.

Analysis Tool Usage

Using a tool to analyze the thread dumps will give many statistics about the thread and its states. However, sometimes it may not reveal the real bottleneck in the system. It is always better to go through the thread dumps manually and do the analysis via tools like Notepad++. Tools like IBM Thread Dump Analyzer can be used if there are many thread dumps to analyze. It can be helpful to see the thread dumps in an organized view to speed up the analysis process. Though it won’t give many sophisticated statistics like the online analysis tools, it can help to visualize the thread dump better, provide a view to see the threads that got blocked due to another thread, and also help to compare the thread dumps.

While analyzing the thread dumps, it is important to know which application server for which the thread dump was taken as that will help to focus on analyzing the right threads. For example, if a thread dump was taken on the WebSphere application server, then the "Web Container" thread pool should be the first place to start the analysis as that is the entry point for the WebSphere application server which will start serving the request that comes to it.

Thread Dump Types

Generally, two kinds of threads will be there in the thread dump. One category of threads is related to the application and helps to execute the application code. Another category would be the threads which will do the operations, such as reading/writing from the network, heartbeat check, and various other operations including JVM internals like GC, etc. Depending upon the problem, the focus should be given to these two thread categories. Most of the time, application code might be the culprit for the performance bottleneck; hence, focus should be given more to the application threads.

Thread Pools

Thread dumps show the various thread pools available in the application. In the WebSphere application server, threads with the name "Web Container: <id>" belong to the WebSphere thread pool. Counting the number of such threads should be equivalent to the thread pool size defined. If it goes beyond, that indicates a thread leak in the thread pool. Different thread pool in the thread dumps needs to be verified for their size.

ForkJoinPool is another thread pool used by Java CompletableFuture to run the tasks asynchronously. If there are too many asynchronous tasks in this pool, then the size of the pool needs to be increased, or another pool with a bigger size needs to be created. Otherwise, this ForkJoinPool will become a bottleneck for asynchronous task execution.

If the application is creating a thread pool using the Java Executor framework, then the default name of "pool-<id1>-thread-<id2>" will be given for those threads. Here "id1" indicates the thread pool number and "id2" indicates the thread count in the thread pool. Sometimes if the developers create new thread pools every time without closing them via the Executor framework, then it will create different pools each time, and the number of threads will increase. It may not create a problem if the threads are not actively executing something, but it will result in an OutOfMemoryError where a new thread can’t be created by reaching the maximum number of thread creation. Looking at different thread pools and ensuring that all of them are within the defined/expected limit is always good while analyzing any thread dumps.

Application Methods 

Focusing on the application methods from the stack trace of the thread dump can help analyze the problem in the application code. If there are synchronized codes or blocks in the application, then the application threads will wait to acquire a lock on an object to enter specific code/block execution. This will be expensive, as only one thread is allowed to enter the code execution by making other threads wait. This kind of situation can be seen in the thread dump where threads wait to acquire the lock of an object. The code can be modified to avoid this synchronization if it is not needed.  

Conclusion

Thread dumps contain various details about the JVM, JVM arguments, memory, GC-related information, the hardware on which it is running, etc. It is always recommended to go through those details which might help the analysis.