How to Get Metrics From a Java Application Inside a Docker Container Using Telegraf

Telegraf, which is part of the TICK Stack, is a plugin-driven server agent for collecting and reporting metrics. It has plugins or integrations to source a variety of metrics directly from the system it’s running on, third-party APIs or even other services like StatsD and Kafka consumers. It also has output plugins to send metrics to a variety of other datastores, services, and message queues (not restricted then to InfluxDB only). In this article I am going to describe the setup to pull metrics from Java applications hosted inside a Docker container. The release of Telegraf I am referring to here is the 1.3 (the latest at the time this article is written). I am also assuming you have the Telegraf agent already installed in the node where the Docker container is going to be shipped.

Telegraf Configuration to Pull Metrics From a Docker Container

Telegraf comes with a plugin to pull metrics from Docker containers. The following is the typical setup you have to specify in your Telegraf agent configuration file (which is /etc/telegraf/telegraf.conf for Linux hosts):

[[inputs.docker]]
  ## Docker Endpoint
  endpoint = "unix:///var/run/docker.sock"
  ## Only collect metrics for these containers, collect all if empty
  container_names = []
  ## Timeout for docker list, info, and stats commands
  timeout = "10s"

  ## Whether to report for each container per-device blkio (8:0, 8:1...) and
  ## network (eth0, eth1, ...) stats or not
  perdevice = true
  ## Whether to report for each container total blkio and network stats or not
  total = false
  ## docker labels to include and exclude as tags.  Globs accepted.
  ## Note that an empty array for both will include all labels as tags
  docker_label_include = []
  docker_label_exclude = []

Please have a look at the Github readme file for this plugin for the full list of metrics you can pull from any Docker container. Those metrics give you a full picture of what's going on for the containers, but no specific info about the Java application running inside. For this reason, you have to configure a second Telegraf plugin.

Telegraf Configuration for Jolokia

In order to pull metrics from any application running on a JVM, Telegraf comes with a specific plugin which uses the Jolokia agent. Jolokia is an agent based approach for remote JMX access. The communication between client and agent goes over HTTP (either GET or POST), where the request and response payload is represented in JSON. The Jolokia plugin collects JVM metrics exposed as MBean's attributes through a Jolokia REST endpoint. In the Telegraf configuration, you have to setup the Jolokia endpoint details for the input plugin:

# Read JMX metrics through Jolokia
[[inputs.jolokia]]
  ## This is the context root used to compose the jolokia url
  ## NOTE that Jolokia requires a trailing slash at the end of the context root
  context = "/jolokia/"

  ## List of servers exposing jolokia read service
  [[inputs.jolokia.servers]]
    name = "as-server-01"
    host = "127.0.0.1"
    port = "8778"

and the metrics you need to collect. Here's an example:

## List of metrics collected on above servers
  ## Each metric consists in a name, a jmx path and either
  ## a pass or drop slice attribute.
  ## This collects all heap memory usage metrics.
  [[inputs.jolokia.metrics]]
    name = "heap_memory_usage"
    mbean  = "java.lang:type=Memory"
    attribute = "HeapMemoryUsage"

  ## This collects thread counts metrics.
  [[inputs.jolokia.metrics]]
    name = "thread_count"
    mbean  = "java.lang:type=Threading"
    attribute = "TotalStartedThreadCount,ThreadCount,DaemonThreadCount,PeakThreadCount"

  ## This collects number of class loaded/unloaded counts metrics.
  [[inputs.jolokia.metrics]]
    name = "class_count"
    mbean  = "java.lang:type=ClassLoading"
    attribute = "LoadedClassCount,UnloadedClassCount,TotalLoadedClassCount"

At this stage, the Telegraf configuration is ready. You need to restart the agent to make it effective. One thing is still missing for this process: a running Jolokia agent listening at the port specified in the plugin configuration and attached to the Java application process.

Jolokia Agent Configuration for the Java Application Inside the Docker Container

The Jolokia agent can be shipped in the same Docker container for the Java application to be monitored. You need to add some instructions to the Dockerfile for your Java application.

You need to add the Jolokia JVM agent to the image:

ADD /<local_path>/jolokia-jvm-<version>-agent.jar /

Then expose the agent port:

 EXPOSE 8778 

and finally provide the following single startup option to install the agent when the Java process starts:

CMD ["<other_options", "-javaagent:jolokia-jvm-<version>-agent.jar=port=8778,host=0.0.0.0"]

Build your image as usual and then ship and start a container in the destination machine where you started the Telegraf agent. When starting the container you have to map the Jolokia agent listening port along with any other port used by the Java application:

 sudo docker run -d -p 8778:8778 ... 

Assuming your Telegraf agent is configured to send data to InfluxDB, you should see in the database a table named jolokia, which contains the following fields keys:

name: jolokia

fieldKey                             fieldType

--------                             ---------

class_count_LoadedClassCount         float

class_count_TotalLoadedClassCount    float

class_count_UnloadedClassCount       float

heap_memory_usage_committed          float

heap_memory_usage_init               float

heap_memory_usage_max                float

heap_memory_usage_used               float

thread_count_DaemonThreadCount       float

thread_count_PeakThreadCount         float

thread_count_ThreadCount             float

thread_count_TotalStartedThreadCount float

Conclusion

Once understood how to set up the proper configurations, the process to implement the monitoring of Java applications inside Docker containers using Telegraf is pretty straightforward.