Getting Started With Prometheus Workshop: Instrumenting Applications

Are you looking to get away from proprietary instrumentation? Are you interested in open-source observability but lack the knowledge to just dive right in?

This workshop is for you, designed to expand your knowledge and understanding of open-source observability tooling that is available to you today.

Dive right into a free, online, self-paced, hands-on workshop introducing you to Prometheus. Prometheus is an open-source systems monitoring and alerting tool kit that enables you to hit the ground running with discovering, collecting, and querying your observability today. Over the course of this workshop, you will learn what Prometheus is, what it is not, install it, start collecting metrics, and learn all the things you need to know to become effective at running Prometheus in your observability stack.

Previously, I shared an introduction to Prometheus, installing Prometheus, an introduction to the query language, exploring basic queries, using advanced queries, relabeling metrics in Prometheus, and discovering service targets as free online labs. In this article, you'll learn all about instrumenting your applications using Prometheus client libraries.

Your learning path takes you into the wonderful world of instrumenting applications in Prometheus, where you learn all about client libraries for the languages you code in. Note this article is only a short summary, so please see the complete lab found online to work through it in its entirety yourself.

The following is a short overview of what is in this specific lab of the workshop. Each lab starts with a goal. In this case, it is as follows:

• This lab introduces client libraries and shows you how to use them to add Prometheus metrics to applications and services. You'll get hands-on and instrument a sample application to start collecting metrics.

You start in this lab reviewing how Prometheus metrics collection works, exploring client library architectures, and reviewing the four basic metrics types (counters, gauges, histograms, and summaries).

If you've never collected any type of metrics data before, you're given two systems to help you get started. One is known as the USE method and is known for systems or infrastructure metrics. The other is the RED method, which targets more applications and services.

The introduction finishes with a few best practices around naming your metrics and warnings on how to avoid cardinality bombs.

Instrumentation in Java

For the rest of this lab, you'll be working on exercises that walk you through instrumenting a simple Java application using the Prometheus Java client library. No previous Java experience is required, but there are assumptions made that you have minimum versions of Java and Maven installed.

You are provided with a Java project that you can easily download and work from using your favorite IDE. If you don't work in an IDE, use any editor you like as the coding you'll be doing is possible with just cutting and pasting from the lab slides. To install the project locally:

1. Download and unzip the Prometheus Java Metrics Demo from GitLab.
2. Unzip the prometheus-java-metrics-demo-main.zip file in your workshop directory.
3. Open the project in your favorite IDE (examples shown in the lab use VSCode).

You'll be building and running the Java application, which is a basic empty service where comments are used to show where your application code would go. Before that block, you see that the instrumentation has been provided for all four of the basic metric types.

Once you have built and started the Java JAR file, the output will show you that the setup has been successful:

$ cd prometheus-java-metrics-demo-main/

$ mvn clean install  (watch for BUILD SUCCESS)

$ java -jar target/java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar 

Java example metrics setup successful... 

Java example service started...

Now it's just waiting for you to validate the endpoint at localhost:7777/metrics, which displays the metrics:

# HELP java_app_s is a summary metric (request size in bytes)
# TYPE java_app_s summary
java_app_s{quantile="0.5",} 2.679717814859738
java_app_s{quantile="0.9",} 4.566657867333372
java_app_s{quantile="0.99",} 4.927313848318692
java_app_s_count 512.0
java_app_s_sum 1343.9017287309503
# HELP java_app_h is a histogram metric
# TYPE java_app_h histogram
java_app_h_bucket{le="0.005",} 1.0
java_app_h_bucket{le="0.01",} 1.0
...
java_app_h_bucket{le="10.0",} 512.0
java_app_h_bucket{le="+Inf",} 512.0
java_app_h_count 512.0
java_app_h_sum 1291.5300871683055
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 512.0
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g 5.5811320747117765

While the metrics are exposed in this example on localhost:7777, they will not be scraped by Prometheus until you have updated its configuration to add this new endpoint. Let's update our workshop-prometheus.yml file to add the Java application job as shown along with comments for clarity (this is the minimum needed, with a few custom labels for fun):

# workshop config
global:
  scrape_interval: 5s

scrape_configs:
  # Scraping Prometheus.
  - job_name: "prometheus"
  static_configs:
    - targets: ["localhost:9090"]

  # Scraping java metrics. 
  - job_name: "java_app" 
  static_configs: 
    - targets: ["localhost:7777"] 
       labels: job: "java_app" 
       env: "workshop-lab8"

Start the Prometheus instance (for container Prometheus, see the workshop for details) and then watch the console where you started the Java application as it will report each time a new scrape is done by logging Handled :/metrics:

$./prometheus --config.file=support/workshop-prometheus.yml 

===========Java application log=============== 

Java example metrics setup successful... 

Java example service started... 

Handled :/metrics 
Handled :/metrics 
Handled :/metrics 
Handled :/metrics
 ...

You can validate that the Java metrics you just instrumented in your application are available in the Prometheus console localhost:9090 as shown. Feel free to query and explore:

Next up, you'll be creating your own Java metrics application starting with the minimal setup needed to get your Java application running and exposing the path /metrics. Instead of coding it all by hand, you're given a starting point class file found in the project.

Instrumenting Basic Metrics

Java was chosen as the language due to many developers using this in enterprises, and exposing you to the Prometheus client library usage for a common developer language is a good baseline. The rest of the lab walks through multiple exercises where you start from a blank application template that's provided and code step-by-step the four basic metrics types.

You're also walked through a custom build and run of the application each step of the way, with the following process used for each metric type as you work from implementation, to build, to validating that it works:

• Add the necessary Java client library import statements for the metric type you are adding.
• Add the code to construct the metric type you are defining.
• Initialize the new metric in a thread with basic numerical values (often random numbers).
• Rebuilding the basic Java application to create an updated JAR file you can run.
• Starting the application and validating the new metric is available on localhost:9999/metrics.

Once all four of the basic metric types have been implemented and tested, you learn to update your Prometheus configuration to pick up your application:

# workshop config
global:
  scrape_interval: 5s

scrape_configs:
  # Scraping Prometheus.
  - job_name: "prometheus"
  static_configs:
    - targets: ["localhost:9090"]

  # Scraping java metrics.
  - job_name: "java_app"
  static_configs:
    - targets: ["localhost:9999"] 
       labels: job: "java_app" 
       env: "workshop-lab8"

Finally, you verify that you are collecting your Java-instrumented application data by checking through the Prometheus query console:

Miss Previous Labs?

This is one lab in the more extensive free online workshop. Feel free to start from the very beginning of this workshop here if you missed anything previously.

You can always proceed at your own pace and return any time you like as you work your way through this workshop. Just stop and later restart Perses to pick up where you left off.

Coming up Next

I'll be taking you through the final lab in this workshop where you'll learn all about metrics monitoring at scale and understanding some of the pain points with Prometheus that arise as you start to scale out your observability architecture and start caring more about reliability..

Stay tuned for more hands-on material to help you with your cloud-native observability journey.