Getting Started With Prometheus Workshop: Service Discovery

You start in this lab exploring the service discovery architecture Prometheus provides and how it is supporting all manner of automated discovery of dynamically scaling targets in your infrastructure, the basic definitions of what service discovery needs to achieve, knowing what targets should exist, knowing how to pull metrics from those targets, and how to use the associated target metadata. You then dive into the two options for installing the lab demo environments, either using source projects or in open-source containers for the exercises later in this lab.

The Demo Environment

Whether you install it using source projects or containers, you'll be setting up the following architecture to support your service discovery exercises using the services demo to ensure your local infrastructure contains the following:

• Production 1 running at http://localhost:11111
• Production 2 running at http://localhost:22222
• Development running at http://localhost:44444

Note that if you have any port conflicts on your machine, you can map any free port numbers you like, making this exercise very flexible across your available machines.

Next, you'll be setting up a file-based discovery integration with Prometheus that allows your applications and pipelines to modify a file for dynamic targeting of the infrastructure you want to scrape. This file (targets.yml) in our exercise will look something like this if you are targeting the above infrastructure:

- targets:
    - "localhost:11111"
    - "localhost:22222"
    labels:
        job: "services"
        env: "production"

- targets:
    - "localhost:44444"
    labels:
        job: "services"
        env: "development"

Configuring your Prometheus instance requires a new file-based discovery section in your workshop-prometheus.yml file:

# workshop config
global:
  scrape_interval: 5s

scrape_configs:
  # Scraping Prometheus.
  - job_name: "prometheus"
  static_configs:
    - targets: ["localhost:9090"]

  # File based discovery.
  - job_name: "file-sd-workshop"
  file_sd_configs:
    - files:
      - "targets.yml"

After saving your configuration and starting your Prometheus instance, you are then shown how to verify that the target infrastructure is now being scraped:

Next up, you'll start adding dynamic changes to your target file and see that they are automatically discovered by Prometheus without having to restart your instance.

Exploring Dynamic Discovery

The rest of the lab walks through multiple exercises where you make dynamic changes and verify that Prometheus is able to automatically scale to the needs of your infrastructure. For example, you'll first change the infrastructure you have deployed by promoting the development environment to become the staging infrastructure for your organization.

First, you update the targets file:

- targets:
    - "localhost:11111"
    - "localhost:22222"
    labels:
        job: "services"
        env: "production"

- targets:
    - "localhost:44444"
    labels:
        job: "services"
        env: "staging"

Then you verify that the changes are picked up, this time using a PromQL query and the Prometheus console without having to restart your Prometheus instance:

Later in the lab, you are given exercises to fly solo and add a new testing environment so that the end results of your dynamically growing observability infrastructure contain production, staging, testing, and your Prometheus instance:

Missed Previous Labs?

This is one lab in the more extensive free online workshop. Feel free to start from the very beginning of this workshop here if you missed anything previously:

You can always proceed at your own pace and return any time you like as you work your way through this workshop. Just stop and later restart Perses to pick up where you left off.

Coming Up Next

I'll be taking you through the following lab in this workshop where you'll learn all about instrumenting your applications for collecting Prometheus metrics.

Stay tuned for more hands-on material to help you with your cloud-native observability journey.