A Hands-On Guide to OpenTelemetry: Programmatic Instrumentation for Developers

Are you ready to start your journey on the road to collecting telemetry data from your applications? Great observability begins with great instrumentation! 

In this series, you'll explore how to adopt OpenTelemetry (OTel) and how to instrument an application to collect tracing telemetry. You'll learn how to leverage out-of-the-box automatic instrumentation tools and understand when it's necessary to explore more advanced manual instrumentation for your applications. By the end of this series, you'll have an understanding of how telemetry travels from your applications to the OpenTelemetry Collector, and be ready to bring OpenTelemetry to your future projects. Everything discussed here is supported by a hands-on, self-paced workshop authored by Paige Cruz. 

In the previous article, we explored how to gain better insights by adding manual instrumentation to our application leveraging the existing auto-instrumentation. In this article, we move on to programmatic instrumentation in our application as developers would in their daily coding. We'll be using OTel libraries and eventually visualize the collected telemetry data in Jaeger.

It is assumed that you followed the previous articles in setting up both OpenTelemetry and the example Python application project, but if not, go back and see the previous articles, as that is not covered here.

Installing Instrumentation

Up until now in this series, our go-to library for OTel instrumentation has been opentelemetry-bootstrap, which we'll continue to use in this article as we explore programmatic instrumentation. Using our container image from a previous article, instead of installing the instrumentation libraries, we can list all available libraries with the following command:

$ podman run -it hello-otel:auto opentelemetry-bootstrap -a requirements

opentelemetry-instrumentation-asyncio==0.46b0
opentelemetry-instrumentation-aws-lambda==0.46b0
opentelemetry-instrumentation-dbapi==0.46b0
opentelemetry-instrumentation-logging==0.46b0
opentelemetry-instrumentation-sqlite3==0.46b0
opentelemetry-instrumentation-threading==0.46b0
opentelemetry-instrumentation-urllib==0.46b0
opentelemetry-instrumentation-wsgi==0.46b0
opentelemetry-instrumentation-asgi==0.46b0
opentelemetry-instrumentation-flask==0.46b0
opentelemetry-instrumentation-grpc==0.46b0
opentelemetry-instrumentation-jinja2==0.46b0
opentelemetry-instrumentation-requests==0.46b0
opentelemetry-instrumentation-urllib3==0.46b0

Many of these are not going to be needed for our application, so installing them is a bit of overkill. They are used to instrument features that are not part of our Flask application, such as the asyncio or urllib3. There are three libraries, shown in bold in the above list, that we want to install and configure for our application:

• opentelemetry-instrumentation-flask: Traces web requests made to our application
• opentelemetry-instrumentation-jinja2: Traces the template loading, compilation, and rendering
• opentelemetry-instrumentation-requests: Traces HTTP requests made by the requests library

Using the downloaded project we installed from previous articles, we can open the file programmatic/Buildfile-prog and add the bold lines below to install the API, SDK, and library instrumentation:

FROM python:3.12-bullseye

WORKDIR /app

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt

RUN pip install opentelemetry-api \
    opentelemetry-sdk \
    opentelemetry-instrumentation-flask \
    opentelemetry-instrumentation-jinja2 \
    opentelemetry-instrumentation-requests

COPY . .

CMD [ "flask", "run", "--host=0.0.0.0"]

Configuring Instrumentation

The OTel SDK provides us with the tools to create, manage, and export tracing spans. To do this we have to configure the following three components in our application:

• Tracer Provider: Constructor method; returns a tracer that creates, manages, and sends spans
• Processor: Hooks into ended spans; options to send spans as they end or in a batch
• Exporter: Sends spans to the configured destination

To get started, we import the OTel API and SDK in our application by adding the bold lines below to our application code found in programmatic/app.py:

import random
import re
import urllib3

import requests
from flask import Flask, render_template, request
from breeds import breeds


from opentelemetry.trace import set_tracer_provider
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
...

Now we can configure the imported tracer, by creating a TraceProvider to send spans as soon as they complete to the console as output destination. In the same file, just below the imports, we are adding the code shown in bold below to the application:

...
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

provider = TracerProvider()
processor = SimpleSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)

set_tracer_provider(provider)
...

Next, we insert the imports needed for flask, jinja2, and requests instrumentation libraries above the section we just created. The code to be added is shown in bold below:

import random
import re
import urllib3

import requests
from flask import Flask, render_template, request
from breeds import breeds


from opentelemetry.trace import set_tracer_provider
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

provider = TracerProvider()
processor = SimpleSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)

set_tracer_provider(provider)
...

The last step for our developer is to configure the programmatic instrumentation for each component in the application. This is done by creating an instance of the FlaskInstrumentor, the Jinja2Instrumentor, and RequestsInstrumentor in the section of our file as shown in bold:

...
provider = TracerProvider()
processor = SimpleSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)

set_tracer_provider(provider)

app = Flask("hello-otel")

FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
...

Note that we only need to pass the application to the FlaskInstrumentor instrumenting constructor, while the other two are left empty. Save and close the file programmatic/app.py and build this container image with the following command:

$ podman build -t hello-otel:prog -f programmatic/Buildfile-prog

Successfully tagged localhost/hello-otel:prog  \
516c5299a32b68e7a4634ce15d1fd659eed2164ebe945ef1673f7a55630e22c8

When we run this container image, we map the flask port from the container to our local 8001 as follows:

$ podman run -i -p 8001:8000 -e FLASK_RUN_PORT=8000 hello-otel:prog

Open a browser, make a request to an endpoint http://localhost:8001, and confirm spans are printed to the console something like what is shown below:

{
    "name": "/",
    "context": {
        "trace_id": "0xd3afc4d7da2f0cd37af1141954aac0a3",
        "span_id": "0xe6a5b15b3bc2d751",
        "trace_state": "[]"
    },
    "kind": "SpanKind.SERVER",
    "parent_id": null,
    "start_time": "2024-04-21T20:20:02.172651Z",
    "end_time": "2024-04-21T20:20:02.174298Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "http.method": "GET",
        "http.server_name": "0.0.0.0",
        "http.scheme": "http",
        "net.host.port": 8000,
        "http.host": "localhost:8001",
        "http.target": "/",
        "net.peer.ip": "10.88.0.60",
        "http.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OSX 10_15_7)...",
        "net.peer.port": 47024,
        "http.flavor": "1.1",
        "http.route": "/",
        "http.status_code": 200
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.25.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

While scrolling through the spans in a console is fun, it's not really ideal for truly visualizing what is happening in your applications and services. To fix that we can add some span visualization tooling to our solution, which will be the focus of the next article.

These examples use code from a Python application that you can explore in the provided hands-on workshop (linked earlier in this article). 

What's Next?

This article introduced a journey into programmatic instrumentation where we instrumented our application as developers would experience it in their daily coding tasks. We saw that the resulting instrumentation was all dumping to the console which is lacking in visualizing the telemetry data, so in the next article, we are adding visualization tooling to our workflow.