Microservices Observability (Part 1)

According to microservices architecture and modern systems design, there are 5 observability patterns that help us to achieve the best in terms of monitoring distributed systems. They are the foundation for all who want to build reliable cloud applications. This tutorial will dive into domain-oriented observability, monitoring, instrumentation and tracing in a business-centered approach with a practical view using open-source projects sustained by the cloud-native computing foundation (CNCF).

WARNING: This is not a production application! It will not integrate with any polar API or device. This project was built in order to demonstrate concepts regarding observability patterns for microservices architecture. The main goal is to demonstrate how to monitor, instrument and trace microservices across the network with different technologies.

The Use-Case Scenario

Almost everyone has a sports watch or a smartwatch. The user synchronizes an activity log after a training session. It could be an ordinary training session or a running session (w/ specific running data added).

The API collects the training session data and propagates through the wire to different 3rd party "example" applications like and google calendar. All data is received and/or enriched to specific 3rd party APIs.

All the communication is traced using OpenTracing API and we can also collect custom metrics in the polar-flow-api like:

counter.activity
counter.running
gauge.burned.calories
gauge.training.load
gauge.running.distance
counter.activity.sport (for different sport types)
gauge.vo2max
gauge.heart.max
gauge.running.distance
gauge.running.cadence
gauge.running.index
timer.user.sync_ [seconds_max, seconds_count, seconds_sum]

JSON Example: Sync an ordinary training session:

JSON Example: Sync a running training session:

The postman collection used in this lab can be downloaded here

With Uber's Jaeger it is possible to trace all communication:

You can navigate through spans  in a trace  of the  POST /sync  operation.

 polar-flow-api  primary endpoints

 *-integration-api  (FUSE) endpoints

 polar-flow-api  secondary endpoints

Observability Labs: Step 1 - Project Creation

export current_project=microservices

# login into openshift platform
oc login https://master.<>.com:443 --token=<>

# create a new project
oc new-project microservices --description="microservices observability" --display-name="microservices"

Observability Labs: Step 2 - Prometheus Operator

1- Go to the Cluster Console menu:

2- On the left menu, navigate through Catalog Sources, under Operators:

3- Change the namespace for our recently created namespace:

4- Under Red Hat Operators section, choose the Prometheus Operator and click on the button: Create Subscription:

5- Check the YAML definition and confirm the subscription clicking on the Create button:

6- Check if the subscription was created. You'll see the info: 1 installed:

7- Click on the 1 installed link to see Prometheus operator overview tab:

8- Create a new service monitor, clicking on the "Create New" button:

9- Change the suggested YAML definition with the following:

The new definition will enable:

• The namespace that Prometheus will work on using namespaceSelector;
• The label that will be used on all service (svc) definitions  monitor: springboot2-api
• The name of the endpoint http that will be used to scrape the Prometheus data and the path that the application will expose Prometheus' endpoint. All applications using these definitions will be scraped with this service monitor configuration. Confirm the service monitor creation clicking on the "Create" button.

10- The service monitor will be created and be ready to scrape application metrics:

11- Next, let's deploy the Prometheus server. Return to operator overview tab and select:

12- Change the suggested YAML definition with the following:

13- Check the deployment under operator instances tab:

14- Expose Prometheus server:

oc expose svc prometheus-operated

Observability Labs: Step 5 - Grafana

oc import-image openshift3/grafana --from=registry.redhat.io/openshift3/grafana --confirm -n openshift

wget -O grafana-standalone.yaml https://raw.githubusercontent.com/aelkz/microservices-observability/master/_configuration/grafana/grafana.yaml

oc new-app -f grafana-standalone.yaml --param NAMESPACE=${current_project}

# OBS. Grafana will be deployed with a oauth-proxy sidecard. In order to access grafana, you'll need access using some service-account and/or cluster admin.

# get grafana's route
echo https://$(oc get route grafana --template='{{ .spec.host }}')

1. access grafana and confirm user permissions.
2. Add a new Prometheus datasource with the Prometheus service at http://prometheus-operated.microservices.svc.cluster.local:9090, with the following configuration. Click Save&Test, then Back.

After that, import the main dashboard that was designed for this use case. The JSON file can be located in this repository at  configuration/grafana/main-dashboard.json 

Observability Labs: Step 6 - Main Application Deployment

Observability Labs: Step 7 - Integration Deployment

Now that the main API is deployed, let’s deploy the integration layer.

Now that the main API and the integration API are deployed, let’s deploy the 3rd party application layer. This layer simulates all 3rd party applications in our example like social networks and medical specific APIs.

And tracing working as expected.

Observability Labs: Conclusion

Openshift Operators An Operator is a method of packaging, deploying and managing a Kubernetes-native application. A Kubernetes-native application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling.

Prometheus is an open-source tool for instrumenting and monitoring metrics. It works in a pull-based manner, makes HTTP requests to your metric endpoint with the pre-determined time intervals(default 10seconds), and store these metrics in its own time-series database. Prometheus also provides a GUI to make queries over these metrics with its own query language PromQL. It provides basic graphics to visualize metrics. Prometheus also has an alert plugin to produce alerts according to metrics values.

Jaeger key features:

• High Scalability – Jaeger backend is designed to have no single points of failure and to scale with the business needs.
• Native support for OpenTracing – Jaeger backend, Web UI, and instrumentation libraries have been designed from the ground up to support the OpenTracing standard.
• Storage Backend – Jaeger supports two popular open-source NoSQL databases as trace storage backends: Cassandra 3.4+ and Elasticsearch 5.x/6.x.
• Modern UI – Jaeger Web UI is implemented in Javascript using popular open-source frameworks like React.
• Cloud-Native Deployment – Jaeger backend is distributed as a collection of Docker images. - Deployment to Kubernetes clusters is assisted by Kubernetes templates and a Helm chart.
• All Jaeger backend components expose Prometheus metrics by default

Grafana Grafana is an open-source metric analytics and visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control.

Thanks for reading and taking the time to comment!

Feel free to create a PR on https://github.com/aelkz/microservices-observability