Distributed Tracing System (Spring Cloud Sleuth + OpenZipkin)
Distributed Tracing System is essential for microservice architecture and analyzes problem areas. Learn how with Spring Cloud Sleuth and analysis tool, Zipkin.
Join the DZone community and get the full member experience.
Join For FreeWhen we build a microservice architecture and the number of services keeps growing, we face the problem of debugging and tracing requests through our entire system. What happened when a user got a 500 error on his request? What service incorrectly processed his request? All these questions can be solved by the Distributed Tracing System. Let's take Spring Cloud Sleuth as an example.
How Spring Cloud Sleuth Works
To trace a request through a distributed system, the concepts TraceID and SpanID are introduced. TraceID is generated when a request enters our system and remains unchanged throughout its path. SpanID changes as the request passes from one service to another. If necessary, it is possible to generate new spans within one service to distinguish business processes.
Traces and spans are passed between systems in HTTP headers. One standard is B3 Propagation, which describes the header format for Distributed Tracing Systems. Context is passed from one service to another via headers of the form X-B3-*
. Having received a request with such headers, we can restore the tracing context and continue the execution flow.
The beauty of Sleuth is that it automatically injects the necessary headers to all requests generated by, for example, RestTemplate or Feign, using interceptors. For this to work automatically, you need to declare HTTP clients as Spring beans.
Let's consider the following architecture consisting of 3 microservices:
- A service for user registration
- A service responsible for the loyalty program
- A service for sending notifications to users
In this example, let's see how to connect Spring Cloud Sleuth to Spring Boot applications. We will use Feign as the HTTP client. After creating a Gradle project, we will add dependencies to each service: Sleuth, the library for sending data to Zipkin and Feign clients that will allow us to make requests to other services.
User Service
build.gradle:
ext {
set('springCloudVersion', "2021.0.0")
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.cloud:spring-cloud-starter-sleuth'
implementation 'org.springframework.cloud:spring-cloud-sleuth-zipkin'
implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'
compileOnly 'org.projectlombok:lombok:1.18.22'
annotationProcessor 'org.projectlombok:lombok:1.18.22'
testCompileOnly 'org.projectlombok:lombok:1.18.22'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.22'
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
}
}
We need to enable Feign functionality:
@SpringBootApplication
@EnableFeignClients
public class UserApplication {
public static void main(String[] args) {
SpringApplication.run(UserApplication.class, args);
}
}
User service when a user registers will form HTTP requests to bonus service, send a notification to the user, and return the identifier of the user and the bonus program. Let's write a controller:
@RestController
@RequestMapping("/user")
@RequiredArgsConstructor
public class UserController {
private final BonusClient bonusClient;
private final NotificationClient notificationClient;
@PostMapping("/register")
RegisterResponse register(@RequestBody RegisterRequest registerRequest) {
log.info("User registration request received");
String userId = UUID.randomUUID().toString(); // generate uniq identifier of user
String bonusId = bonusClient.register(); // register in loyalty program
notificationClient.send(new NotificationRequest(userId, "email")); // send notification to user
return new RegisterResponse(userId, bonusId); //return response to user
}
}
Using Feign, let's describe BonusClient and NotificationClient:
@FeignClient(name = "bonus", url = "http://localhost:8081/bonus")
public interface BonusClient {
@RequestMapping(value = "/register", method = RequestMethod.POST)
String register();
}
@FeignClient(name = "notification", url = "http://localhost:8082/notification")
public interface NotificationClient {
@PostMapping("/send")
String send(@RequestBody NotificationRequest notificationRequest);
}
Add the identifier of the application to be used by Sleuth:
spring:
application:
name: user-service
Let's implement the remaining services in the same way. The project dependencies will be the same.
Bonus Service
For simplicity, write only one controller:
@RestController
@RequestMapping("/bonus")
@Slf4j
public class BonusController {
@PostMapping("/register")
@SneakyThrows
String register() {
log.info("Bonus program registration request received");
//e.g. load from DB
Thread.sleep(new Random().nextInt(100, 1000));
return UUID.randomUUID().toString();
}
}
Here we will receive the request, simulate a delay and respond with the generated identifier. Similarly, add the application name to the application.yml:
spring:
application:
name: bonus-service
server:
port: 8081
We also specified a different port so our applications would not conflict with each other when we run them on the same instance.
Notification Service
Let's make the notification service a bit more complicated. It will receive the message and call the external service by simulating a call to the email gateway.
@RestController
@RequiredArgsConstructor
@RequestMapping("/notification")
@Slf4j
public class NotificationController {
private final ExternalEmailGateway gateway;
@PostMapping("/send")
@SneakyThrows
String send(@RequestBody NotificationRequest notificationRequest) {
log.info("Notification request received");
//e.g. load from DB & logic
Thread.sleep(new Random().nextInt(100, 1000));
gateway.sendEmail(1);
return "ok";
}
}
@FeignClient(name = "externalEmailGateway", url = "https://httpbin.org")
public interface ExternalEmailGateway {
@GetMapping("/delay/{delay}")
String sendEmail(@PathVariable int delay);
}
Here we will call https://httpbin.org
to simulate a delayed response.
application.yml:
spring:
application:
name: notification-service
server:
port: 8082
OpenZipkin
By default, our services will try to send trace messages to localhost:9411
. If OpenZipkin runs at a different address, you need to specify it in the settings of each service:
spring:
zipkin:
base-url: http://<host>:<port>
Run Zipkin using Docker:
docker run -d -p 9411:9411 openzipkin/zipkin
Let's launch all our applications and execute a user registration request:
./gradlew user-service:bootRun
./gradlew bonus-service:bootRun
./gradlew notification-service:bootRun
curl -L -X POST 'http://localhost:8080/user/register' \
-H 'Content-Type: application/json' \
--data-raw '{}'
{"userId":"b5d04d9a-0447-4cf5-9002-b10841181e9f","bonusProgramId":"bda336f6-f5ab-4fc9-97b9-58df741d1de2"}
We can now analyze the results by going to localhost:9411
Here you can see the total time of the request and how much time was spent on each of them. If one of the services returns an error, we will see the following:
Also, if you look in the logs, you will see records with the service name, TraceID, and SpanID in square brackets. As you can see within one request, TraceID remains constant, and SpanID changes from service to service:
2022-02-16 08:48:58.617 INFO [user-service,15f8d0cc9c8c9e4a,15f8d0cc9c8c9e4a] 7028 --- [nio-8080-exec-4] c.e.user.controller.UserController : User registration request received
...
2022-02-16 08:48:58.619 INFO [bonus-service,15f8d0cc9c8c9e4a,d8d79fe594bc4522] 12664 --- [nio-8081-exec-3] c.e.bonus.controller.BonusController : Bonus program registration request received
...
2022-02-16 08:48:59.366 INFO [notification-service,15f8d0cc9c8c9e4a,3d31b3eb8062cbe0] 25796 --- [nio-8082-exec-3] c.e.n.controller.NotificationController : Notification request received
Conclusion
Distributed Tracing System is essential for microservice architecture and allows you to analyze problem areas in the system. Spring Cloud Sleuth makes this as easy as possible with almost no added code, and Zipkin is a very easy-to-use analysis tool.
The project code is available on GitHub.
Opinions expressed by DZone contributors are their own.
Comments