HTTP Headers Forwarding in Microservices

Microservices are not a trend anymore. Like it or not, they are here to stay. Yet, there’s a huge gap between embracing a microservices architecture and implementing it correctly. As a reminder, one might first want to check the many fallacies of distributed computing. Among all the requirements necessary to overcome them is the ability to follow one HTTP request along microservices involved in a specific business scenario — for monitoring and debugging purposes.

One possible implementation of it is a dedicated HTTP header with an immutable value passed along every microservice involved in the call chain. In the Spring ecosystem, the Spring Cloud Sleuth is the library dedicated to that:

Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud, borrowing heavily from Dapper, Zipkin, and HTrace. For most users, Sleuth should be invisible, and all your interactions with external systems should be instrumented automatically. You can capture data simply in logs, or by sending it to a remote collector service.

Within Spring Boot projects, adding the Spring Cloud Sleuth library to the classpath will automatically add 2 HTTP headers to all calls:

X-B3-Traceid

Shared by all HTTP calls of a single transaction i.e. the wished-for transaction identifier

X-B3-Spanid

Identifies the work of a single microservice during a transaction

Spring Cloud Sleuth offers some customization capabilities, such as alternative header names, at the cost of some extra code.

Diverging From Out-of-the-Box Features

Those features are quite handy when starting from a clean slate. Unfortunately, the project I’m working has a different context:

• The transaction ID is not created by the first microservice in the call chain — a mandatory façade proxy does
• The transaction ID is not numeric — and Sleuth handles only numeric values
• Another header is required. Its objective is to group all requests related to one business scenario across different call chains
• A third header is necessary. It’s to be incremented by each new microservice in the call chain

A solution architect’s first move would be to check among API management products, such as Apigee (recently bought by Google) and search for one that offers the feature matching those requirements. Unfortunately, the current context doesn’t allow for that.

Coding the Requirements

In the end, I ended up coding the following using the Spring framework:

1. Read and store headers from the initial request.
2. Write them in new microservice requests.
3. Read and store headers from the microservice response.
4. Write them in the final response to the initiator, not forgetting to increment the call counter.

The first step is to create the entity responsible for holding all necessary headers. It’s unimaginatively called HeadersHolder. Blame me all you want, I couldn’t find a more descriptive name.

private const val HOP_KEY = "hop"
private const val REQUEST_ID_KEY = "request-id"
private const val SESSION_ID_KEY = "session-id"

data class HeadersHolder (var hop: Int?,
                          var requestId: String?,
                          var sessionId: String?)

The interesting part is deciding which scope is more relevant to put instances of this class in. Obviously, there must be several instances, so this makes singleton unsuitable. Also, because data must be stored across several requests, it cannot be prototype. In the end, the only possible way to manage the instance is through a ThreadLocal.

Though it’s possible to manage ThreadLocal, let’s leverage Spring’s features, as it allows us to easily add new scopes. There’s already an out-of-the-box scope for ThreadLocal, one just needs to register it in the context. This directly translates into the following code:

internal const val THREAD_SCOPE = "thread"

@Scope(THREAD_SCOPE)
annotation class ThreadScope

@Configuration
open class WebConfigurer {

    @Bean @ThreadScope
    open fun headersHolder() = HeadersHolder()

    @Bean open fun customScopeConfigurer() = CustomScopeConfigurer().apply {
        addScope(THREAD_SCOPE, SimpleThreadScope())
    }
}

Let’s implement requirements 1 and 4 above: read headers from the request and write them to the response. Also, headers need to be reset after the request-response cycle to prepare for the next one.

This also mandates for the holder class to be updated to be more OOP-friendly:

data class HeadersHolder private constructor (private var hop: Int?,
                                              private var requestId: String?,
                                              private var sessionId: String?) {
    constructor() : this(null, null, null)

    fun readFrom(request: HttpServletRequest) {
        this.hop = request.getIntHeader(HOP_KEY)
        this.requestId = request.getHeader(REQUEST_ID_KEY)
        this.sessionId = request.getHeader(SESSION_ID_KEY)
    }

    fun writeTo(response: HttpServletResponse) {
        hop?.let { response.addIntHeader(HOP_KEY, hop as Int) }
        response.addHeader(REQUEST_ID_KEY, requestId)
        response.addHeader(SESSION_ID_KEY, sessionId)
    }

    fun clear() {
        hop = null
        requestId = null
        sessionId = null
    }
}

To keep controllers free from any header-management concern, related code should be located in a filter or a similar component. In the Spring MVC ecosystem, this translates into an interceptor.

abstract class HeadersServerInterceptor : HandlerInterceptorAdapter() {

    abstract val headersHolder: HeadersHolder

    override fun preHandle(request: HttpServletRequest,
                           response: HttpServletResponse, handler: Any): Boolean {
        headersHolder.readFrom(request)
        return true
    }

    override fun afterCompletion(request: HttpServletRequest, response: HttpServletResponse,
                                 handler: Any, ex: Exception?) {
        with (headersHolder) {
            writeTo(response)
            clear()
        }
    }
}

@Configuration open class WebConfigurer : WebMvcConfigurerAdapter() {

    override fun addInterceptors(registry: InterceptorRegistry) {
        registry.addInterceptor(object : HeadersServerInterceptor() {
            override val headersHolder: HeadersHolder
                get() = headersHolder()
        })
    }
}

Note the invocation of the clear() method to reset the headers holder for the next request.

The most important bit is the abstract headersHolder property. As its scope, thread, is smaller than the adapter’s, it cannot be injected directly, as it will be only be injected during Spring’s context startup. Hence, Spring provides lookup method injection. The above code is its direct translation in Kotlin.

The previous code assumes the current microservice is at the end of the caller chain: it reads request headers and writes them back in the response (not forgetting to increment the ‘hop’ counter). However, monitoring is relevant only for a caller chain having more than one single link. How is it possible to pass headers to the next microservice (and get them back) — requirements two and three above?

Spring provides a handy abstraction to handle that client part — ClientHttpRequestInterceptor, that can be registered to a REST template. Regarding scope mismatch, the same injection trick as for the interceptor handler above is used.

abstract class HeadersClientInterceptor : ClientHttpRequestInterceptor {

    abstract val headersHolder: HeadersHolder

    override fun intercept(request: HttpRequest, 
                           body: ByteArray, execution: ClientHttpRequestExecution): ClientHttpResponse {
        with(headersHolder) {
            writeTo(request.headers)
            return execution.execute(request, body).apply {
                readFrom(this.headers)
            }
        }
    }
}

@Configuration
open class WebConfigurer : WebMvcConfigurerAdapter() {

    @Bean open fun headersClientInterceptor() = object : HeadersClientInterceptor() {
        override val headersHolder: HeadersHolder
            get() = headersHolder()
    }

    @Bean open fun oAuth2RestTemplate() = OAuth2RestTemplate(clientCredentialsResourceDetails()).apply {
        interceptors = listOf(headersClientInterceptor())
    }
}

With this code, every REST call using the oAuth2RestTemplate() will have headers managed automatically by the interceptor.

The HeadersHolder just needs a quick update:

data class HeadersHolder private constructor (private var hop: Int?,
                                              private var requestId: String?,
                                              private var sessionId: String?) {

    fun readFrom(headers: org.springframework.http.HttpHeaders) {
        headers[HOP_KEY]?.let {
            it.getOrNull(0)?.let { this.hop = it.toInt() }
        }
        headers[REQUEST_ID_KEY]?.let { this.requestId = it.getOrNull(0) }
        headers[SESSION_ID_KEY]?.let { this.sessionId = it.getOrNull(0) }
    }

    fun writeTo(headers: org.springframework.http.HttpHeaders) {
        hop?.let { headers.add(HOP_KEY, hop.toString()) }
        headers.add(REQUEST_ID_KEY, requestId)
        headers.add(SESSION_ID_KEY, sessionId)
    }
}

Conclusion

Spring Cloud offers many components that can be used out-of-the-box when developing microservices. When requirements start to diverge from what it provides, the flexibility of the underlying Spring Framework can be leveraged to code those requirements.