Interesting Java Jersey and RxJava Finding

It so happens that a few of the apps I am working on here at Netflix use Jersey libraries inside a Tomcat container. As such, I use the JSR annotations a lot, since I feel safe in the knowledge that Jersey will take care of all the plumbing work and deal with routing and serialization/deserialization. This is so I can concentrate on just writing the actual business logic in my web application.

Apart from the (very common) @Path and @GET@POST@Producesand @Consumes, I find myself occasionally needing to access the actual raw HttpServletRequest object. Typically, this is in order to parse some headers and/or read cookies, but I guess for the purpose of this exercise you can imagine any other operation you might need to apply to the request object. The point being that, in such situations, I use the @Context annotation and have Jersey automatically inject the object I need (whether it is the request, response on the SecurityContext, etc.) And, this is where I came across this obscure, yet super-interesting, issue. It’s not a bug. In fact, in the Jersey doccos, it seems this behaviour is actually documented. However, it also seems that rather little is known.

Let’s look at this  simple code. As I said, you will need to run it in the context of Jersey and JSR-311 inside a Tomcat container. (Those details don’t really matter, — apart from the Jersey bit — however, it might make it easier for you to set up a project to repro this if you need to. Also, it might make it easier to identify, if you are hitting exactly this problem when you are reading this.)

import javax.inject.Singleton;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;

@Singleton
@Path("unsecure")
public final class UnsecureResource {
   @Path("language")
   @GET
   @Produces({MediaType.TEXT_PLAIN})
   public String usingRequest(@Context HttpServletRequest request) {
      return request.getHeader("accept-language");
   }
}

This class is a simple REST resource, which does something very simple. When the user hits the /unsecure/language HTTP endpoint, it simply prints out the value of the user’s browser Accept-Language header. In my case, it looked something like this: en-US,en;q=0.9.

Nothing spectacular, right? The request object gets injected by Jersey. I’m calling a method on it and displaying the result.

Now, let’s throw RxJava into this. To do this, I’m going to basically take the request object, emit it via Observable.just(), and then go from there:

@Path("language-rx")
@GET
@Produces({MediaType.TEXT_PLAIN})
public String usingRxRequest(@Context HttpServletRequest request) {
   return Observable.just(request)
      .map(req -> req.getHeader("accept-language"))
      .toBlocking()
      .single();
}

So, now, if I hit (GET) /unsecure/language-rx, I get the same thing! So far, not surprised, right?

OK, well the surprise comes if you start running this in a multi-user environment. If you get multiple users hitting that endpoint at the same time, you will get a surprise. But, I don’t want to ruin it for you! Instead, I will tell you exactly how to reproduce it for yourself, without the need for other users.

In this case, the only operation I am applying on the request object is the getHeader   — but, let’s say that I know I will be doing all sorts of things to the poor object. In that case, in an Rx environment, I would probably indicate to the framework that I’m hammering this and I probably need to run this in the computation scheduler (though you can try this with the I/O scheduler or another one). So, the above code will just become:

@Path("language-rx")
@GET
@Produces({MediaType.TEXT_PLAIN})
public String usingRxRequest(@Context HttpServletRequest request) {
   return Observable.just(request)
      .map(req -> req.getHeader("accept-language"))
      .subscribeOn(Schedulers.computation())
      .toBlocking()
      .single();
}

Now, let’s hit our endpoint again:

HTTP STATUS 500 – JAVA.LANG.ILLEGALSTATEEXCEPTION: NO THREAD LOCAL VALUE IN SCOPE FOR PROXY OF CLASS COM.SUN.PROXY.$PROXY101

type — Exception report
message — java.lang.IllegalStateException: No thread local value in scope for proxy of class com.sun.proxy.$Proxy101
description — The server encountered an internal error that prevented it from fulfilling this request.

Wooow!!! What gives?

Well, it turns out that, as per our request, our Rx code is now running in a different thread, and when it is trying to access our HttpServletRequest object, we get that nasty error.

Looking into this, it turns out that in the case of the @Context annotation, Jersey will inject a proxy object which uses a ThreadLocal (see here). So, when your proxy is finally evaluated, it tries to access the ThreadLocal instance. But, in this last case, we are by now in a different thread (the Rx computation thread). In this thread, our ThreadLocal instance has NOT been intiatialized, because the Jersey injector does not run in this thread. Therefore, we end up with this error.

So, how do we get around this? Well, there are two ways. First, as shown above, there is the option to only use the HttpServletRequest object in the same thread as the request came in. In this case, simply extracting this operation outside of Rx might suffice, as shown in the first version of this.

However, that is not always viable. In the above case, we only call one method on the request object, so it’s easy to extract this. But, what if we had tons of lengthy processing applied to it? In that case, we definitely want that wrapped in an Observable.defer() and ran in the correct scheduler (computation or I/O or whatever is suitable). If we want to do that, then the trick is to have Jersey inject the ThreadLocal  directly and materialize it upfront:

@Path("language-rx")
@GET
@Produces({MediaType.TEXT_PLAIN})
public String usingRxRequest(@Context ThreadLocal<HttpServletRequest> request) {
   final HttpServletRequest r = request.get();
   return Observable.just(r)
      .map(req -> req.getHeader("accept-language"))
      .subscribeOn(Schedulers.computation())
      .toBlocking()
      .single();
}

In this case, we’re ensuring that we are getting a handle of the request object in the same thread as the request by extracting it outside all of the Rx operations. This is a very fast operation, so it’s not that costly. Then, you will need to throw that into the Rx chain. From there, we are dealing with a “materialized” instance of a raw request.

This goes even deeper. I have actually found this because of my usage of RxJava, but, at the core of the issue, it occurs because of the evaluation of the HttpServletRequest(proxy) in a different thread. So, having your own thread pool and passing this proxy into one of your threads in would render the same issue. In fact, depending how you implement it, it might NOT throw any exceptions, but you could end up with the very first request object being used to initialize the ThreadLocal and then consecutive calls. Needless to say, this would be even more difficult to track down and detect, as you would get some weird errors in other parts of the code and would take you ages before you realize it. There are pros and cons to this approach and I don’t know if Jersey will ever change this part of their implementation, but, for now, just be weary of this if you use any other thread apart from the original one for processing requests.