Java 9: Reflection and Package Access Changes

Java 1.9 is out, and many companies will now be evaluating Java 1.9 and migrating their codebase over to it. I am not going through a similar process.

I have some fairly simple code that I was using for HTTP requests. This was originally written in Java 1.8. To keep my codebase as simple as possible, and because I wanted to use no external libraries, I used an HttpURLConnection for HTTP requests.

This started to fail on Java 1.9.

When I looked at my code more carefully, I realized that it wasn’t HttpURLConnection that was failing in Java 1.9 — it was my use of Reflection to bypass some constraints of the HttpUrlConnection that was failing.

What Constraints?

With the HttpUrlConnection , I could not find a way to get the actual request headers being sent.

I could see that the URLConnection class does have all the request headers:

private MessageHeader requests;

But it keeps them all to itself.

The class does support a mechanism to return these with the getRequestProperties method, which is public.

public Map<String, List<String>> getRequestProperties() {

Huzzah!

But, wait a minute.

public Map<String, List<String>> getRequestProperties() {
    this.checkConnected();
    return this.requests == null?Collections.emptyMap():this.requests.getHeaders((String[])null);
}

getRequestProperties does not actually want to return them if we are connected, the checkConnected method throws an exception if we try.

Fair enough.

New plan: I debug the code and find out when we can get them, and I’ll take them after they are set, but before we are connected.

But I could not actually find that point.

int statusCode = con.getResponseCode();

Prior to the above line, I am not connected, and there are no request headers accessible.

After the above line, the request has been made — I am connected, and the headers are not accessible.

I tried disconnecting, and a whole bunch of stuff, but since I had a problem to solve and the ‘class’ was getting in my way, I turned to Reflection to solve it.

What I Did That Failed

Since I knew that the requests field on the URLConnection had the information that I needed, I simply used reflection to go and get the information that I needed.

Field field = con.getClass().getDeclaredField("requests");
field.setAccessible(true);
MessageHeader requestHeaders = (MessageHeader) field.get((URLConnection) con);
System.out.println(requestHeaders.getHeaders().size());

lastRequest = new HttpRequestDetails();

int keyIndex = 0;
while(keyIndex < requestHeaders.getHeaders().size()){
    lastRequest.addHeader(requestHeaders.getKey(keyIndex), requestHeaders.getValue(keyIndex));
    keyIndex++;
}

And all of that was very jolly, until Java 1.9 came along.

Migrating to Java 1.9

After debugging the problem I faced, I found the following blog post from codefx.org. It provided useful information:

• Migrating to Java 1.9

The problem I faced was that when I migrated to Java 1.9, I faced the issue of:

Error:(7, 19) java: package sun.net.www does not exist
Error:(169, 13) java: cannot find symbol
  symbol:   class MessageHeader
  location: class com.javafortesters.course.casestudy.http.HttpRequestSender
Error:(169, 45) java: cannot find symbol
  symbol:   class MessageHeader
  location: class com.javafortesters.course.casestudy.http.HttpRequestSender

Now, I found this odd.

The import of sun.net.www.MessageHeader wasn’t throwing any errors. Until compilation.

import sun.net.www.MessageHeader;

And when I looked in the code for URLConnection, the responses field was still there and was still of type sun.net.www.MessageHeader

So the package does exist, but just not for me.

Java 1.9 has finally made proprietary packages inaccessible. This was a long-time warning in Java, and it has now come to fulfillment.

It is still in the JRE, but not accessible to me due to Java 1.9 changes.

Very well then.

Since the getRequestProperties method still exists, I decided to bypass the check for this.checkConnected();.

I’m sure the library probably has very good reasons for not wanting me to access the List while we are connected, but in my code, I view this as low-risk — I have a single thread, I’ve made the request, and I just want to document the headers that were sent.

So I used reflection to toggle the connected field value so that the checkConnected() method would not throw an exception.

Field connected = con.getClass().getSuperclass().getSuperclass().getDeclaredField("connected");
connected.setAccessible(true);
connected.setBoolean((URLConnection) con, false);

Map<String, List<String>> requestHeaders = con.getRequestProperties();
for(String header : requestHeaders.keySet()){
    StringBuilder headerValue = new StringBuilder();
    for(String headerText : requestHeaders.get(header)){
        headerValue.append(headerText);
    }
    lastRequest.addHeader(header, headerValue.toString());
}
connected.setBoolean((URLConnection) con, true);
connected.setAccessible(false);

And this worked in Java 1.9, and 1.8 and 1.7

But… Java 1.9 Doesn’t Like Reflection as Much

But Java 1.9 tells me that this reflective access operation is ILLEGAL!

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.javafortesters.course.casestudy.http.HttpRequestSender
 (file:/) to field java.net.URLConnection.connected
WARNING: Please consider reporting this to the maintainers of com.javafortesters.course.casestudy.http.HttpRequestSender
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I assume this is due to the new module system in Java 1.9.

So while my code works, it will probably fail again in the near future.

Reflection Comes With Risk

Reflection always comes with the risk that the underlying code you are using changes.

And it has every right to do so.

I should not be accessing its private fields and methods.

But sometimes, when testing, I need access to features and functionality that, for seemingly arbitrary reasons, the class does not support.

Had this library been written to fully support a ‘testing’ process, where we don’t just want to make requests, we want to know exactly what details were sent in that request, then I wouldn’t have this problem.

The reason I learned to use reflection in Java in the first place was because I’ve worked with ‘frameworks’ that do stuff that I want to use as part of my test process to validate results, or to trigger situations that are necessary for testing, but that the ‘framework’ decided that no-one would want to do.

In theory, there is nothing wrong with the URLConnection telling me what the request headers are. It could have added them to a synchronized or immutable collection — it is pure documentation.

Sometimes, good solid technical reasons for functionality get in the way of testing.

Alternatives

I could have added another HTTP library to my code to make the requests and give me access to the raw data.

I could have added a code-based HTTP Proxy, sent all requests through the proxy, and used the proxy to gain access to the headers.

All of that was possible, but it went against my strategic aims of keeping the codebase as free of external dependencies as possible, so I adopted a tactical maneuver to gain access to what I needed.

Final Results

In the public release of the above code, I have removed all the reflection. But I have kept it in the code I use for internal testing because I can control the JVM and settings that the code runs against.

I will continue to use reflection to bypass constraints in libraries and frameworks that I use as a tactical measure to get work done.

Relying on this approach for strategic work is a risk that might bite you at a time when you are least flexible to respond, so care needs to be taken when you do this.