Hacking Lambda Expressions in Java

In this article, we will show some little-known tricks with lambda expressions in Java 8 and their limitations. The main audience are senior Java developers, researchers, and instrumentation tool writers. We will use only the public Java API without com.sun or other internal classes so the code is portable across different JVM implementations.

Quick Intro

Lambda expressions wereintroduced in Java 8 as a way to implement anonymous methods and, in some cases, as alternatives for anonymous classes. At the bytecode level, a lambda expression is replaced with an invokedynamic instruction. This instruction is used to create implementations of a functional interface. and its single method delegates a call to the actual method with code defined inside of a lambda body.

For instance, we have the following code:

void printElements(List<String> strings){
    strings.forEach(item -> System.out.println("Item = %s", item));
}

This code will be translated by the Java compiler to something like this:

private static void lambda_forEach(String item) { //generated by Java compiler
    System.out.println("Item = %s", item);
}

private static CallSite bootstrapLambda(Lookup lookup, String name, MethodType type) { //
    //lookup = provided by VM
    //name = "lambda_forEach", provided by VM
    //type = String -> void
    MethodHandle lambdaImplementation = lookup.findStatic(lookup.lookupClass(), name, type);
    return LambdaMetafactory.metafactory(lookup,
        "accept",
        MethodType.methodType(Consumer.class), //signature of lambda factory
        MethodType.methodType(void.class, Object.class), //signature of method Consumer.accept after type erasure  
        lambdaImplementation, //reference to method with lambda body
        type);
}

void printElements(List < String > strings) {
    Consumer < String > lambda = invokedynamic# bootstrapLambda, #lambda_forEach
    strings.forEach(lambda);
}

The invokedynamic instruction can be roughly represented as the following Java code:

private static CallSite cs;

void printElements(List < String > strings) {
    Consumer < String > lambda;
    //begin invokedynamic
    if (cs == null)
        cs = bootstrapLambda(MethodHandles.lookup(), "lambda_forEach", MethodType.methodType(void.class, String.class));
    lambda = (Consumer < String > ) cs.getTarget().invokeExact();
    //end invokedynamic
    strings.forEach(lambda);

}

As you can see, LambdaMetafactory is used for producing a call site with the target method handle representing a factory method. This factory method returns an implementation of a functional interface using invokeExact. If the lambda has enclosed variables, then invokeExact accepts these variables as actual arguments.

In Oracle JRE 8, the metafactory dynamically generates a Java class using ObjectWeb Asm, which implements a functional interface. Additional fields to the generated class can be added if the lambda expression encloses external variables. This approach is similar to anonymous classes in the Java language — with the following differences:

• The anonymous class is generated by the Java compiler at compile-time.

• The class for lambda implementation is generated by the JVM at runtime.

Implementation of the metafactory depends on the JVM vendor and version

Of course, the invokedynamic instruction is not exclusively used for lambda expressions in Java. Primarily, the instruction is introduced for dynamic languages running on top of the JVM. The Nashorn JavaScript engine provided by Java out-of-the-box heavily utilizes this instruction.

Later in this article, we will focus on the LambdaMetafactory class and its capabilities. The next sections in this article are based on assumption that you completely understand how metafactory methods work and what MethodHandle is.

Tricks With Lambdas

In this section, we show how to use dynamic construction of lambdas for day-to-day tasks.

Checked Exceptions and Lambdas

It is not a secret that all functional interfaces provided by Java do not support checked exceptions. Checked versus unchecked exceptions in Java is an old holy war.

What if you want to use code with checked exceptions inside of lambdas used in conjunction with Java Streams? For example, we need to transform a list of strings into a list of URLs like this:

Arrays.asList("http://localhost/", "https://github.com")
.stream()
.map(URL::new)
.collect(Collectors.toList())

URL(String) has declared a checked exception in the throws section, therefore, it cannot be used directly as a method reference for Function.

You say "Yes, this is possible using tricks like this":

public static <T> T uncheckCall(Callable<T> callable) {
  try { return callable.call(); }
  catch (Exception e) { return sneakyThrow(e); }
}

private static <E extends Throwable, T> T sneakyThrow0(Throwable t) throws E { throw (E)t; }

public static <T> T sneakyThrow(Throwable e) {
  return Util.<RuntimeException, T>sneakyThrow0(e);
}

// Usage sample
//return s.filter(a -> uncheckCall(a::isActive))
//        .map(Account::getNumber)
//        .collect(toSet());

This is a dirty hack. Here's why:

• Using a try-catch block.

• Re-throwing of exceptions.

• Dirty utilization of type erasure in Java.

This problem can be solved in more "legal" way using the following facts:

• Checked exceptions are recognized only by a compiler of the Java programming language.

• The section throws is just metadata for the method without semantical meaning at the JVM level.

• Checked and unchecked exceptions are not distinguishable at the bytecode and JVM levels.

The solution is just a wrapping invocation of Callable.call into the method without the throws  section:

static <V> V callUnchecked(Callable<V> callable){
    return callable.call();
}

This code will not be compiled by Java compiler because method  Callable.call  has checked exception in its  throws  section. But we can erase this section using dynamically constructed lambda expression.

At first, we should declare a functional interface that has no throws section but is able to delegate a call to Callable.call:

@FunctionalInterface
interface SilentInvoker {
    MethodType SIGNATURE = MethodType.methodType(Object.class, Callable.class);//signature of method INVOKE
    <V> V invoke(final Callable<V> callable);
}

The second step is to create an implementation of this interface using LambdaMetafactory and to delegate a call of the method SilentInvoker.invoke to the method Callable.call. As was said previously, the throws section is ignored at the bytecode level, therefore, the method SilentInvoker.invoke is able to call the method Callable.call without declaring checked exceptions:

private static final SilentInvoker SILENT_INVOKER;

final MethodHandles.Lookup lookup = MethodHandles.lookup();
final CallSite site = LambdaMetafactory.metafactory(lookup,
                    "invoke",
                    MethodType.methodType(SilentInvoker.class),
                    SilentInvoker.SIGNATURE,
                    lookup.findVirtual(Callable.class, "call", MethodType.methodType(Object.class)),
                    SilentInvoker.SIGNATURE);
SILENT_INVOKER = (SilentInvoker) site.getTarget().invokeExact();

Third, write a utility method that calls Callable.call without declaration of checked exceptions:

public static <V> V callUnchecked(final Callable<V> callable) /*no throws*/ {
    return SILENT_INVOKER.invoke(callable);
}

Now, we can rewrite our stream without any problems with checked exceptions:

Arrays.asList("http://localhost/", "https://dzone.com")
.stream()
.map(url -> callUnchecked(() -> new URL(url)))
.collect(Collectors.toList());

This code will compile successfully because callUnchecked has no declared checked exception. Moreover, calling this method may be inlined using Monomorphic Inline Caching because there is only one class in the JVM that implements the interface SilentInvoker.

If implementation of  Callable.call throws some exception at runtime, then it would be caught by calling without any problem:

try{
    callUnchecked(() -> new URL("Invalid URL"));
} catch (final Exception e){
    System.out.println(e);
}

Despite the capabilities of this method, it is recommended to use the following recommendation:

Hide checked exceptions with callUnchecked only if the absense of the exception is guaranteed by the calling code

The following example demonstrates this approach:

callUnchecked(() -> new URL("https://dzone.com")); //this URL is always valid and the constructor never throws MalformedURLException 

Complete implementation of this utility method can be found here as a part of the open-source project SNAMP.

Working With Getters and Setters

This section is useful for writers of serialization/deserialization for different data formats such as JSON, Thrift, etc. Moreover, it might be pretty useful if your code heavily relies on Java Reflection for JavaBean getters and setters.

A getter declared in a JavaBean is a method with the name getXXX without parameters and a non-void return type. A setter declared in a JavaBean is a method with the name setXXX with a single parameter and a void return type. These two notations can be represented as functional interfaces:

• A getter can be represented as a Function where the argument of the function is a this  reference.

• A setter can be represented as a BiConsumer where the first argument is a this reference and the second is a value to be passed into the setter.

Now we create two methods that are able to convert any getter or setter into these functional interfaces. It doesn't matter that both functional interfaces are generics. After type erasure, the actual types are equal to an Object. Automatic casting of a return type and arguments can be done by LambdaMetafactory. Additionally, Guava's Cache helps to cache lambdas for the same getter or setter.

At first, it is necessary to declare a cache for getters and setters. Method from the Reflection API represents an actual getter or setter and is used as a key. The value in the cache represents a dynamically constructed functional interface for the particular getter or setter.

private static final Cache<Method, Function> GETTERS = CacheBuilder.newBuilder().weakValues().build();
private static final Cache<Method, BiConsumer> SETTERS = CacheBuilder.newBuilder().weakValues().build();

Second, create factory methods that create an instance of a functional interface from the method handle pointing to a getter or setter:

private static Function createGetter(final MethodHandles.Lookup lookup,
                                         final MethodHandle getter) throws Exception{
        final CallSite site = LambdaMetafactory.metafactory(lookup, "apply",
                MethodType.methodType(Function.class),
                MethodType.methodType(Object.class, Object.class), //signature of method Function.apply after type erasure
                getter,
                getter.type()); //actual signature of getter
        try {
            return (Function) site.getTarget().invokeExact();
        } catch (final Exception e) {
            throw e;
        } catch (final Throwable e) {
            throw new Error(e);
        }
}


private static BiConsumer createSetter(final MethodHandles.Lookup lookup,
                                           final MethodHandle setter) throws Exception {
        final CallSite site = LambdaMetafactory.metafactory(lookup,
                "accept",
                MethodType.methodType(BiConsumer.class),
                MethodType.methodType(void.class, Object.class, Object.class), //signature of method BiConsumer.accept after type erasure
                setter,
                setter.type()); //actual signature of setter
        try {
            return (BiConsumer) site.getTarget().invokeExact();
        } catch (final Exception e) {
            throw e;
        } catch (final Throwable e) {
            throw new Error(e);
        }

}

Automatic casting between Object-based arguments in functional interfaces after type erasure and actual types of arguments and a return type in a getter or setter is reached through a difference between samMethodType and instantiatedMethodType (the third and fifth arguments of the method metafactory, respectively). The instantiated method type is a specialization of the method that provides the implementation of a lambda.

Third, create a facade for these factories with the support of caching:

public static Function reflectGetter(final MethodHandles.Lookup lookup, final Method getter) throws ReflectiveOperationException {
        try {
            return GETTERS.get(getter, () -> createGetter(lookup, lookup.unreflect(getter)));
        } catch (final ExecutionException e) {
            throw new ReflectiveOperationException(e.getCause());
        }
}


public static BiConsumer reflectSetter(final MethodHandles.Lookup lookup, final Method setter) throws ReflectiveOperationException {
        try {
            return SETTERS.get(setter, () -> createSetter(lookup, lookup.unreflect(setter)));
        } catch (final ExecutionException e) {
            throw new ReflectiveOperationException(e.getCause());
        }
}

Method information obtained as a Method instance using the Java Reflection API can be easily transformed into a MethodHandle. Take into account that instance methods always have hidden first arguments used for passing this into the method. Static methods do not have these hidden parameters. For example, the method Integer.intValue() has the actual signature of int intValue(Integer this). This trick is used in our implementation of functional wrappers for getters and setters.

Now it is time to test the code:

final Date d = new Date();
final BiConsumer<Date, Long> timeSetter = reflectSetter(MethodHandles.lookup(), Date.class.getDeclaredMethod("setTime", long.class));
timeSetter.accept(d, 42L); //the same as d.setTime(42L);
final Function<Date, Long> timeGetter = reflectGetter(MethodHandles.lookup(), Date.class.getDeclaredMethod("getTime"));
System.out.println(timeGetter.apply(d)); //the same as d.getTime()
//output is 42

This approach with cached getters and setters can be used effectively in serialization/deserialization libraries (such as Jackson) that use getters and setters during serialization and deserialization.

The invocation of functional interfaces with dynamically generated implementations using LambdaMetafactory is significantly faster than invocation through the Java Reflection API

You can find the complete code here as a part of the open-source project SNAMP.

Limitations and Bugs

In this section, we will show some bugs and limitations associated with lambdas in the Java compiler and the JVM. All these limitations are reproducible on OpenJDK and Oracle JDK with javac 1.8.0_131 for Windows and Linux.

Constructing Lambdas From Method Handles

As you know, a lambda can be constructed dynamically using LambdaMetafactory. To achieve that, you should specify a MethodHandle with points to an implementation of a single method declared by a functional interface. Let's take a look at this simple example:

final class TestClass {
            String value = "";

            public String getValue() {
                return value;
            }

            public void setValue(final String value) {
                this.value = value;
            }
        }

final TestClass obj = new TestClass();
obj.setValue("Hello, world!");
final MethodHandles.Lookup lookup = MethodHandles.lookup();
final CallSite site = LambdaMetafactory.metafactory(lookup,
                "get",
                MethodType.methodType(Supplier.class, TestClass.class),
                MethodType.methodType(Object.class),
                lookup.findVirtual(TestClass.class, "getValue", MethodType.methodType(String.class)),
                MethodType.methodType(String.class));
final Supplier<String> getter = (Supplier<String>) site.getTarget().invokeExact(obj);
System.out.println(getter.get());

This code is equivalent to:

final TestClass obj = new TestClass();
obj.setValue("Hello, world!");
final Supplier<String> elementGetter = () -> obj.getValue();
System.out.println(elementGetter.get());

But what if we replace the method handle pointing to getValue with a method handle that represents a field getter:

final CallSite site = LambdaMetafactory.metafactory(lookup,
                "get",
                MethodType.methodType(Supplier.class, TestClass.class),
                MethodType.methodType(Object.class),
                lookup.findGetter(TestClass.class, "value", String.class), //field getter instead of method handle to getValue
                MethodType.methodType(String.class));

This code should work as expected because findGetter returns a method handle that points to the field getter and has a valid signature. But if you run the code, you will see the following exception:

java.lang.invoke.LambdaConversionException: Unsupported MethodHandle kind: getField

Interestingly, the field getter works pretty well if we use MethodHandleProxies:

final Supplier<String> getter = MethodHandleProxies
                                       .asInterfaceInstance(Supplier.class, lookup.findGetter(TestClass.class, "value", String.class)
                                       .bindTo(obj));

Note that MethodHandleProxies is not a decent way to create lambdas dynamically because this class just wraps MethodHandle into a proxy class and delegates the call of InvocationHandler.invoke to the method MethodHandle.invokeWithArguments. This approach uses Java Reflection and works very slowly.

As we saw previously, not all method handles can be used for constructing lambdas at runtime. 

Only several types of method handles related to methods can be used for dynamic construction of lambda expressions

Here they are:

• REF_invokeInterface: can be constructed by Lookup.findVirtual for interface methods

• REF_invokeVirtual: can be constructed by Lookup.findVirtual for virtual methods provided by a class

• REF_invokeStatic: can be constructed by Lookup.findStatic for static methods

• REF_newInvokeSpecial: can be constructed by Lookup.findConstructor for constructors

• REF_invokeSpecial: can be constructed by Lookup.findSpecial for private methods and early binding to virtual methods provided by a class

Other method handles will cause a LambdaConversionException.

Generic Exceptions

This bug is associated with the Java compiler and the ability to declare generic exceptions in throws sections. The following sample code demonstrates this behavior:

interface ExtendedCallable<V, E extends Exception> extends Callable<V>{
        @Override
        V call() throws E;
}


final ExtendedCallable<URL, MalformedURLException> urlFactory = () -> new URL("http://localhost");
urlFactory.call();

This code should be successfully compiled because the URL constructor throws MalformedURLException. But it is not. The compiler produces the following error message:

Error:(46, 73) java: call() in <anonymous Test$> cannot implement call() in ExtendedCallable

overridden method does not throw java.lang.Exception

But if we replace the lambda expression with an anonymous class, then the code is compiled successfully:

final ExtendedCallable<URL, MalformedURLException> urlFactory = new ExtendedCallable<URL, MalformedURLException>() {
            @Override
            public URL call() throws MalformedURLException {
                return new URL("http://localhost");
            }
        };
urlFactory.call();

The conclusion is simple

Type inferences for generic exceptions is not working correctly when used in conjunction with lambdas

Generic Bounds

A generic with multiple bounds can be constructed using the ampersand sign:  <T extends A & B & C & ... Z>. This kind of generic parameter definition is used rarely, but it has some impact on lambdas in Java due to its limitations:

• Every bound, except the first one, must be an interface.

• The raw version of the class with such a generic takes into account only the first bound in the constraint.

The second limitation produces different behavior of the Java compiler at compile time and the JVM at runtime when linkage of the lambda expression occurs. This behavior can be reproduced using the following code:

final class MutableInteger extends Number implements IntSupplier, IntConsumer { //mutable container of int value
    private int value;

    public MutableInteger(final int v) {
        value = v;
    }

    @Override
    public int intValue() {
        return value;
    }

    @Override
    public long longValue() {
        return value;
    }

    @Override
    public float floatValue() {
        return value;
    }

    @Override
    public double doubleValue() {
        return value;
    }

    @Override
    public int getAsInt() {
        return intValue();
    }

    @Override
    public void accept(final int value) {
        this.value = value;
    }
}

static < T extends Number & IntSupplier > OptionalInt findMinValue(final Collection < T > values) {
    return values.stream().mapToInt(IntSupplier::getAsInt).min();
}


final List < MutableInteger > values = Arrays.asList(new MutableInteger(10), new MutableInteger(20));
final int mv = findMinValue(values).orElse(Integer.MIN_VALUE);
System.out.println(mv);

This code is absolutely correct and is successfully compiled by a Java compiler. Class MutableInteger satisfies the multiple bounds of generic T:

• MutableInteger inherits from Number

• MutableInteger implements IntSupplier

But this code will throw an exception at runtime:

java.lang.BootstrapMethodError: call site initialization exception
    at java.lang.invoke.CallSite.makeSite(CallSite.java:341)
    at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307)
    at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297)
    at Test.minValue(Test.java:77)
Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver type class java.lang.Number; not a subtype of implementation type interface java.util.function.IntSupplier
    at java.lang.invoke.AbstractValidatingLambdaMetafactory.validateMetafactoryArgs(AbstractValidatingLambdaMetafactory.java:233)
    at java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:303)
    at java.lang.invoke.CallSite.makeSite(CallSite.java:302) 

That happens because the pipeline of a Java Stream captures only a raw type, which is the Number class. The Number class doesn't implement an interface IntSupplier itself. This issue can be fixed with an explicit definition of a parameter type in a separated method used as a method reference:

private static int getInt(final IntSupplier i){
    return i.getAsInt();
}

private static <T extends Number & IntSupplier> OptionalInt findMinValue(final Collection<T> values){
    return values.stream().mapToInt(UtilsTest::getInt).min();
}

This example demonstrates an incorrect type inference of the Java compiler and runtime.

Handling of multiple bounds in conjunction with lambdas at compile time and at runtime is not consistent in Java