Increasing Resiliency with Circuit Breakers Using Failsafe

It's inevitable that your system will have failures whether they are caused by bugs, network partitions, load, etc. What is important is being able to recover and recovering quickly. There have been some great resiliency libraries released in Java over the past few years. We found Netflix's Hystrix to have a higher learning curve and a bit more complicated to use. We opted for Failsafe which has more of an opt in model to only use what you want. We will outline how to circuit break your web server to help fail fast and recover quicker.

What Is a Circuit Breaker?

The Wikipedia definition of a circuit breaker:

 "A circuit breaker is an automatically operated electrical switch designed to protect an electrical circuit from damage caused by overcurrent, typically resulting from an overload or short circuit. Its basic function is to interrupt current flow after a fault is detected. Unlike a fuse, which operates once and then must be replaced, a circuit breaker can be reset (either manually or automatically) to resume normal operation."

In software, we can think of this as a fail-fast mechanism that will stop a code path once it notices a certain failure rate. This will hopefully give the system time to recover or at least allow other code paths to function while this one is down.

Example Failsafe Circuit Breaker

private static final CircuitBreaker CIRCUIT_BREAKER = new CircuitBreaker()
    // Trigger circuit breaker failure on exceptions or bad requests
    .failIf((HttpServerExchange exchange, Throwable ex) -> {
        boolean badRequest = exchange != null && StatusCodes.BAD_REQUEST == exchange.getStatusCode();
        return badRequest || ex != null;
    })
    // If 7 out of 10 requests fail Open the circuit
    .withFailureThreshold(7, 10)
    // When half open if 3 out of 5 requests succeed close the circuit
    .withSuccessThreshold(3, 5)
    // Delay this long before half opening the circuit
    .withDelay(2, TimeUnit.SECONDS)
    .onClose(() -> log.info("Circuit Closed"))
    .onOpen(() -> log.info("Circuit Opened"))
    .onHalfOpen(() -> log.info("Circuit Half-Open"));

 View on GitHub

Circuit Breaker HttpHandler

Let's utilize a CircuitBreakerto create a reusable HttpHandlerthat can circuit break based on a passed in CircuitBreaker. This could be useful if some resource pool is constrained and we want to help relieve some back-pressure by not sending any new requests. Another use case is potentially we have some bug such as forgetting to paginate a SQL query and just one specific endpoint is hanging non stop. Utilizing a CircuitBreakerwe can automatically start shutting off individual endpoints or maybe an entire service as a whole using middleware. Keep in mind that techniques such as this can hurt you just as much as they help you if they are not configured well.

public class CircuitBreakerHandler implements HttpHandler {
    private static final Logger log = LoggerFactory.getLogger(CircuitBreakerHandler.class);

    private final CircuitBreaker circuitBreaker;
    private final HttpHandler delegate;
    private final HttpHandler failureHandler;

    public CircuitBreakerHandler(CircuitBreaker circuitBreaker, HttpHandler delegate, HttpHandler failureHandler) {
        super();
        this.circuitBreaker = circuitBreaker;
        this.delegate = delegate;
        this.failureHandler = failureHandler;
    }

    @Override
    public void handleRequest(HttpServerExchange exchange) throws Exception {
        Failsafe.with(circuitBreaker)
                .withFallback(() -> failureHandler.handleRequest(exchange))
                // We need to call get here instead of execute so we can return the
                // mutated exchange to run checks on it
                .get(() -> {
                    delegate.handleRequest(exchange);
                    return exchange;
                });
    }
}

 View on GitHub

Example Handlers

Here we have a HttpHandler that we can easily mimic some errors in as well as a HttpHander that returns a 500 server error.

// Actual Circuit Breaker Handler
private static final HttpHandler CIRCUIT_BREAKER_HANDLER =
        new CircuitBreakerHandler(CIRCUIT_BREAKER,
                                  FailsafeWebserver::circuitClosed,
                                  FailsafeWebserver::serverError);

// Handler return a 500 server error
private static final void serverError(HttpServerExchange exchange) {
    exchange.setStatusCode(StatusCodes.INTERNAL_SERVER_ERROR);
    Exchange.body().sendText(exchange, "500 - Internal Server Error");
}

// This handler helps simulate errors, bad requests, and successful requests.
private static final void circuitClosed(HttpServerExchange exchange) {
    boolean error = Exchange.queryParams().queryParamAsBoolean(exchange, "error").orElse(false);
    boolean exception = Exchange.queryParams().queryParamAsBoolean(exchange, "exception").orElse(false);
    if (error) {
        exchange.setStatusCode(StatusCodes.BAD_REQUEST);
        Exchange.body().sendText(exchange, "Bad Request");
    } else if (exception) {
        throw new RuntimeException("boom");
    } else {
        Exchange.body().sendText(exchange, "Circuit is open everything is functioning properly.");
    }
}

 View on GitHub

Making Some Requests

Here is a quick little method to use OkHttp to send a HTTP request to our example server.

private static void request(String message, boolean error, boolean exception) {
    HttpUrl url = HttpUrl.parse("http://localhost:8080")
                         .newBuilder()
                         .addQueryParameter("error", String.valueOf(error))
                         .addQueryParameter("exception", String.valueOf(exception))
                         .build();

    Request request = new Request.Builder().get().url(url).build();
    try {
        log.info(message + " " + HttpClient.globalClient().newCall(request).execute().body().string());
    } catch (IOException e) {
        e.printStackTrace();
    }
}

 View on GitHub

Example Server

Finally, we can start up a server and send some requests at it with a ScheduledExecutorService.

public static void main(String[] args) {

    HttpHandler exceptionHandler =
            CustomHandlers.exception(CIRCUIT_BREAKER_HANDLER)
                          .addExceptionHandler(Throwable.class, FailsafeWebserver::serverError);

    SimpleServer server = SimpleServer.simpleServer(exceptionHandler);
    server.start();


    // Warm-up the circuit breaker it needs to hit at least max executions
    // Before it will reject anything. This will make that easier.
    for (int i = 0; i < 10; i++) {
        request("warmup", false, false);
    }
    ScheduledExecutorService schedExec = Executors.newScheduledThreadPool(1);

    // A simple request that should always succeed
    schedExec.scheduleAtFixedRate(() -> request("ping", false, false), 0, 500, TimeUnit.MILLISECONDS);

    // Send a batch of 15 bad requests to trigger the circuit breaker
    Runnable errors = () -> {
        log.info("Start: Executing bad requests!");
        for (int i = 0; i < 15; i++) {
            request("bad request", true, false);
        }
        log.info("End: Executing bad requests!");
    };
    schedExec.schedule(errors, 1, TimeUnit.SECONDS);

    // Send a batch of 15 requests that throw exceptions
    Runnable exceptions = () -> {
        log.info("Start: Executing requests that throw exceptions!");
        for (int i = 0; i < 15; i++) {
            request("exception request", false, true);
        }
        log.info("End: Executing requests that throw exceptions!");
    };
    schedExec.schedule(exceptions, 5, TimeUnit.SECONDS);
}

 View on GitHub

Output

2018-02-05 23:43:16.168 [main] INFO  c.s.common.undertow.SimpleServer - ListenerInfo{protcol='http', address=/0:0:0:0:0:0:0:0:8080, sslContext=null}
2018-02-05 23:43:16.542 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.549 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.553 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.554 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.556 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.557 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.559 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.561 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.562 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.564 [main] INFO  c.s.e.failsafe.FailsafeWebserver - warmup Circuit is open everything is functioning properly.
2018-02-05 23:43:16.568 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:17.068 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:17.568 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:17.568 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - Start: Executing bad requests!
2018-02-05 23:43:17.569 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.570 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.572 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.574 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.575 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.576 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.577 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request Bad Request
2018-02-05 23:43:17.578 [XNIO-1 I/O-2] INFO  c.s.e.failsafe.FailsafeWebserver - Circuit Opened
2018-02-05 23:43:17.580 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.581 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.583 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.584 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.585 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.587 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.588 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.589 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - bad request 500 - Internal Server Error
2018-02-05 23:43:17.589 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - End: Executing bad requests!
2018-02-05 23:43:18.070 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:18.568 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:19.068 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:19.570 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:20.070 [XNIO-1 I/O-2] INFO  c.s.e.failsafe.FailsafeWebserver - Circuit Half-Open
2018-02-05 23:43:20.071 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:20.573 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:21.068 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:21.569 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:21.570 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - Start: Executing requests that throw exceptions!
2018-02-05 23:43:21.571 [XNIO-1 I/O-2] INFO  c.s.e.failsafe.FailsafeWebserver - Circuit Closed
2018-02-05 23:43:21.572 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.574 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.575 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.576 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.578 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.579 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.581 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.583 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.584 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.585 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.586 [XNIO-1 I/O-2] INFO  c.s.e.failsafe.FailsafeWebserver - Circuit Opened
2018-02-05 23:43:21.587 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.587 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.589 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.590 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.591 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - exception request 500 - Internal Server Error
2018-02-05 23:43:21.591 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - End: Executing requests that throw exceptions!
2018-02-05 23:43:22.071 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:22.572 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:23.072 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:23.573 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping 500 - Internal Server Error
2018-02-05 23:43:24.068 [XNIO-1 I/O-2] INFO  c.s.e.failsafe.FailsafeWebserver - Circuit Half-Open
2018-02-05 23:43:24.069 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:24.573 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:25.069 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:25.572 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:26.068 [XNIO-1 I/O-2] INFO  c.s.e.failsafe.FailsafeWebserver - Circuit Closed
2018-02-05 23:43:26.069 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:26.569 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:27.069 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:27.570 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:28.073 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:28.572 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:29.072 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.
2018-02-05 23:43:29.569 [pool-1-thread-1] INFO  c.s.e.failsafe.FailsafeWebserver - ping Circuit is open everything is functioning properly.