Aggregating REST APIs Calls Using Apache Camel

What is Apache Camel?

As described by the creator itself, Claus Ibsen on Apache Camel in Action, 2nd edition:

Among the most diverse integration solutions that we can build with Camel. In this article, we're gonna build an aggregator for API calls using the Enricher EIP. The project presented in this article is available on Github.

The Scenario

To demonstrate how the Enricher EIP can be used with Camel we're going to build an application that aggregates calls from two APIs:

• Authors API with the endpoints:

  • [GET] /authors

  • [GET] /authors/{name}

• Books API with the endpoint:

  • [GET] /books/{authorId}

Our Camel application has two endpoints:

• [GET] /integration/authors - Query Authors API.

• [GET] /integration/authors/{name} - Search author by name (eg: `jr-tolkien`, `jk-rowling`) and enriches the response with all authors' books.

The Camel Application

Creating Project

To create the project, we can access https://start.spring.io/ and fill the project metadata with these parameters: 

Spring Boot: 2.3.1

Group: br.com.camel.bookstore

Artifact: camel-bookstore-aggregator

Version: 1.0

Name: camel-bookstore-aggregator

Dependencies: 'Apache Camel'

Next, click on Generate to create the project and then edit the pom.xml file to add the dependencies that we're going to use (undertow, rest, http).

The pom.xml should look like the following:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <parent>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-parent</artifactId>
          <version>2.3.1.RELEASE</version>
          <relativePath/> <!-- lookup parent from repository -->
     </parent>

     <groupId>br.com.camel.bookstore</groupId>
     <artifactId>camel-bookstore-aggregator</artifactId>
     <version>1.0</version>
     <name>camel-bookstore-aggregator</name>
     <properties>
          <java.version>1.8</java.version>
          <camel.version>3.1.0</camel.version>
     </properties>

     <dependencies>
          <dependency>
               <groupId>org.apache.camel.springboot</groupId>
               <artifactId>camel-spring-boot-starter</artifactId>
               <version>${camel.version}</version>
          </dependency>        

          <dependency>
               <groupId>org.apache.camel.springboot</groupId>
               <artifactId>camel-rest-starter</artifactId>
               <version>${camel.version}</version>
          </dependency>

          <dependency>
               <groupId>org.apache.camel</groupId>
               <artifactId>camel-undertow</artifactId>
               <version>${camel.version}</version>
          </dependency>

          <dependency>
               <groupId>org.apache.camel</groupId>
               <artifactId>camel-http</artifactId>
               <version>${camel.version}</version>
          </dependency>
       
       ...
       
     </dependencies>
</project>

The Rest Route

We're going to create two classes with Camel routes. The endpoint will be in the RestRoute and the routes that access the APIs and enrich content will be in the RestAggregatorRoute class.

@Component
public class RestRoute extends RouteBuilder {

  @Override
  public void configure() throws Exception {
    //Define the server configurations like host address and port
    restConfiguration()
      .host("0.0.0.0").port(8080)
      .bindingMode(RestBindingMode.auto);
 
    //Mapping the REST endpoints
    rest("/integration")
      //Endpoint that queries all authors
      .get("/authors")
        .route().routeId("rest-all-authors")
        .to("direct:call-rest-all")
      .endRest()
     
      //Endpoint with the enrich EIP
      .get("/authors/{name}")
        .route().routeId("rest-author-by-name")
        .to("direct:call-rest-author")
      .endRest();
  }
}

The endpoints /integration/authors e /integration/authors/{name} when called invoke the routes direct:call-rest-all and direct:call-rest-author respectively, they are defined in the other class.

The Integration Route

Querying the Authors API

Our route uses the http component to consume the Authors API and return all authors.

from("direct:call-rest-all")
  .routeId("all-service")
  .removeHeaders("CamelHttp*")
  .setHeader(Exchange.HTTP_METHOD, constant("GET"))
.to("http://{{authors.url}}/authors");

In the route, we use the removeHeaders("CamelHttp*") ensuring that our API call has only HTTP component-related headers, and we're going to use the HTTP_METHOD header with GET as value. The .to method receives "http://{{authors.url}}/authors" as a parameter, when we use double brackets around authors.url, Camel is able to replace the value from the application.properties, so the file must contain the URL value, for instance:

#application.properties

authors.url=localhost:8081

That is all we need for this route. The Authors API endpoint response is returned straight from the Camel application.

Enriching and Aggregating Author and Books Responses

The direct:call-rest-author route search for the author by name and we use the response to retrieve the authors id, after we enrich the response with the author's books, according to the image:

The code for the route is the following:

from("direct:call-rest-author")
  .routeId("call-rest-services")
  .to("direct:author-service")
    .process(new GetAuthorIdProcessor())
  .enrich("direct:books-service", new JsonRestCallsAggregator());

Let’s approach the parts individually:

• to("direct:author-service"): Like the endpoint that returns all authors, we make a similar call to the authors API:

from("direct:author-service")
  .routeId("author-service")
  .removeHeaders("CamelHttp*")
  .setHeader(Exchange.HTTP_METHOD, constant("GET"))
.toD("http://{{authors.url}}/authors/${header.name}");

When we call the /integration/authors/{name} endpoint from our Camel app, the name passed as path becomes available as a header of the exchange (eg: /integration/authors/jr-tolkien; /integration/authors/jane-austin), because the path is variable depending on the name we have to use .toD instead of the .to method.

The author API response is a JSON with the author information, to retrieve the id we can parse this JSON, but it's located encapsulated inside the exchange message,  that Camel uses to pass data between integrations. In the http component, the response is stored in the body of the In message, like the following image:

More details about the exchange can be found in the JavaDoc: (https://www.javadoc.io/doc/org.apache.camel/camel-api/latest/org/apache/camel/Exchange.html) or in the Camel in Action book (https://livebook.manning.com/book/camel-in-action-second-edition).

• process(new GetAuthorIdProcessor()): For retrieving the id we're using a processor to read the exchange, the GetAuthorIdProcessor class implements org.apache.camel.Processor for that purpose.

public class GetAuthorIdProcessor implements Processor {
     @Override
     public void process(Exchange exchange) throws Exception {
        String author = exchange.getIn().getBody(String.class);
        JsonParser parser = JsonParserFactory.getJsonParser();
        Map<String, Object> jsonMap = parser.parseMap(author);
        String authorId = (String) jsonMap.get("id");   

        exchange.getIn().setHeader("id", authorId);
        exchange.getIn().setBody(author);
    }
}

The process method is responsible for reading the authors' API response, which lies in the In message body. For parsing the json we're using some classes available in the Spring Boot framework. After retrieving the ID, we create a header with the name id and set the author response JSON back in the In message body.

• enrich("direct:books-service", new JsonRestCallsAggregator()): This line is responsible for making the call for the book API and aggregating the responses (author and books) in the class `JsonRestCallsAggregator`.

from("direct:books-service")
  .routeId("books-service")
  .removeHeaders("CamelHttp*")
  .setHeader(Exchange.HTTP_METHOD, constant("GET"))
.toD("http://{{books.url}}/books/${header.id}");

Similar to the direct:author-service call we use the http component to call the book API. Because the endpoint receives an id in the path we use the one that we extract from the authors' response in the process(new GetAuthorIdProcessor()), like in the following image:

The value {{books.url}} should be declared in the application.properties, like for instance:

#application.properties
authors.url=localhost:8081
books.url=localhost:8082

The class responsible for the responses aggregation implements org.apache.camel.AggregationStrategy:

public class JsonRestCallsAggregator implements AggregationStrategy {
    @Override
    public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
        JsonParser parser = JsonParserFactory.getJsonParser();

        String books = newExchange.getIn().getBody(String.class);
        String author = oldExchange.getIn().getBody(String.class);       

        JsonObject authorJson = new JsonObject(parser.parseMap(author));        
        authorJson.put("books", new JsonArray(parser.parseList(books)));

        newExchange.getIn().setBody(authorJson.toJson());
        return newExchange;
    }
}

The aggregate method has two parameters, the exchange before and after the enrich invocation. Using Spring Boot classes to parse and manipulate JSON, we can retrieve the author JSON (oldExchange) and aggregate with the books from the author JSON (newExchange), like in the image:

Our response aggregation will generate a final response JSON with the following format:

{
    "author": {
        "id": "...",
        "name": "...",
        "fullName": "...",
        "books": [ ... ]
    }
}

Handling Failure

With our Camel application running, we can make calls to the /integration/authors/{name} endpoint, and the app are going to make calls to the two APIs and aggregate the result. But what happens if we have a failure in one of the APIs?

Let's cover two cases and refactor our Camel route to handle those failures. 

Failure Scenarios

Book API failure: In our scenario, if we call the book API endpoint (/books/{authorId}) using an authorId that does not have books registered, the API will respond with 404 (Not Found). To avoid errors in our Camel app, we are going to refactor our RestAggregatorRoute class.

The http Camel component already had implemented to throw a HttpOperationFailedException if the HTTP response status code returned 404 or above.

//Without error handler
from("direct:books-service")
  .routeId("books-service")
  .removeHeaders("CamelHttp*")
  .setHeader(Exchange.HTTP_METHOD, constant("GET"))
.toD("http://{{books.url}}/books/${header.id}");


//With error handler
from("direct:books-service")
  .routeId("books-service")
  //Exception handler
  .onException(HttpOperationFailedException.class)
    .handled(true)
    .setBody(constant("[]"))
  .end()
  .removeHeaders("CamelHttp*")
  .setHeader(Exchange.HTTP_METHOD, constant("GET"))
.toD("http://{{books.url}}/books/${header.id}");

With the refactor, every time we have a 404 response from the books API, the Camel app returns a json empty array to complete the enrichment and aggregation.

Authors API failure: In our scenario, if we call the authors API endpoint (/authors/{name}) using a name not registered, the API will return 204 (No Content). If that happens, we must terminate the request at that point, avoiding the enrich and aggregation. The 204 status code is in the success family, so Camel http doesn't throw HttpOperationFailedException. To handle this we can check the response status code, and if it is 204 return another status, like 204 itself, 404, or other status code.

//Without error handler
from("direct:call-rest-author")
  .routeId("call-rest-services")
  .to("direct:author-service")
    .bean("authors", "getId")
  .enrich("direct:books-service", new JsonRestCallsAggregator());


//Variables
private static final int OK_CODE = 200;
private static final int APP_RESPONSE_CODE = 204;

//With error handler
from("direct:call-rest-author")
  .routeId("call-rest-services")
  .to("direct:author-service")
  .choice()
    .when(header(Exchange.HTTP_RESPONSE_CODE).isEqualTo(OK_CODE))
      .bean("authors", "getId")
      .enrich("direct:books-service", new JsonRestCallsAggregator())
  .otherwise()
    .setHeader(Exchange.HTTP_RESPONSE_CODE).constant(APP_RESPONSE_CODE);

With this refactor, every time the authors API returns 204 the Camel app returns the response code defined in the APP_RESPONSE_CODE variable, ending the request, avoiding a call to the books API and a possible JsonParseException.

Conclusion

The Apache Camel is a lightweight framework capable of building the most diverse types of integrations as presented in this article. The Enricher EIP, just like others, can be very handy to build those applications. For more details about each one, you can read the documentation here.

Next Steps

The Camel application implemented in this article has some improvement possibilities like the addition of a new endpoint or APIs, an improvement on the error handling (checking new exceptions), the inverse mapping searching a book, and enriching the response with the author information. I hope the article gave you insights into Camel and motivates you to build integration apps using Apache Camel.