Event-Driven Orchestration: Effective Microservices Integration Using BPMN and AMQP

In today’s IT world, microservices architecture becomes attractive. Comparing with the traditional monolithic application, microservices provides various benefits, such as ideal fit for agile methodology, native to cloud-based development, greater separation of concern, higher scalability and flexibility, and faster turnaround. However, the business organizations will face the following challenges when migrating the monolithic enterprise application into microservices:

• Increased complexities in the integration of the distributed services
• Difficulties in managing distributed transactions that span more than one services
• Information barriers that lack an end-to-end governance of the business flow across the services

In this article, with the study of a sample “submit shopping cart” business process, we will discuss the two classic microservice integration patterns: service choreography and orchestration. By evaluating the pros and cons of each approach, we will go a step further to look at a proposed solution: event-driven orchestration. We will investigate the design practices of building a loose-coupled and high-reusable application using Spring Boot, Camunda Business Process Model and Notation (BPMN) engine and RabbitMQ, a message broker based on Advanced Message Queuing Protocol (AMQP). A fully working sample project can be found at GitHub.

Example Use Case of “Submit a Shopping Cart”

Let’s look at a typical “submit shopping cart” business process which has five main steps:

1. Validate shipping address
2. Reserve payment
3. Allocate inventory
4. Place order
5. Notify customer

The business process will execute the five steps in sequence.  In case of exception during the workflow, rollback should be executed to maintain the transaction’s ACID (atomic, consistent, isolated, and durable).

How can this “submit shopping cart” business process be implemented as a group of interactions between the microservices?

Solution Option 1: Event-Driven Service Choreography

One possible solution is to follow the microservice principle “Smart Endpoint and Dumb Pipe” and build a decentralized service choreography.

As shown in Figure1 below, Services are connected to a message bus and subscribe events they are interested in. Once an event occurs that matters of the service, the service performs the appropriate action. The “submit shopping cart” process will be initiated by an event from a front-end user and be processed by the involved microservices in a choreography mode. This approach provides high cohesive and low coupling.

Figure 1. Service Choreography using Message Broker.

However, this approach has the following disadvantages:

• Difficult to maintain: Due to the decentralized solution, the business flow is spreading across multiple services. Any business change to the process flow will lead to changes in multiple services.
• Difficult to manage: The runtime state of the process instance is stored separately.
• Difficult to rollback: The business process is choreographed by multiple services which leave the transaction’s ACID becomes extremely difficult.

Solution Option 2: Centralized Service Orchestration

Another solution is to implement a centralized orchestration service using BPMN workflow and REST integrations. As shown in Figure2, a “Shopping Cart Microservice” is implemented as centralized orchestration for the shopping cart process.

Figure 2. Service Orchestration using BPMN and REST.

Using a BPMN process engine and the REST integrations, the problems of the option 1 are addressed:  

• Easy to maintain: the business flow is modeled as graphical BPMN process in a centralized orchestration microservice. The standard Business Process Model and Notation (BPMN) grants the businesses with the capability of understanding their internal business procedures in a graphical notation.  Furthermore, organizations can better communicate and report these procedures in a standard manner. Any further changes to the process flow will be mitigated by implementing the workflow diagrams.
• Easy to manage: BPMN process engine keeps track of all process instances, their state, audit and statistics information.
• Easy to rollback: Transaction's ACID can be guaranteed by the compensation flow as the BPMN out-of-the-box feature.

Unfortunately, this approach also has the drawbacks:

• Tight-coupling: The orchestration pattern must build and maintain a point-to-point connection between the services. Point-to-point means that one service calls the API of another service which results in a mesh of communication paths between all services. Integrating, changing or removing services from the service repository will be a hard task since you must be aware of each connection between the services.
• Complexity in building service adapters: Orchestration service need to develop adapters to the peer services, which must maintain all details of the service communications, such as service location, service interfaces, and data model translation. Whenever a peer service changes, the adapter will be impacted.

What is the better solution?

Proposed Solution: Event-Driven Service Orchestration

A solution of event-driven with centralized orchestration is proposed. As shown in Figure3, the shopping cart service is implemented as an orchestration service using BPMN workflow, but the service adapters are eliminated through a common out-of-the-box AMQP adapter. An AMQP event bus is designed as the message broker to decouple the services.

Figure 3. Event-Driven Orchestration using BPMN and AMQP.

As illustrated in Figure 3, this approach consists of the following architectural components:

Centralized Orchestration Service Using BPMN Workflow

The centralized orchestration service is composed by the following sub-components:

A BPMN Enabled Spring Boot Application

@SpringBootApplication 
@EnableProcessApplication
public class ShoppingCartServiceApplication {
    public static void main(String... args) {
        SpringApplication.run(ShoppingCartServiceApplication.class, args);
    }
}

BPMN Workflows

The main workflow of “submit shopping cart” is shown as below:

Figure 4. Sample BPMN workflow for “submit shopping cart.”

The compensation flow for transaction rollback to handle exceptions:

Figure 5. Sample BPMN compensation flow for exceptions.

BPMN Workflow Activities

The workflow activities are designed as Spring components that implement the Camunda JavaDelegate interface. By using the out-of-the-box AMQP adapter and the common event model, the business flow and its activities will be able to be simplified to focus on its business logic. A sample of reserve payment activity of invoking the payment service through AMQP “RPC” is shown as below:

@Component
public class ReservePaymentActivity implements JavaDelegate {
    . . . 
    public static final String SERVICE_ACTION = "reserve";

    @Autowired
    AmqpRpcClient amqpRpcClient;

    @Override
    public void execute(DelegateExecution execution) throws Exception {
        BusinessEntity sc = (BusinessEntity) execution.getVariable(ProcessConstants.VAR_SC);
        ServiceResponse response = amqpRpcClient.invokeService(
        ProcessUtil.buildServiceRequest(sc, ProcessConstants.SERVICE_NAME_PAYMENT, SERVICE_ACTION));
        ProcessUtil.processResponse(execution, response);
        execution.setVariable(ProcessConstants.VAR_PAYMENT_RESERVED, true);
    }
}

AMQP Event Bus

To decouple the integrated services and eliminate the complexity of service adapter implementation, the event-driven architecture is introduced. As illustrated in Figure 6, RabbitMQ is chosen to build an AMQP event bus.

RabbitMQ is a lightweight, reliable, scalable and portable message broker. Unlike JMS, RabbitMQ provides the communications between the services via a platform-neutral, wire-level protocol: Advanced Message Queuing Protocol (AMQP).

Figure 6. Event-Driven Architecture using AMQP.

To design a loosely-coupled and highly-reusable solution, the following design practices are proposed:

Canonical Message Model

A common canonical message model is designed for the communication across the microservices. In the sample project, the message models are designed as unified common models using JSON schema.

Below is a sample “Service Request” model which relies on a common data model as “Business Entity”. The Business Entity can be modeled for any real entities, such as a shopping cart, a product, a payment item, or an address, etc.

In a real project, the industry standard domain models are recommended, such as TM Forum SID for the telecommunications industry.

{
  "$schema": "http://json-schema.org/draft-06/schema#",
  "id": "ServiceRequest.json",
  "title": "Service Request",
  "type": "object",
  "javaType": "com.microservice.orchestration.demo.entity.ServiceRequest",
  "additionalProperties": false,
  "properties": {
    "id": {
      "type": "string"
    },
    "serviceName": {
      "type": "string"
    },
    "serviceAction": {
      "type": "string"
    },
    "createdDate": {
      "type": "string"
    },
    "createdBy": {
      "type": "string"
    },
    "items": {
      "type": "array",
      "items": {
        "$ref": "BusinessEntity.json#"
      }
    }
  }
}

Out-of-the-Box AMQP Adapter

As shown in Figure 6, a generic AMQP adapter is designed as a reusable component for the microservices. The adapter supports two flavors of service interactions: Pub/Sub and RPC.

The adapter hides the details of AMQP configurations and provides the out-of-box components as below:

• Components for event producer
  • AmqpPublisher: a spring bean publishes the event to the AMQP event bus.
  • AmqpRpcClient: a spring bean invokes API against peer service synchronously by sending a request to AMQP bus and getting the response.
• Components for event consumer
  • AmqpSubscriber: a spring bean consumes the event from the AMQP event bus and executes the business logic asynchronously.
  • AmqpRpcServer: a spring bean executes the business logic for a request from the client and responds back to client synchronously.

Configuration-Based AMQP Event Routing

The event producer service will specify the AMQP “exchanges” that are used to invoke a service or publish a message. By passing the configurations from microservice instance to the pre-built AMQP adapter, the event producer can be instantiated without any coding effort.

Below is the sample configuration for the shopping cart service as an event producer: the application.yaml configuring the exchanges for Pub/Sub and RPC.

spring:
  rabbitmq:
exchange:
  rpc: shoppingcart.rpc  
  pub: shoppingcart.pub

The event consumer will configure the message routings in the pattern of “Exchange”->”Routing Key”->”Event Queue”.This pattern can be better understood and maintained.  

Below is the sample configuration for the Location Service as an event consumer: the event routing paths are configured in a simple and intuitive way. The "serviceRoutings" defines the event paths for the synchronous RPC event flows, and the "eventRoutings" defines the event flows for the asynchronous Pub/Sub.

spring:
  rabbitmq:
    serviceRoutings:
    - shoppingcart.rpc->LocationService.*->LocationServiceQueue
    - customer.rpc->LocationService.*->LocationServiceQueue
    eventRoutings:
    - shoppingcart.pub->LocationService.*->LocationEventQueue
    - customer.pub->LocationService.*->LocationEventQueue

Service Providers With Injectable Event Handler

In the event-driven architecture, the microservices that are providing the business functions are registered as AMQP event consumers.

The event consumer services will serve the business function by implementing the “EventHandler” interface. This event handler service will be injected into the pre-built AMQP consumer. By doing so, the common AMQP adapter is decoupled from the business details of the actual event consumer service.

A sample event handler of the Order Service looks like as below:

@Service
public class OrderServiceEventHandler implements EventHandler {
. . . 

    @Override
    public ServiceResponse processRequest(ServiceRequest request) {
        //Dispatch service request to worker
        . . . 
        ServiceResponse response = OrderServiceDelegate.invokeService(request);
        return response;
    }

    @Override
    public void processEvent(ServiceRequest request) {
        //Dispatch service event to handler
        OrderServiceDelegate.handleEvent(request);
    }
}

Conclusion

This article discusses the widely-used approaches of microservices integration: service choreography and orchestration. By analyzing the pros and cons of each way, a better solution of event-driven orchestration is proposed.

A sample project can be found in GitHub using Spring Boot, the Camunda BPMN process engine, and the Rabbit MQ. Looking at the sample code, we have discussed the design practices in building the event-driven solution as loosely coupled, highly reusable, more manageable and maintainable. Moreover, the centralized orchestration using BPMN allows us to manage the complex business flows and the distributed transactions in an intuitive way.