Enterprise Integration Patterns From ESB to ESP and API

Today, when we talk about enterprise application integration, we cannot avoid the pervasively-used enterprise service bus (ESB) pattern, which implements a communication system between mutually interacting software applications in a service-oriented architecture (SOA) and has been adopted in most enterprise applications.

ESB can be used as an integration platform and primarily used in enterprise application integration (EAI) of heterogeneous and complex service landscapes. ESB is the backbone of SOA infrastructure. It has platform-agnostic nature and the ability to integrate with anything IT system with mixed environments.

The advantage to adopt ESB integration can be summarized as:

• Clean decoupled between the service provider and the consumer
• Provided enterprise level security, audit, High Availability, logging, throttling, and service mediation and routing
• Provided event handling, queuing and sequencing, data mapping
• Support multiple protocol binding and data exchange & transformation, such as BPEL, SOAP, WSDL, SXD, XML, REST, JMS, HTTP/HTTPS, FTP, JCA etc.
• Provided low cost and configurable integration platform

Although ESB has been widely adopted in past years, it has been demonstrated over time with some disadvantages:

• Increasing latency during the services communication by introducing a layer of data mapping and data format transformation
• More administration effort on defining message templates and configuring
• Less support on DevOps with centre deployment model
• Single point of failure integration channel
• Complex and difficult to maintain and manage the service orchestration
• High development effort

Facing challenges of ineffective application integrations, modernizing integration has been radically changed to adapt the application and data integration on:

• Business-to-business, application-to-application integration
• Cloud, hybrid cloud and on-premises service integration
• Mobile and IoT application integration
• Event-stream-oriented message integration
• Integrating AI/smart machines and Robotic Process Automation (RPA) software

iPaaS and iSaaS, along with API management and event stream processing technologies are adopted to replace the ESB to increase agility and capabilities required for digital business. With modern integration approaches, integration approaches are no longer associated with centralized solution and teams. Diversified solutions and project teams across the entire business will be engaged in achieving business integration expectations. Two typical approaches, ESP and API, are outstanding out of a range of iPASS and iSaaS infrastructure and application integration offerings.

Event Stream Processing (ESP)

Event stream processing (ESP) is a set of technologies designed to assist the construction of event-driven information systems. ESP technologies include event visualization, event databases, event-driven middleware, and event processing languages, or complex event processing (CEP) and Event ML & analysing.

The event refers to each data segment as well as data streams in the system, and stream refers to the ongoing delivery of those event segments. Actions applied on events include aggregations, transformations, analytics, enrichment, and ingestion.

Cloud-managed ESP services are increasingly popular for new application integration. Like their on-premises integrations, those services don’t target specific use cases or verticals. Instead, they are generic stream processors requiring customization to meet diversified business challenges.

One of the key capabilities around ESP is enabling AI machine learning over event streams. This continues to be an immature area in the ESP market for both content and behavior analytics.

Inside stream processing, the event data is often stored in a database, data warehouse, a file system, cloud data lake, or other forms of mass storage. Applications would query the data or compute over the data as needed.

Some advantages to use ESP are:

• Single platform operating model with standard management tools set for all integrations, including business activity monitoring and reporting capabilities.
• Provide the deep learning on event tracking and analytics.
• Real time or near real time processing

However, there are other cautions to use ESP:

• Development and administration effort and skills are high
• Mixed products to build the integration platform, which could be complex to manage and maintain, and requires a lot of hardware to scale.

ESP is best used in on-premises application integration scenarios including legacy applications, on-premises packaged applications and custom cross-domain applications. It provides integrations at very large scale with low latency and high throughputs. It can also send push notifications to users based on models about their behavior.

Since the series of events data is created as continuous streams over time, such as IoT sensor events, user activity on a website, banking transactions, financial trades, and so on, ESP is usually built together with on-premises data warehouse or data lakes. With machine learning (ML) technologies, event data could be classified, analyzed, and recommended with reporting for business decisions.

Application Programming Interface (API)

An application programming interface (API), also referred as Web APIs, are the defined interfaces through which interactions happen between an enterprise and applications.

Web API approach is an architectural pattern that revolves around providing a program interface to a set of services to different applications serving different types of consumers. APIs also build on existing application architecture based on SOA. It is one of important means that SOA may be designed and delivered.

API is often referred with a Microservices Architecture. By meshing microservices level access, it enables an organization to take advantage of microservice benefits such as web-scale architecture, fast development, agility and speed of deployment.

API management is an integration solution that helps expose service level interfaces for eternal or public consumption in a secure and controlled fashion: In general, it includes:

• Service Gateway (Or Mediation engine)
  • QoS enforcement
  • Security & Access Control
  • Mediation and routing
  • Runtime activity / SLA monitoring
  • Analytics / Reporting
• APIs store
  • API meta information
  • API lifecycle management (Versioning/Publish/Update/Promote/Retiring)
  • Searchable and discovery

Designing APIs should base on the requirements of your organization's ecosystem. It should be designed based on consumer-centric from the outside-in instead of inside-out.

ESB, ESP, and API Integration Pattern Comparing

Comparing with other two patterns, ESB is lack of support from cloud and DevOps technologies. It becomes the coupled single point element during the DevOps process as a center deployment model. The effort to maintain the service orchestration is complex and high. On the other hand, Cloud-based ESP and Web API have the potential to reduce the cost of developing and more flexible on supporting decoupled enterprise agnostic integration.

Due to the business demands for greater responsiveness, mixed integration, and awareness of context in business decisions, organizations have taken a more strategic approach to supporting ESP and Web API. With Gartner note, three patterns can be addressed and compared as below:

Conclusion

In the modern application development, enterprise integration is the key. Digital businesses are looking for technologies addressing new and blended scenarios, integrating heterogeneous enterprise applications, managing human interface devices, integrating with mobile portal, and analysing content and behavior from IoT events, etc. Widely-adopted ESBs alone is no longer fit all scope of initiatives. iPaaS and iSaaS, along with API management and event driven technologies are coming to the solution picture among diversified integration technologies.

However, every solution pattern is strong in a particular business scenario. We need consider the existing infrastructure and enterprise applications features, such as data centric, multi-step process, composite service, event driven, etc. before we choose a flexible and cohesive or mixed solution.