APIs Outside, Events Inside

This is an article from DZone's 2022 Enterprise Application Integration Trend Report.

For more:

Read the Report

In the echo chambers of application development, we constantly hear the mantra "API-first," but this slogan has a fundamental flaw: APIs should typically be the last choice when building a distributed application. The correct war cry ought to instead be: "APIs outside, events inside."

This is not only because a well-designed distributed application should always be built using an event-driven architecture, but more importantly, because the needs of an application’s external users are very often the polar opposite of that application’s own internal requirements.  

Real-Time Responses vs. Guaranteed Responses

Real-time responses typically depend upon synchronous communication. On the other hand, passing through the intermediary of a broker to coordinate asynchronous communication adds several steps to the communication chain. REST APIs, which still dominate today’s application development headlines, are of course (almost always) synchronous: You make the call, you hold the line, and you get your response — as long as you don’t receive a timeout error. 

In situations where the external client that called the API is (ultimately) a human, it is easy to understand the desire for realtime responses. However, in the context of a distributed application, the client most often is not a human but instead one of the other software components that work together to form the distributed application as a whole. In this case, the client is a machine, and suddenly our priorities change. 

In a well-architected distributed application (DA), software components must be entirely decoupled from one another. This is because if any constituent software component (SC) becomes unavailable, the other components making up the DA should continue to work normally — robustness being a key goal for any DA. 

Given this fundamental DA requirement, all internal communication between SCs must be asynchronous: It must be "eventfirst." The correct functioning of a DA does not depend upon real-time responses between its components but, instead, upon guaranteed responses between its SCs, and only asynchronous communication can provide the necessary guarantees. 

In order to have fully decoupled components, DAs must respect the Single-Entity-Owner principle.  

Error Responses vs. Error Monitoring

External clients of a distributed application — that is to say users — demand real-time error responses. This is far from surprising. If you launch an operation to buy tickets in an application, for example, you need to know immediately whether the operation succeeded. If the user has no response, and if they are pressed (e.g., to buy tickets), they will launch the same operation again. Then, after waiting a few days, they’ll check their bank statements to see what happened, and then... 

This situation is far from acceptable, which is precisely why external API callers expect to receive error responses in real time, including timeout errors in the case of network problems: the antithesis of asynchronous communication. 

Communication between a distributed application’s software components must conversely be always asynchronous. As such, an internal software component will never wait for an error response. It is not even concerned if errors occur in other SCs because each SC is responsible only for its own internal consistency. When an operation is successfully performed by an SC, other SCs are immediately informed via the publication of a state-changed event to the broker at the heart of the DA. 

If an operation fails, there is quite simply no event published: The state has not changed. This reality gives rise to a need for error monitoring: It’s fine if an SC goes down, but what if it stays down?

Certainty vs. Agility 

It goes without saying that external clients of an application calling the same API version — the same endpoint — with the same input parameters expect to see the same response payload over time. The need of end users for such certainty is once again understandable but stands in stark contrast to the requirements of the DA itself. 

In order for distributed applications to evolve and grow at the speed required in today’s world, those autonomous development teams assigned to each constituent component need to be able to publish often-changing, forward-and-backward-compatible payloads as a single event to the same fixed endpoints using a technique I call "version-stacking." 

Using this technique, development teams can evolve the payloads they publish with complete agility without risking disruption to subscribed components — as long as those SCs use the same technique when collapsing the payloads they receive. 

Stability vs. Flexibility

Clients external to an application prefer if the API catalogue it exposes changes infrequently to avoid frequent changes to their own code base. If new API versions were to be deployed every few months, external clients could not possibly keep their code up to date: They prefer stability.

Conversely, the architects of a distributed application need flexibility; they need to be able to connect new SCs to the DA when they choose as well as be able to swap out existing SCs with more capable replacements without disrupting the ongoing operation of the application as a whole. Using an API-based approach, it takes many months to integrate new components into an existing distributed application, as hundreds of new APIs are incorporated. 

Using an event-first approach and the "Requested Event Pattern" that I described in an earlier article, new SCs can be integrated into the landscape within weeks.

Security vs. Performance

A key concern of architects when exposing their applications to external clients via APIs is — quite rightly — security. Those APIs allow external users to affect changes within the application itself, so they must be rigorously protected, requiring many and frequent authorization steps. These security steps have obvious implications for performance, but regardless, they do seem necessary. 

Necessary, that is, for external clients. Once again, this priority stands in contrast to a distributed application’s own internal needs, considering that those software components making up a distributed application will typically reside within the same network and will absolutely share the same goodwill towards the whole. 

Furthermore, for those SCs to publish or subscribe to topics managed by the event broker — the central nervous system of the DA — they must first be authenticated. 

Once authorized by the broker, any additional security steps become redundant and, therefore, represent an unnecessary hit to performance. Rather than using an API-first approach, the services exposed (internally) by software components should instead be exposed as "Macroservices": event-triggered services operating at the entity-type level.

Conclusion

Although REST APIs first started to appear in the early 2000s, the number of publicly listed REST APIs only passed the 1,000 mark in 2009. Given there are more than 24 times that number today — and given that REST APIs were first "productized" during the 2010s — I believe it is appropriate to refer to the 2010s as the "REST decade" (the 2000s having been the SOAP decade). It’s time for us now to adapt to the 2020s: the "events decade." 

This is an article from DZone's 2022 Enterprise Application Integration Trend Report.

For more:

Read the Report