RESTful Applications in An Event-Driven Architecture

I love REST-based applications, especially when I am developing a new app and I need to get some data from another application. Getting a response from their team that their application has a REST API that is well documented and accessible via Swagger API is the best thing ever... besides having some sunny days after a freezing winter in NYC.

You take a look at the Swagger page, figure out the REST endpoints you need to hit to get the data you need and you are good to go! But as you start to work on integrating these REST calls in your new shiny application, you need to build yet another integration. You realize you have done this a few times already before. You have worked on numerous existing and new applications that need to either share data with other apps or consume data from them. You have built these one-to-one integrations so many times that it’s a piece of cake for you now.

Though while it’s easy, it does take your valuable time and resources. You have to communicate with the other team, read the Swagger documentation, create JIRA tickets, have sprint meetings, test the API, implement failure logic (retries/time-outs) in case the endpoint is not available or is taking too long, and most annoyingly, poll for data at regular intervals.

There are many use cases where REST is just the ideal way to build your applications/microservices. However, increasingly, there is more and more demand for applications to become real-time. If your application is customer-facing, then you know customers are demanding a more responsive, real-time service. You simply cannot afford to not process data in real-time anymore. Batch processing (in many modern cases) will simply not be sufficient.

RESTful services, inherently, are polling-based. This means they constantly poll for data as opposed to being event-based where they are executed/triggered based on an event. RESTful services are akin to the kid on a road trip continuously asking you “are we there yet?”, “are we there yet?”, “are we there yet?”, and just when you thought the kid had gotten some sense and would stop bothering you, he asks again “are we there yet?”

Of course, there are some calls that must be synchronous. For example, if your app is responsible for opening a bank account, you must run some security/background checks before approving the account. However, other services, such as the one responsible for updating customer addresses, don’t need to provide a response immediately.

Advantage of Being Real-Time

Embrace yourself for some nostalgia. Remember those pre-covid days when we used to be able to fly? My favorite example is Delta’s mobile app. If you have flown Delta recently, then you know what I am talking about. Delta has one of the best apps of all the airlines I have flown. As soon as I am eligible for early check-in, I get a push notification with confirmation. As I pass the security, the app keeps me informed of the plane’s status. If the gate gets changed last-minute, the information is updated in real-time and not after some regular interval. (Because it would do me no good to know my gate changed from C-1 to B-10 30 mins later if my flight is about to leave in 20 mins). If you check in your bag right before boarding the flight because the plane is full, your app will tell you your bag just got scanned. And finally, as soon as you land, you will be notified which baggage carousel to pick up your bag from!

The key here is not the information is made available to the customer but it is made available in real-time as I move through the airport to my flight. That’s what makes a difference to the user experience and while it may not seem like a differentiator, I have definitely picked Delta over other airlines simply due to their excellent mobile app and user experience.

To be able to provide such an experience, applications need to implement event-driven architecture using event brokers (such as Solace PubSub+) and streaming APIs.

With such an architecture, your applications will publish data to your event broker in real-time and your broker will be responsible for routing those events to any downstream subscriber(s). For example, as soon as I check-in at the airport, an event can be published to a well-defined topic: delta/customer/{customerID}/{airportID}/checkedIn. That’s all that specific application needs to do. It doesn’t need to worry about calling some REST endpoint, waiting for it to reply back, figuring out failure/retry logic, etc. And, in the future, if it needs to publish the same event to another app, it doesn’t need to worry about integrating with another REST endpoint. In fact, it doesn’t need to do anything. The downstream subscriber simply subscribes to the topic and gets the event.

Once the event has been published, it is the router’s job to ensure it is delivered to all the downstream subscribers interested in this event. This is known as the pub/sub messaging pattern. Solace PubSub+ broker supports guaranteed messaging which means zero message loss. If your subscribers are online, they can subscribe using wildcards such as delta/customer/> or delta/customer/*/JFK/> and the event(s) will be pushed to them. If one or more subscribers are not online for some reason, events will be enqueued in queues with ordering preserved. Whenever the subscriber comes online, it will simply consume those messages and pick up where it left off.

These are just some of the benefits of implementing the event-driven architecture. There are plenty more such as scalability, agility, high availability, and efficiency (via pub/sub). However, as with anything, make sure to do a deep-dive, document all the requirements, and see if event-driven architecture makes sense for your use case.

Coexistence of REST APIs and Streaming APIs

As I mentioned earlier, REST APIs are not going anywhere. There are many use cases that don’t need the additional complexity of an event broker and streaming APIs. For example, many user-facing applications should be REST-based as it is the standard for web-based applications. For example, if I am accessing a website, I am simply submitting a GET request. If I am entering some user information via a form, it’s a POST request. If I am scanning a QR code to submit a payment, it is just another RESTful endpoint behind the scenes. The user will submit a request over REST which most likely goes to some API gateway and is then routed to a broker. In this case, our publisher is REST-based. However, our downstream subscribers can choose to use REST, streaming APIs, or both!

This is another benefit of implementing event-driven architecture via Solace PubSub+ event broker. It supports a variety of APIs and open protocols which means your applications can choose which language (i.e., Java, Python, C, etc.) or protocol (i.e., MQTT, AMQP, SMF, etc.) to use.

To support such a hybrid architecture, you will need an event broker that supports REST out-of-box without any plugins or proxy services. It will need to handle protocol translation from REST to other open protocols such as MQTT and AMQP reliably.

If your use case does lend to full evolution from REST to streaming then having a broker which supports REST protocol natively will make it much easier for you to migrate your services. You can select one service at a time to support streaming and others can continue using REST.

In a subsequent post, I will demo a hybrid architecture such as the one shown above. All you need to know for now is that as you consider moving to an event-driven architecture, you don’t and shouldn’t necessarily get rid of all your RESTful services. Your evolved architecture might be a hybrid one!