Event-Driven APIs With Webhook and WebSub

With all the recent advancements in the API architecture and the requirements for the real-time web, the event-driven API design is playing a key role. Webhooks and WebSub are two major protocols in event-driven API design. 

So, in this article, I will discuss Webhook and Websub and how to implement a subscriber using Ballerina programming language practically.

What Is Webhook?

A webhook is an API concept that can be thought of as user-defined HTTP callbacks, usually triggered by an event. Webhook is also called a web callback, Reverse API, or HTTP push API. 

Webhooks are used everywhere now, and it notifies you when something interesting is happening. For example, scenarios such as events in source code repo, comments for a blog when a new order is placed, etc. So, that means we don’t have to poll continuously to see whether the interesting event occurred. 

How Does Webhook Work?

Webhooks are a user-defined HTTP Callback-URL, which needs to be registered at the service provider’s platform. That URL is called Webhook endpoint, and it should be public. This URL will be called by the service provider using a POST request to send information about an event to the vendor. The data format, the kind of events the client wants to be notified about, etc can be different depending on the webhook provided.

Webhook vs APIs

With web APIs, clients have to periodically ask the servers to send new data, i.e., request-based. But with webhooks, clients don’t have to call, but the server will call the clients when something new comes up, i.e., event-based.

With web APIs, clients have to ask servers to send new data constantly. With webhooks, a server tells a client: "You don’t have to call. I'll call you when something new comes up." In other words, API calls are request-based, whereas webhooks are event-based. With webhooks, you configure in advance the events under which the server automatically sends data.

What Is WebSub?

WebSub is an open protocol for distributed publish-subscribe communication based on HTTP webhooks. It was previously called PubSubHubbub, PubSub, or PuSH. There are three entities involved in WebSub. 

1. Publisher: The producer that generates the content and publishes it to the Internet.
2. Hub: Acts as an intermediary party between the publisher and subscriber. It accepts the content from the publishers and pushes them to subscribers based on what they are subscribed to.
3. Subscriber: The consumers in WebSub. They find the Hubs advertised by publisher’s topic and subscribe to them to get notifications whenever the publisher publishes new content to the topic they are subscribed to.

Webhook vs WebSub

WebSub can be thought of as webhooks with additional benefits:

• WebSub requires less effort for publishers as they declare the hub they are using and then send the new content to the hub. Then hub will handle publishing that data to all subscribers.
• There is not much difference for subscribers as they still have to make a subscription request, but in WebSub, it is to the hub instead of to the publisher directly. 
• The message sources are decoupled from message consumers in WebSub.
• WebSub allows processing the messages asynchronously.

WebSub is ideal when you have multiple consumers interested in the same message or more than one message producer. 

For example, in an e-commerce website, let’s say we need to update the delivery department, the store database and send a mail to the customer when a customer makes a purchase. If three services do the above things: delivery, store service, and mailing service, we can say all three services are interested in the same message (i.e., purchase details). So, we can publish this message to the hub, and then it will route it to all the subscribed services.

Create Webhook Powered GitHub Integration

Now, let’s explore what we can do with Webhooks using the Ballerina programming language. Webhooks are supported by many popular vendors such as GitHub, Stripe, Twilio, etc. For this article, I am going to GitHub based event notification sample. 

Pre-Requisites:

• GitHub account
• Ballerina Programming Language
• Ngrok

Note: If Ballerina is not installed already, follow the instructions here and install Ballerina. Refer to the following video for more details:

This sample code uses the Ballerina Swan Lake Beta2 version.

Webhook powered GitHub integration has two steps in high-level:

1. Setup how we need our webhook to behave: i.e., what events it should listen to, configure listener URL in client-side, etc.
2. Setup how our client-side service should behave after receiving data

Now, let’s set up the webhook and local client subscriber service step by step.

Step 1: 

If the subscriber service on the client-side is running in the local machine, we need to expose it to the public network so that server can access that URL. For testing purposes, we can use Ngrok as it exposes local servers behind NATs and firewalls to the public internet over secure tunnels.

For example, if we plan to run the subscriber service in port 9090, run the below command after installing Ngrok.

./ngrok http -bind-tls=true 9090

You can see an output such as follows and highlighted URL will be the callback URL of subscriber service.

Step 2:

Register GitHub webhook for a repository where you want to receive notifications.

Go to repository home page (i.e. https://github.com/<ORG_NAME>/<Repo_NAME>)  -> Settings -> Webhooks -> Click the Add Webhook button.

Fill in the information as follows and click the "Add webhook" button.

• Payload URL : https://fc58-112-134-161-227.ngrok.io/events (This is <ABOVE_NGROK_URL/<Service_path_name of ballerina service that we are going to create>).
• Content-Type: application/JSON
• Select "send me everything."

Step 4:

Create the subscriber_service.bal file and add the following code. Configure the SubscriberServiceConfig properties target, callback and token parameters accordingly.

import ballerina/websub;
import ballerina/io;

@websub:SubscriberServiceConfig {
    target: [
        "https://api.github.com/hub", 
        "https://github.com/anupama-pathirage/BallerinaDemo/events/*.json" //Add teh correct path to required repo
    ],
    callback: "https://fc58-112-134-161-227.ngrok.io", // Add the ngrok URL 
    httpConfig: {
        auth: {
            token: "ghp_xxxxxxxxxxxxxxxxx" //GitHub PAT
        }
    }
}
service /events on new websub:Listener(9090) {
    remote function onEventNotification(websub:ContentDistributionMessage event) 
                        returns error? {
        var retrievedContent = event.content;
        if (retrievedContent is json) {
            if (retrievedContent.action is string){
                io:println("Action performed: ", retrievedContent.action);
            }
        } else {
            io:println("Unrecognized content type, hence ignoring");
        }
    }
}

Step 5:

Run the .bal file using the following command:

bal run subscriber_service.bal

Output is as follows: That means service successfully started in port 9090.

Compiling source

        test.bal

Running executable

time = 2021-09-12T02:44:21.210+05:30 level = WARN module = ballerina/websub message = "HTTPS is recommended but using HTTP"

[ballerina/http] started HTTP/WS listener 0.0.0.0:9090

Step 6:

Now go to the repo where we added the webhook and do some changes such as star/unstar repo, create/close issue, delete a label, etc.

Then, you will see the event is received to the onEventNotification method, and the complete output after a few such actions is as follows:

Action performed: reopened

Action performed: created

Action performed: started

Action performed: closed

Now, you have successfully configured a webhook for GitHub and received the events using a subscriber written in Ballerina programming language. 

Summary

Webhooks is a good choice for the event-driven notification system when the consumer needs to be notified on every event occurrence with their account at the providers' platform.

WebSub is a good choice for use-cases such as broadcasting public information/data from a single publisher (or a small number of publishers) to millions of subscribers via the hub. 

In this article, we implemented a WebSub subscriber service using the Ballerina websub module. Also, Ballerina has websubhub module, which provides APIs for WebSub Hub service and WebSub Publisher client. 

Refer to the following resources for more details:

• The websub module.
• The websubhub module.