Deploying an Apache Kafka mock service with Microcks

Developers are working with new applications every day using Apache Kafka as the backbone to implement an event-driven architecture (EDA) to support distributed systems. However, this adds new challenges when sharing across teams, even within the same organization. What endpoints are available? What is the structure of the message? That’s why payload examples became critical to speed up development. For this reason, having a reliable and enterprise-grade service to mock Apache Kafka should be an item in your EDA checklist. This post will do a quick review of the Microcks General Availability (GA) version and their support to Kafka.

What is Microcks?

Let’s start from the beginning. Microcks is an open source Kubernetes-native platform for API mocking and testing. The project started converting OpenAPI contracts through Postman collections and SOAP UI projects into live mocks and has evolved into a complete platform. Recently, Microcks announced their first GA release in August 2020 and have since published a couple of minor versions. Microcks support open standards for API contract definitions, including SOAP Web services, REST APIs, and event-driven APIs.

The Microcks team is also committed to supporting event-driven APIs and actively participating in improving the AsyncAPI specification with the GA version. The AsyncAPI support is a critical part of solving some of the developer’s initial concerns when working with EDA. 

Event-driven APIs with AsyncAPI

Event-driven APIs highly benefit from a contract-first approach, a process that creates the contract for the service beforehand. The service definition allows developers, either consumers or producers, to know in advance the information of the endpoint, allowing them to work independently and speed up the delivery. It also fosters effective collaboration among teams, users, or partners.

AsyncAPI is an open source initiative that seeks to improve the current state of event-driven architectures. The AsyncAPI specification describes event-driven APIs using messaging technologies like MQTT, AMQP, and Apache Kafka. It started as a sister specification to OpenAPI, using the same syntax and JSON Schema. Like the contracts defined with OpenAPI, AsyncAPI helps you achieve visibility and agreement for your event-driven APIs.

The section of the specification that catches the eye of developers coming from Kafka is the message section. This section defines the content type and structure of a message’s headers and payloads. As you can see, in practice, this part of the specification is for determining the messages’ schema. We can check the structure of the events flowing through Kafka and some examples of such payloads.

Kafka mocks with Microcks

You can use any AsyncAPI editor, like Apicurio Studio, to create your event-driven API in the same way you define an OpenAPI document. Don’t forget to include some examples, as Microcks will use them to send events every specific time into the topics. You can see a simple example in the GitHub repo here.

You will need to configure the additional features. I recommend installing the operator on your favorite Kubernetes distribution as it is straightforward to configure the MicrocksInstall custom resource and let the operator deploy all the components. Add the `async` feature like this:

xxxxxxxxxx

spec:

  ...

  features:

    async:

      enabled: true

      defaultBinding: KAFKA

      defaultFrequency: 10

      kafka:

        install: true

        url: <KUBE_APPS_URL>

After the software installs, you can import the document either by uploading the file directly from the web console or by creating an importer from your git repository. The import will enable the Microcks worker to publish the examples from the specification every 10 seconds as it is the default frequency we configure in the async configuration. 

For more information on the configuration and installation process, you can check the official documentation or watch the following video:

Summary

Apache Kafka is everywhere. We can apply all the learning from the REST APIs contract-first approach to Kafka through event-driven APIs. However, it requires additional tooling to ease the developer journey. The AsyncAPI is here to allow developers to define their Kafka brokers and topics efficiently. This definition includes the structure of the messages and events, and finally, the possibility to add examples. Microcks is a platform that allows developers to import their AsyncAPI specifications. It uses the examples defined within the document to publish events every time so consumers can get a glimpse of the expected payload when connecting to the live service. You can check more on Microcks on their site, and if you find it useful, don’t forget to star the GitHub project’s repository.,