Evaluating Message Brokers

What Is a Message Broker?

A message broker is an important component of asynchronous distributed systems. It acts as a bridge in the producer-consumer pattern. Producers write messages to the broker, and the consumer reads the message from the broker. The broker handles queuing, routing, and delivery of messages. The diagram below shows how the broker is used in the producer-consumer pattern:

This article discusses the popular brokers used today and when to use them. 

Simple Queue Service (SQS)

Simple Queue Service (SQS) is offered by Amazon Web Services (AWS) as a managed message queue service. AWS fully manages the queue, making SQS an easy solution for passing messages between different components of software running on AWS infrastructure. The section below details what is supported and what is not in SQS

Supported

1. Pay for what you use: SQS only charges for the messages read and written to the queue. There is no recurring or base charge for using SQS.
2. Ease of setup: SQS is a fully managed AWS service, no infrastructure setup is required for using SQS. Reading and writing are also simple either using direct REST APIs provided by SQS or using AWS lambda functions.
3. Support for FIFO queues: Besides regular standard queues, SQS also supports FIFO queues. For applications that need strict ordering of messages, FIFO queues come in handy. 
4. Scale: SQS scales elastically with the application, no need to worry about capacity and pre-provisioning. There is no limit to the number of messages per queue, and queues offer nearly unlimited output. 
5. Queue for failed messages/dead-letter queue: All the messages that can't be processed are sent to a "dead-letter" queue. SQS takes care of moving messages automatically into the dead-letter queue based on the retry configuration of the main queue.

Not Supported

1. Lack of message broadcast: SQS doesn't have a way for multiple consumers to retrieve the same message with its "exactly once" transmission. For multiple consumer use cases, SQS needs to be used along with AWS SNS, which needs multiple queues subscribed to the same SNS topic.
2. Replay: SQS doesn't have the ability to replay old messages. Replay is sometimes required for debugging and testing.

Kinesis

Kinesis is another offering of AWS. Kinesis streams enable large-scale data ingestion and real-time processing of streaming data. Like SQS, Kinesis is also a fully managed service. Below are details of what is supported and what is not in Kinesis.

Supported

1. Ease of setup: Kinesis like SQS is a fully managed AWS service, no infrastructure setup is required.
2. Message broadcast: Kinesis allows multiple consumers to read the same message from the stream concurrently.
3. AWS integration: Kinesis integrates seamlessly with other AWS services as part of the other AWS services.
4. Replay: Kinesis allows the replay of messages as long as seven days in the past, and provides the ability for a client to consume messages at a later time.
5. Real-time analytics: Provides support for ingestion, processing, and analysis of large data streams in real-time.

Not Supported

1. Strict message ordering: Kinesis supports in-order processing within a shard, however, provides no guarantee on ordering between shards. 
2. Lack of dead-letter queue: There is no support for dead dead-letter queue out of the box. Every application that consumes the stream has to deal with failure on its own.
3. Auto-scaling: Kinesis streams don't scale dynamically in response to demand. Streams need to be provisioned ahead of time to meet the anticipated demand of both producer and consumer.
4. Cost: For a large volume of data, pricing can be really high in comparison to other brokers.

Kafka

Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation. Apache is famous for its high throughput and scalability. It excels in real-time analytics and monitoring. Below are details of what is supported and what is not in Kafka.

Supported

1. Message broadcast: Kafka allows multiple consumers to read the same message from the stream.
2. Replay: Kafka allows messages to be replayed from a specific point in a topic. Message retention policy decides how far back a message can be replayed.
3. Unlimited message retention: Kafka allows unlimited message retention based on the retention policy configured.
4. Real-time analytics: Provides support for ingestion, processing, and analysis of large data streams in real-time.
5. Open source: Kafka is an open project, which resulted in widespread adoption and community support. It has lots of configuration options available which gives the opportunity to fine-tune based on the specific use case.

Not Supported

1. Automated setup: Since Kafka is an open source, the developer needs to set up the infrastructure and Kafka cluster setup. That said, most of the public cloud providers provide managed Kafka.
2.  Simple onboarding: For Kafka clusters that are not through managed services understanding the infrastructure can become a daunting task. Apache does provide lots of documentation, but it takes time for new developers to understand.
3. Queue semantics: In the true sense, Kafka is a distributed immutable event log, not a queuing system. It does not inherently support distributing tasks to multiple workers so that each task is processed exactly once.
4. Dynamic partition: It is difficult to dynamically change a number of Kafka topics. This limits the scalability of the system when workload increases. The large number of partitions needs to be pre-provisioned to support the maximum load.

Pulsar

Pulsar is an open-source, distributed messaging and streaming platform. It is an open-source system developed by the Apache Software Foundation.  It provides highly scalable, flexible, and durable for real-time data streaming and processing. Below are details of what is supported and what is not in Pulsar.

Supported

1. Multi-tenancy: Pulsar supports multi-tenancy as a first-class citizen. It provides access control across data and actions using tenant policies. 
2. Seamless geo-replication: Pulsar synchronizes data across multiple regions without any third-party replication tools.
3. Replay: Similar to Kafka, Pulsar allows messages to be replayed from a specific point in a topic. Message retention policy decides how far back a message can be replayed.
4. Unlimited message retention: Similar to Kafka, Pulsar allows unlimited message retention based on the retention policy configured.
5. Flexible models: Pulsar supports both streaming and queuing models. It provides strict message ordering within a partition. 

Not Supported

1. Automated setup: Similar to Kafka, Apache is open-source, and the developer needs to set up the infrastructure.
2. Robust ecosystem: Pulsar is relatively new compared to Kafka. It doesn't have large community support similar to Kafka.
3. Out-of-the-box Integration: Pulsar lacks out-of-the-box integration and support compared to Kafka and SQS.

Conclusion

Managed services require minimal maintenance effort, but non-managed services need regular, dedicated maintenance capacity. On the flip side, non-managed services provide better flexibility and tuning opportunities than managed services. In the end, choosing the right broker depends on the project's needs. Understanding the strengths and gaps of each broker helps developers make informed decisions.