Understanding When to Use RabbitMQ or Apache Kafka

How do humans make decisions? In everyday life, emotion is often the circuit-breaking factor in pulling the trigger on a complex or overwhelming decision. But for experts making complex decisions that have long term consequences, it can’t be pure impulse. High performers typically use the circuit breaker of “instinct,” “gut feeling,” or other emotions only once their expert, unconscious mind has absorbed all the facts required to make a decision.

Today there are dozens of messaging technologies, countless ESBs, and nearly 100 iPaaS vendors in the market. Naturally, this leads to questions about how to choose the right messaging technology for your needs - particularly for those already invested in a particular choice. Do we switch wholesale? Just use the right tool for the right job? Have we correctly framed the job at hand for the business need? Given that, what is the right tool for me? Worse, an exhaustive market analysis might never finish, but due diligence is critical given the average lifespan of integration code.

This post endeavors to give the unconscious, expert mind some even-handed treatment to consider, starting with the most modern, popular choices today: RabbitMQ and Apache Kafka.  Each has its own origin story, design intent, uses cases where it shines, integration capabilities, and developer experience. Origins are revealing about the overall design intent for any piece of software, and make a good starting point.  

Origins

RabbitMQ is a “traditional” message broker that implements a variety of messaging protocols. It was one of the first open source message brokers to achieve a reasonable level of features, client libraries, dev tools, and quality documentation. RabbitMQ was originally developed to implement AMQP, an open wire protocol for messaging with powerful routing features. While Java has messaging standards like JMS, it’s not helpful for non-Java applications that need distributed messaging, which is severely limiting to any integration scenario, microservice or monolithic. With the advent of AMQP, cross-language flexibility became real for open source message brokers.

Apache Kafka is developed in Scala and started out at LinkedIn as a way to make data ingest to Hadoop from Apache Flume easier. Ingesting and exporting from multiple data sources and destinations with tools like Flume meant writing separate data pipelines for each source and destination pairing. Kafka helped LinkedIn standardize the data pipelines and allowed getting data out of each system once and into each system once, making pipelines (and operation) simpler. Kafka is well adopted today within the Apache Software Foundation ecosystem of projects. In particular, it is well integrated with Apache Zookeeper, which forms the backbone of Kafka’s distributed partitions. While many view the requirement for Zookeeper with a high degree of skepticism, it does confer clustering benefits for Kafka users.

Architecture and Design

RabbitMQ is designed as a general purpose message broker, employing several variations of point to point, request/reply, and pub-sub communication styles patterns. It uses a smart broker/dumb consumer model, focused on consistent delivery of messages to consumers that consume at a roughly similar pace as the broker keeps track of consumer state. It is mature, performs well when configured correctly, is well supported (client libraries Java, .NET, node.js, Ruby, PHP and many more languages), and has dozens of plugins available that extend it to more use cases and integration scenarios.

Figure 15 - Simplified overall RabbitMQ architecture (source). 

Communication in RabbitMQ can be either synchronous or asynchronous as needed. Publishers send messages to exchanges, and consumers retrieve messages from queues. Decoupling producers from queues via exchanges ensures that producers aren't burdened with hardcoded routing decisions. RabbitMQ also offers a number of distributed deployment scenarios (and does require that all nodes be able to resolve hostnames). It can be setup for multi-node clusters to cluster federation and does not have dependencies on external services (but some cluster formation plugins can use AWS APIs, DNS, Consul, etc).  

Apache Kafka is designed for high volume publish-subscribe messages and streams, meant to be durable, fast, and scalable. At its essence, Kafka provides a durable message store, similar in some ways to a database, run in a server cluster, that stores streams of records in categories called topics.

Figure 11 - Global Apache Kafka architecture (with 1 topic, 1 partition, replication factor 4). (source).

Every message consists of a key, a value, and a timestamp.  Nearly the opposite of RabbitMQ, Kafka employs a dumb broker and uses smart consumers to read its buffer.  Kafka does not attempt to track which messages were read by each consumer and only retain unread messages; rather, Kafka retains all messages for a set amount of time, and consumers are responsible for tracking their location in each log (consumer state). Consequently, with the right developer talent creating the consumer code, Kafka can support a large number of consumers and retain large amounts of data with very little overhead. As the diagram above shows, Kafka does require external services to run - in this case, Apache Zookeeper, which is often regarded as non-trivial to understand, setup, and operate.    

Requirements and Use Cases

Many developers begin exploring messaging when they realize they have to connect lots of things together, and other integration patterns such as shared databases are not feasible or too dangerous.

Apache Kafka describes itself as a distributed streaming platform but is better known for being a durable storage repository, with good Hadoop/Spark support. The documentation does a good job of discussing popular use cases like Website Activity Tracking, Metrics, Log Aggregation, Stream Processing, Event Sourcing and Commit logs. One of those use cases it describes is messaging, which can generate some confusion.  So let’s unpack that a bit and get some clarity on which messaging scenarios are best for Kafka for, like:

• Stream from A to B without complex routing, with maximal throughput (100k/sec+), delivered in partitioned order at least once.
• When your application needs access to stream history, delivered in partitioned order at least once. Kafka is a durable message store and clients can get a “replay” of the event stream on demand, as opposed to more traditional message brokers where once a message has been delivered, it is removed from the queue.
• Stream processing
• Event Sourcing

RabbitMQ is a general purpose messaging solution, often used to allow web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures while the user waits for the result. It’s also good for distributing a message to multiple recipients for consumption or for balancing loads between workers under high load (20k+/sec).  When your requirements extend beyond throughput, RabbitMQ has a lot to offer: features for reliable delivery, routing, federation, HA, security, management tools and other features.  Let’s examine some scenarios best for RabbitMQ, like:

• Your application needs to work with any combination of existing protocols like AMQP 0-9-1, STOMP, MQTT, AMQP 1.0.
• You need a finer-grained consistency control/guarantees on a per-message basis (dead letter queues, etc.)
• Your application needs variety in point to point, request/reply, and publish/subscribe messaging.
• Complex routing to consumers, integrating multiple services/apps with non-trivial routing logic.
• When integration with your existing IT infrastructure is important, RabbitMQ shines.

RabbitMQ can also effectively address several of Kafka’s strong uses cases above, but with the help of additional software. RabbitMQ is often used with Apache Cassandra when an application needs access to stream history, or with the LevelDB plugin for applications that need an “infinite” queue, but neither feature ships with RabbitMQ itself.

For a deeper dive on microservice, specific use cases with Kafka and RabbitMQ, head over to the Pivotal blog and read this short post by Fred Melo.

Developer Experience

RabbitMQ officially supports Java, Spring, .NET, PHP, Python, Ruby, JavaScript, Go, Elixir, Objective-C, Swift - with many other clients and devtools via community plugins. The RabbitMQ client libraries are mature and well documented.

Apache Kafka has made strides in this area, and while it only ships a Java client, there is a growing catalog of community open source clients, ecosystem projects, and well as an adapter SDK allowing you to build your own system integration.  Much of the configuration is done via .properties files or programmatically.

The popularity of these two options has a strong influence on many other software providers who make sure that RabbitMQ and Kafka work well with or on their technology.

Security and Operations

Both security and operations are strengths of RabbitMQ.  RabbitMQ management plugin provides an HTTP API, a browser-based UI for management and monitoring, plus CLI tools for operators. External tools like CollectD, Datadog, or New Relic are required for longer term monitoring data storage. RabbitMQ also provides API and tools for monitoring, audit and application troubleshooting. Besides support for TLS, RabbitMQ ships with RBAC backed by a built-in data store, LDAP or external HTTPS-based providers and supports authentication using x509 certificate instead of username/password pairs. Additional authentication methods can be fairly straightforwardly developed with plugins.

These domains pose a challenge for Apache Kafka. On the security front, the Kafka 0.9 release added TLS, JAAS role based access control and Kerberos/plain/scram auth, using a CLI to manage security policy. This made a substantial improvement on earlier versions where you could only lock down access at the network level, which didn’t work well for sharing or multi-tenancy.

Kafka uses a management CLI comprised of shell scripts, property files, and specifically formatted JSON files. Kafka Brokers, Producers, and Consumers emit metrics via Yammer/JMX but do not maintain any history, which pragmatically means using a third party monitoring system.  Using these tools, operations is able to manage partitions and topics, check consumer offset position, and use the HA and FT capabilities that Apache Zookeeper provides for Kafka. For example, a 3-node Kafka cluster the system is functional even after two failures. However, if you want to support as many failures in Zookeeper you need an additional 5 Zookeeper nodes as Zookeeper is a quorum based system and can only tolerate N/2+1 failures. These obviously should not be co-located with the Kafka nodes - so to stand up a 3 node Kafka system you need about eight servers.  Operators must take the properties of the ZK cluster into account when reasoning about the availability of any Kafka system, both in terms of resource consumption and design.

Performance

Kafka shines here by design: 100k/sec performance is often a key driver for people choosing Apache Kafka. It is achieved in part by placing a fair amount of responsibility on the developer writing the consumer code.

Of course, message per second rates are tricky to state and quantify since they depend on so much including your environment and hardware, the nature of your workload, which delivery guarantees are used (e.g. persistent is costly, mirroring even more so), etc.

20K messages per second is easy to push through a single Rabbit queue, indeed rather more than that isn't hard, with not much demanded in the way of guarantees. The queue is backed by a single Erlang lightweight thread that gets cooperatively scheduled on a pool of native OS threads - so it becomes a natural choke point or bottleneck as a single queue is never going to do more work than it can get CPU cycles to work in.

Increasing the messages per second often comes down to properly exploiting the parallelism available in one's environment by doing such things as breaking traffic across multiple queues via clever routing (so that different queues can be running concurrently). When RabbitMQ achieved one million message-per-second case it basically came down entirely to doing that judiciously - but was achieved using a lot of resources, around 30 RabbitMQ nodes.  Most RabbitMQ users enjoy excellent performance with clusters made up of anywhere from three to seven RabbitMQ nodes.

Making the Call

Absorb some research on a few of the other top options on the market. If you want to go deeper with the most popular options, a master’s thesis from Nicolas Nannoni inspired this article, and it features a side by side comparison table in section 4.4 (page 39) that is still reasonably accurate two years later- worth the read.  

While researching, loop back with the stakeholders and the business as often as possible. Understanding the business use case is the single largest factor in making the right choice for your situation.  Then, if you are pop psychology fan, your best bet is to sleep on it, let it percolate, and let your instincts take over.  You got this.