Kafka Consumer Architecture - Consumer Groups and Subscriptions

this article covers some lower level details of kafka consumer architecture. it is a continuation of the kafka architecture , kafka topic architecture , and kafka producer architecture articles.

this article covers kafka consumer architecture with a discussion consumer groups and how record processing is shared among a consumer group as well as failover for kafka consumers.

kafka consumer groups

you group consumers into a consumer group by use case or function of the group. one consumer group might be responsible for delivering records to high-speed, in-memory microservices while another consumer group is streaming those same records to hadoop. consumer groups have names to identify them from other consumer groups.

a consumer group has a unique id. each consumer group is a subscriber to one or more kafka topics. each consumer group maintains its offset per topic partition. if you need multiple subscribers, then you have multiple consumer groups. a record gets delivered to only one consumer in a consumer group.

each consumer in a consumer group processes records and only one consumer in that group will get the same record. consumers in a consumer group load balance record processing.

consumers remember offset where they left off reading. consumers groups each have their own offset per partition.

kafka consumer load share

kafka consumer consumption divides partitions over consumer instances within a consumer group. each consumer in the consumer group is an exclusive consumer of a “fair share” of partitions. this is how kafka does load balancing of consumers in a consumer group. consumer membership within a consumer group is handled by the kafka protocol dynamically. if new consumers join a consumer group, it gets a share of partitions. if a consumer dies, its partitions are split among the remaining live consumers in the consumer group. this is how kafka does fail over of consumers in a consumer group.

kafka consumer failover

consumers notify the kafka broker when they have successfully processed a record, which advances the offset.

if a consumer fails before sending commit offset to kafka broker, then a different consumer can continue from the last committed offset.

if a consumer fails after processing the record but before sending the commit to the broker, then some kafka records could be reprocessed. in this scenario, kafka implements the at least once behavior, and you should make sure the messages (record deliveries ) are idempotent.

offset management

kafka stores offset data in a topic called "__consumer_offset" . these topics use log compaction, which means they only save the most recent value per key.

when a consumer has processed data, it should commit offsets. if consumer process dies, it will be able to start up and start reading where it left off based on offset stored in "__consumer_offset" or as discussed another consumer in the consumer group can take over.

what can kafka consumers see?

what records can be consumed by a kafka consumer? consumers can’t read un-replicated data. kafka consumers can only consume messages beyond the “high watermark” offset of the partition. “log end offset” is offset of the last record written to log partition and where producers writes to next.

“high watermark” is the offset of the last record that was successfully replicated to all partition’s followers. consumer only reads up to the “high watermark”.

consumer to partition cardinality - load sharing redux

only a single consumer from the same consumer group can access a single partition. if consumer group count exceeds the partition count, then the extra consumers remain idle. kafka can use the idle consumers for failover. if there are more partitions than consumer group, then some consumers will read from more than one partition.

notice that server 1 has topic partition p2, p3, and p4, while server 2 has partition p0, p1, and p5. notice that consumer c0 from consumer group a is processing records from p0 and p2. notice that no single partition is shared by any consumer from any consumer group. notice that each partition gets its fair share of partitions for the topics.

multi-threaded kafka consumers

you can run more than one consumer in a jvm process by using threads.

consumer with many threads

if processing a record takes a while, a single consumer can run multiple threads to process records, but it is harder to manage offset for each thread/task. if one consumer runs multiple threads, then two messages on the same partitions could be processed by two different threads which make it hard to guarantee record delivery order without complex thread coordination. this setup might be appropriate if processing a single task takes a long time, but try to avoid it.

thread per consumer

if you need to run multiple consumers, then run each consumer in their own thread. this way kafka can deliver record batches to the consumer and the consumer does not have to worry about the offset ordering. a thread per consumer makes it easier to manage offsets. it is also simpler to manage failover (each process runs x num of consumer threads) as you can allow kafka to do the brunt of the work.

kafka consumer review

what is a consumer group?

a consumer group is a group of related consumers that perform a task, like putting data into hadoop or sending messages to a service. consumer groups each have unique offsets per partition. different consumer groups can read from different locations in a partition.

does each consumer group have its own offset?

yes. the consumer groups have their own offset for every partition in the topic which is unique to what other consumer groups have.

when can a consumer see a record?

a consumer can see a record after the record gets fully replicated to all followers.

what happens if there are more consumers than partitions?

the extra consumers remain idle until another consumer dies.

what happens if you run multiple consumers in many threads in the same jvm?

each thread manages a share of partitions for that consumer group.