Kafka Internals: Topics and Partitions

Let's start discussing how messages are stored in Kafka. In regard to storage in Kafka, we always hear two words: Topic and Partition. 

Topic

A topic is a logical grouping of Partitions.

Partition

A partition is an actual storage unit of Kafka messages, which can be assumed as a Kafka message queue. The number of partitions per topic is configurable while creating it. Messages in a partition are segregated into multiple segments to ease finding a message by its offset. The default size of a segment is very high, i.e., 1GB, which can be configured. Each segment is composed of the following files:

1. Log: Messages are stored in this file.
2. Index: stores message offset and its starting position in the log file.
3. Timeindex: not relevant to the discussion.

Let’s imagine there are six messages in a partition and that a segment size is configured such that it can contain only three messages (for the sake of explanation). Thus the Partition contains these segments as follows:

• Segment – 00 contains 00.log, 00.index and 00.timeindex files
• Segment – 03 contains 03.log, 03.index and 03.timeindex files
• Segment – 06 contains 06.log, 06.index and 06.timeindex files

The segment name indicates the offset of the first message in the segment.

Sample log file:

Starting offset: 0

offset: 0 position: 0 CreateTime: 1533443377944 isvalid: true keysize: -1 valuesize: 11 producerId: -1 headerKeys: [] payload: Hello World
offset: 1 position: 79 CreateTime: 1533462689974 isvalid: true keysize: -1 valuesize: 6 producerId: -1 headerKeys: [] payload: intuit

Sample index file:

offset: 0 position: 0
offset: 2 position: 79

Let’s discuss the time complexity of finding a message in a topic, given its partition and offset.

So, the total complexity is O(1) + O(log (SN, 2)) + O(log  (MN, 2)).

Replication

A topic replication factor is configurable while creating it. Assume there are two brokers in a broker cluster, and a topic, `freblogg,` is created with a replication factor of 2.

Among the multiple partitions, there is one `leader,` and the remaining are `replicas/followers` to serve as backup. Kafka always allows consumers to read only from the leader partition. A leader and follower of a partition can never reside on the same broker for obvious reasons. Followers are always in sync with a leader. The broker chooses a new leader among the followers when a leader goes down. A topic is distributed across broker clusters as each partition in the topic resides on different brokers in the cluster.

Parallelism With Partitions

Kafka allows only one consumer from a consumer group to consume messages from a partition to guarantee the order of reading messages from a partition. So, it's important point to note that the order of message consumption is not guaranteed at the topic level. To increase consumption, parallelism is required to increase partitions and spawn consumers accordingly.