Resolve Apache Kafka Starting Issue Installed on Single/Multi-Node Cluster
Without integrating Apache Zookeeper, Kafka alone won’t be able to form the complete Kafka cluster.
Join the DZone community and get the full member experience.
Join For FreeThis short article explains how to resolve the error: ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)kafka.common.InconsistentClusterIdException
when we start the Apache Kafka installed and configured on a multi-node cluster. You can read the steps for setting up a multi-node Apache Kafka cluster here.
Without integrating Apache Zookeeper, Kafka alone won’t be able to form the complete Kafka cluster. Because ZooKeeper handles the leadership election of Kafka brokers and manages service discovery as well as cluster topology. It also tracks when topics are created or deleted from the cluster and maintains a topic list. Overall, ZooKeeper provides an in-sync view of the Kafka cluster.
The above-mentioned error is quite common and can be encountered while we are updating the server.properties
file on each Kafka Broker in the existing cluster to add an additional zookeeper server entry because of the new Kafka Broker addition into the cluster. It might be possible to face this issue on a single node Kafka Broker, too, after a system reboot if performed any configuration changes on the Zookeeper’s zoo.cfg
or Kafka’s server. properties
file to replace/update the /tmp
directory location.
The root cause of the issue is when Zookeeper data are stored in the temporary folder and Kafka logs are stored in the persistent folder or vice versa. Ideally, the default value of dataDir
is /tmp/zookeeper
in zoo.cfg
or Zookeeper to store the snapshots, and similarly, the default value of log.dirs
in server.properties
for Kafka Broker is /tmp/kafka-logs
. After the system restarts, files that are stored in that temporary directory get cleaned, and a new cluster ID would be generated due to the complete new registration of the Kafka cluster. Eventually, this configuration mismatch or cluster id number variation causes the above mention error.
There are two ways to resolve the above error quickly, but they are not recommended for the production environment.
Option 1: Update the Cluster ID in the file meta.properties
The meta.properties
file can be located at /tmp/kafka-logs
if the value of log.dirs
key in server.properties
was not updated with a persistent folder location. Open the meta.properties
, the contents will be similar as shown below. Replace the cluster.id
value with the new cluster ID present in the error log and restart the Kafka server.
Option 2: Delete /tmp/kafka-logs/meta.properties
which saved the wrong cluster.id
of last/failed session.
Hope you have enjoyed this read. Please like and share if you feel this composition is valuable.
Published at DZone with permission of Gautam Goswami, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments