Ultimate Guide to Installing Kafka Docker on Kubernetes
In this post, we discuss hows to get Kafka Docker upon and running on Kubernetes so you can utilize the full power of your cloud-based data.
Join the DZone community and get the full member experience.
Join For FreeIn this ultimate guide I will give you a simple step-by-step tutorial on installing Kafka Docker on Kubernetes. This post includes a complete video walk-through.
There has been a lot of interest lately about deploying Kafka to a Kubernetes cluster. If you are wanting to take the deep dive yourself then you found the right article. Now that we have Kafka Docker, deploying a Kafka cluster to Kubernetes is a snap.
Deploy ZooKeeper to Kubernetes
Kafka relies on ZooKeeper to keep track of its configuration including what topics are available.
Before we deploy Kafka we need to deploy ZooKeeper.
Create a file called zookeeper.yml and add these contents:
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: zookeeper-deployment-1
spec:
template:
metadata:
labels:
app: zookeeper-1
spec:
containers:
- name: zoo1
image: digitalwonderland/zookeeper
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_ID
value: "1"
- name: ZOOKEEPER_SERVER_1
value: zoo1
---
apiVersion: v1
kind: Service
metadata:
name: zoo1
labels:
app: zookeeper-1
spec:
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper-1
This creates a Kubernetes Deployment that will schedule zookeeper pods and a Kubernetes Service to route traffic to the pods. The service has a short name of zoo1 which we will use later when we deploy the Kafka Brokers.
Create the resource:
$ kubectl create -f zookeeper.yml
Now lets start deploying Kafka.
Deploying a Kafka Docker Service
The first thing we need to do is deploy a Kubernetes Service that will manage our Kafka Broker deployments.
Create a new file called kafka-service.yml and add the following contents:
---
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
name: kafka
spec:
ports:
- port: 9092
name: kafka-port
protocol: TCP
selector:
app: kafka
id: "0"
type: LoadBalancer
You might notice that we have set the type to LoadBalancer
. If your Kubernetes Cluster is deployed to bare-metal don't freak out. There is a new Kubernetes add-on called MetalLB that allows this. Checkout my article Kubernetes metallb bare metal loadbalancer for instructions on how to enable it. It will make your life much easier.
Create the service.
$ kubectl create -f kafka-service
Now we need to get the external IP for the service, because we will need it in order to spin up a Kafka Broker in the next section.
$ kubectl describe svc kafka-service
Name: kafka-service
Namespace: default
Labels: name=kafka
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"kafka"},"name":"kafka-service","namespace":"default"},"spec":{"ports...
Selector: app=kafka,id=0
Type: LoadBalancer
IP: 10.105.148.62
LoadBalancer Ingress: 192.168.1.240
Port: kafka-port 9092/TCP
TargetPort: 9092/TCP
NodePort: kafka-port 30718/TCP
Endpoints: 10.44.0.4:9092
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
In the example above, I would note that the LoadBalancer Ingress is set to 192.168.1.240. Now we can start our Kafka Broker.
Deploying the Kafka Broker to Kubernetes
We have the Kubernetes Service deployed but all it does is load balance our Kafka pods which are not deployed yet.
Follow these steps to deploy them.
Create a new file called kafka-broker.yml and add the following contents:
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: kafka-broker0
spec:
template:
metadata:
labels:
app: kafka
id: "0"
spec:
containers:
- name: kafka
image: wurstmeister/kafka
ports:
- containerPort: 9092
env:
- name: KAFKA_ADVERTISED_PORT
value: "30718"
- name: KAFKA_ADVERTISED_HOST_NAME
value: 192.168.1.240
- name: KAFKA_ZOOKEEPER_CONNECT
value: zoo1:2181
- name: KAFKA_BROKER_ID
value: "0"
- name: KAFKA_CREATE_TOPICS
value: admintome-test:1:1
Notice that the KAFKA_ADVERTISED_HOST_NAME is set to the IP address we noted earlier. Also note we tell the Kafka Broker to automatically create a topic admintome-test with 1 partition and 1 replica. You can create multiple topics using the same vernacular and separating them by commas (i.e. - topic1:1:1,topic2:1:1
).
Save the file and create the resource.
$ create -f kafka-broker.yml
You can validate that everything is running.
$ kubectl get pod kafka-broker0
To scale your Kafka Brokers, create another file but give it a different name (i.e. kafka-broker1) and update the ID to match.
Lets test our Kafka Deployment
Testing With KafkaCat
We are going to test our Kafka deployment by using an application called KafkaCat.
To install:
$ apt-get install kafkacat
After the application is installed we will run it in consumer mode (which is the default).
kafkacat -b 192.168.1.240:9092 -t admintome-test
This should not show anything yet because we haven't sent anything to our topic yet...
To send stuff we can copy any text file into our current directory and send it to our Kafka Topic. In another window, run the following command.
$ cat README | kafkacat -b 192.168.1.240 -t admintome-test
You should see the output in the first window which has KafkaCat still running in consumer mode.
Congratulations! You have successfully deployed a Kafka Cluster to Kubernetes.
Published at DZone with permission of Bill Ward, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments