DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Building Scalable AI-Driven Microservices With Kubernetes and Kafka
  • Big Data Realtime Data Pipeline Architecture
  • Streaming Data Pipeline Architecture
  • Kafka: Powerhouse Messaging

Trending

  • How Businesses Use Modern Development Platforms to Streamline Automation
  • LLMops: The Future of AI Model Management
  • Microservices with .NET Core: Building Scalable and Resilient Applications
  • Fetching Data From One-to-One, One-to-Many, and Many-to-Many Relationships With jOOQ
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Kafka SSL Client Authentication in Multi-Tenancy Architecture

Kafka SSL Client Authentication in Multi-Tenancy Architecture

By 
Swarnava Chakraborty user avatar
Swarnava Chakraborty
·
Apr. 21, 20 · Tutorial
Likes (4)
Comment
Save
Tweet
Share
13.5K Views

Join the DZone community and get the full member experience.

Join For Free

Apache Kafka is the key product for not only messaging transformations but also real-time data processing, in addition to many other use cases. Architectures hosted inside the cloud claim to be secure in terms of communication and providing general security. But when it comes to the multiple client/consumer communication from a server/producer, Kafka provides in-built support for SSL as well as user-based authentication. In the below article, we will set up such an authentication mechanism step-by-step.

Kafka Server-Client SSL Authentication

The solution is divided into three parts: 

  1. SSL support for one or more brokers: Generate the key and the certificate for each machine in the cluster. You can use Java's KeyTool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.
  2. Kafka Configurations (We used Kafka 2.11-2.3.0).
  3. Running the whole set up.

Instructions to Install This Use Case

SSL support for one or more brokers. We will use Java's key tool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.

We are going to use one Kafka server and two clients (consumers). Also, here, we are using self-signed certificates. Otherwise, we need to have TrustStore and KeyStore JKSs for each server.

Points to note: 

  1. Please create a folder for creating and keeping all cert files.
  2. Please provide identical details and passwords for all. In my case I have used:
Plain Text 
xxxxxxxxxx
1
 
1 
issuer=C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = 


Only the CN for client2 I have given a different user for testing purposes. Please generate your certificate carefully, else there will be a problem in the next part.

Shell 
xxxxxxxxxx
1
37
 
1 
keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA
2
3 
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
4
5 
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert 
6
7 
keytool -keystore kafka.client1.truststore.jks -alias CARoot -import -file ca-cert
8
9 
keytool -keystore kafka.client2.truststore.jks -alias CARoot -import -file ca-cert
10
11 
keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
12
13 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial 
14
15 
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
16
17 
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed
18
19 
keytool -keystore kafka.client1.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA
20
21 
keytool -keystore kafka.client1.keystore.jks -alias localhost -certreq -file cert-file
22
23 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial
24
25 
keytool -keystore kafka.client1.keystore.jks -alias CARoot -import -file ca-cert
26
27 
keytool -keystore kafka.client1.keystore.jks -alias localhost -import -file cert-signed
28
29 
keytool -keystore kafka.client2.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA
30
31 
keytool -keystore kafka.client2.keystore.jks -alias localhost -certreq -file cert-file
32
33 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial 
34
35 
keytool -keystore kafka.client2.keystore.jks -alias CARoot -import -file ca-cert
36
37 
keytool -keystore kafka.client2.keystore.jks -alias localhost -import -file cert-signed


Once everything is generated, you can see the generated files:

Plain Text 
xxxxxxxxxx
1
 
1 
λ ls
2
3 
ca-cert ca-key cert-signed kafka.client1.truststore.jks kafka.client2.truststore.jks kafka.server.truststore.jks ca-cert.srl cert-file kafka.client1.keystore.jks kafka.client2.keystore.jks kafka.server.keystore.jks


Kafka Configuration

Chang the server.properties file with below lines:

Properties files 
xxxxxxxxxx
1
 
1 
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093


Also add:

Properties files 
xxxxxxxxxx
1
14
 
1 
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
2 
ssl.endpoint.identification.algorithm=
3 
ssl.keymanager.algorithm=SunX509
4 
ssl.keystore.location=<p>/kafka.server.keystore.jks
5 
ssl.keystore.password=<p>
6 
ssl.keystore.type=JKS
7 
ssl.protocol=TLS
8 
ssl.trustmanager.algorithm=PKIX
9 
ssl.truststore.location=<p>/kafka.server.truststore.jks
10 
ssl.truststore.password=<p>
11 
ssl.truststore.type=JKS
12 
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
13 
allow.everyone.if.no.acl.found=true
14 
ssl.client.auth=required


Then, we can create the necessary new files: client-ssl.properties, client-ssl1.properties, and client-ssl2.properties inside kafka_2.11-2.3.0\config.

Properties files 
xxxxxxxxxx
1
 
1 
security.protocol=SSL
2 
ssl.truststore.location=<p>/kafka.client1.truststore.jks
3 
ssl.truststore.password=<p>
4 
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
5 
ssl.truststore.type=JKS
6 
ssl.endpoint.identification.algorithm=
7 
ssl.keystore.location=<p>/kafka.client1.keystore.jks
8 
ssl.keystore.password=<p>
9 
ssl.key.password=<p>


Properties files 
xxxxxxxxxx
1
 
1 
security.protocol=SSL
2 
ssl.truststore.location=<p>/kafka.client1.truststore.jks
3 
ssl.truststore.password=<p>
4 
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
5 
ssl.truststore.type=JKS
6 
ssl.endpoint.identification.algorithm=
7 
ssl.keystore.location=<p>/kafka.client1.keystore.jks
8 
ssl.keystore.password=<p>
9 
ssl.key.password=<p>


Properties files 
xxxxxxxxxx
1
 
1 
security.protocol=SSL
2 
ssl.truststore.location=<p>/kafka.client2.truststore.jks
3 
ssl.truststore.password=<p>
4 
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
5 
ssl.truststore.type=JKS
6 
ssl.endpoint.identification.algorithm=
7 
ssl.keystore.location=<p>/kafka.client2.keystore.jks
8 
ssl.keystore.password=<p>
9 
ssl.key.password=<p>


First, run Kafka and ZooKeeper:

Shell 
xxxxxxxxxx
1
 
1 
 Zkserver
2
3 
 .\bin\windows\kafka-server-start.bat .\config\server.properties


Then, open a new terminal and create a new topic:

Shell 
xxxxxxxxxx
1
 
1 
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --topic test1 --partitions 1 --replication-factor 1
2
3 
.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181


After this, check the created certificate:

Shell 
xxxxxxxxxx
1
 
1 
openssl s_client -debug -connect localhost:9093 -tls1


It will return the following details at the end. It means your certificate is generated properly.

Shell 
xxxxxxxxxx
1
17
 
1 
Wtmz24ChQdgNcygKXLq1AHgDetoHz57hrx5f75/gh31nDdgHpv4xKyO40TSIH+8v
2
3 
PqgvvrogH0lgLCwsJfqwPEJbWZjL6pvLsBfPB8NMICMXpL50ZA==
4
5 
-----END CERTIFICATE-----
6
7 
subject=C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c
8
9 
issuer=C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c@technaura.com
10
11 
 ---
12
13 
Acceptable client certificate CA names
14
15 
C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c@technaura.com
16
17 
Client Certificate Types: ECDSA sign, RSA sign, DSA sign


We will run the set up for three different scenarios, i.e. without authentication, only server-side authentication, server, and client-side authentication.

Running the Whole Setup

The command for producing using console producer:

.\bin\windows\kafka-console-producer.bat --broker-list <broker host:port> --topic <topic-name> --producer.config config\<config file>

The command for consuming using console consumer:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server <server host:port> --topic <topic-name> --consumer.config config\<config file>

Without Authentication

Producer

Running Kafka producer

Consumer

Running Kafka consumer


Only Server-Side Authentication

Created another topic "test2".

Producer

Creating Kafka producer

Consumer 1

Creating first Kafka consumer

Consumer 2

Creating second Kafka consumer

Note: Please check the used config files.

Server and Client-Side Authentication

For authorization of topic – ‘test2’ only for the user – swarnava.c, use the two following commands:

Shell 


xxxxxxxxxx
1
 
1 
.\bin\windows\kafka-acls.bat --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:"CN=swarnava.c,OU=consulting,O=Technaura,L=Mainz,ST=RF,C=de" --cluster --producer --topic test2


Shell 


xxxxxxxxxx
1
 
1 
.\bin\windows\kafka-acls.bat --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:"CN=swarnava.c,OU=consulting,O=Technaura,L=Mainz,ST=RF,C=de" --group=* --consumer --topic test2


Authorization of current topic


Producer

Creating Kafka producer

Consumer 1 (with User – swarnava.c)

Consumer 1 (with User – swarnava.c)

Consumer 2 (without User – swarnava.c)

Consumer 2 (without User – swarnava.c)


Congrats. You are done. This is the complete implementation of the SSL implementation in Kafka.

 

kafka authentication Architecture

Opinions expressed by DZone contributors are their own.

Related

  • Building Scalable AI-Driven Microservices With Kubernetes and Kafka
  • Big Data Realtime Data Pipeline Architecture
  • Streaming Data Pipeline Architecture
  • Kafka: Powerhouse Messaging

Partner Resources

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: