Apache Kafka Series Kafka Security | SSL SASL Kerberos ACL - vidyasekaran/current_learning GitHub Wiki

Apache Kafka Series - Kafka Security | SSL SASL Kerberos ACL

Kafka Security in 0.10 version

Aditional s/w :

Download and Install Ubuntu Bash for Windows: https://docs.microsoft.com/en-us/windows/wsl/install-win10 MobaXterm for all the activities in the terminals if you want to have the setup Gerd has: https://mobaxterm.mobatek.net/https://github.com/simplesteph/kafka-security-manager

Source code : https://courses.datacumulus.com/downloads/kafka-security-a42/

Kafka Security

Encryption (SSL)

All data being sent is fully visible on network (encryption)

**Encryption - ** kafka producer/consumer -----Encrypted Data-----> kafka brokers (9093 SSL)

Authentication (SSL & SASL) **

Currently any client can access your kafka cluster (authentication)

kafka producer/consumer -----Authentication Data-----> kafka brokers (Verify and Authenticate kafka producer/consuemers)

In this course we do SSL Auth and SASL Kerberos Auth

SSL Authentication : clients authenticate to kafka using SSL Certificates

SASL Authentication :

PLAIN : clients auth using username/password (weak)
Kerberos : such as microsoft active directory (strong - hard to setup)
SCRAM : username /password (strong - medium to setup)

Authorisation (ACL)

Clients can publish/consumer any topic data (authorisation)

Once Authenticated, Kafka can verfiy its identity

It still needs to be combined with authorisation, so that kafka knows that

User alice can view topic finance
User bob cannot view topic trucks

ACL (Access Control Lists) have to be maintained by adminstrations and onboard new users

Course Structure

Part 1 : Encryption

Setup Kafka in AWS + SSL Encryption

Part 2 : Authentication

SSL Authentication + SASL Kerberos Authentication

Part 3 : Authorisation

ACLs in Zookeeper

Run EC2 we already had - I have created an assigned elastic ip
download kafka from https://www.apache.org/dyn/closer.cgi?path=/kafka/2.5.1/kafka_2.12-2.5.1.tgz

wget https://mirrors.estointernet.in/apache/kafka/2.5.1/kafka_2.12-2.5.1.tgz

Starting zookeeperr

/home/kafka/kafka_2.12-2.6.0/bin> ./zookeeper-server-start.sh -daemon /home/kafka/kafka_2.12-2.6.0/config/zookeeper.properties Check logs tail -n 5 /home/kafka/kafka_2.12-2.6.0/logs/zookeeper.out

Start kafka server and check logs /home/kafka/kafka_2.12-2.6.0/bin> ./kafka-server-start.sh -daemon /home/kafka/kafka_2.12-2.6.0/config/server.properties /home/kafka/kafka_2.12-2.6.0/logs>tail -n 10 ~/logs/kafkaServer.out

Utility Scripts to start/stop all available

Source : https://courses.datacumulus.com/downloads/kafka-security-a42/ D:\Kafka-Security-Udemy\code_v3\Kafka-Security-master

Refer "Service" file extension - files are setup files For Zookeeper the setup script : D:\Kafka-Security-Udemy\code_v3\Kafka-Security-master\Setup-Zookeeper-Security\zookeeper

To start stop cd D:\Kafka-Security-Udemy\code_v3\Kafka-Security-master\Setup-Zookeeper-Security sudo systemctl enable zookeeper

cd D:\Kafka-Security-Udemy\code_v3\Kafka-Security-master\Setup-Kafka sudo systemctl enable kafka

NOTE: Copy the start/stop utility file to AWS EC2 or anywhere and use it.

Stop Zookeeper

/home/kafka/kafka_2.12-2.6.0/bin [root@ip-172-31-39-218 bin]# ./zookeeper-server-stop.sh

Stop Kafka

/home/kafka/kafka_2.12-2.6.0/bin [root@ip-172-31-39-218 bin]# ./kafka-server-stop.sh

stop kafka server and zookeeper-server.stop.sh

/home/kafka/kafka_2.12-2.6.0/bin/kafka-server-stop /home/kafka/kafka_2.12-2.6.0/bin/zookeeper-server.stop.sh

To automatically start /stop service in linix

you create a file /etc/systemd/system/zookeeper.service and place below content in there... same for kafka also

[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target

[Service] Type=simple ExecStart=/home/kafka/kafka_2.12-2.6.0/bin/zookeeper-server-start.sh /home/kafka/kafka_2.12-2.6.0/zookeeper.properties ExecStop=/home/kafka/kafka_2.12-2.6.0/bin/zookeeper-server-stop.sh

[Install] WantedBy=multi-user.target

once above process is done u can start stop kafka/zookeeper sudo systemctl enable zookeeper sudo systemctl enable kafka

Producer / Consumer Test

Add below ports for zookeeper to EC2 instance

Custom TCP - ports : 2181 (zookeeper) - with myip (source) - description zookeeper Custom TCP - ports : 9092 (kafka) - with myip (source)- description kafka plain text port

Open /home/kafka/kafka_2.12-2.6.0/server.properties

Modify the AWS public DNS for below configs in Comment already existing below config and enable value for both

advertised.listeners=PLAINTEXT://##your-public-DNS##:9092 zookeeper.connect=##your-public-DNS##:2181

Start zookeeper and kafka

[root@ip-172-31-39-218 bin]# /home/kafka/kafka_2.12-2.6.0/bin/zookeeper-server-start.sh /home/kafka/kafka_2.12-2.6.0/config/zookeeper.properties [root@ip-172-31-39-218 bin]# /home/kafka/kafka_2.12-2.6.0/bin/kafka-server-start.sh /home/kafka/kafka_2.12-2.6.0/config/server.properties

Create Kafka Topic

~/kafka/bin/kafka-topics.sh --zookeeper ec2-18-196-169-2.eu-central-1.compute.amazonaws.com:2181 --create --topic kafka-security-topic --replication-factor 1 --partitions 2

Create Kafka Consumer

[root@ip-172-31-39-218 bin]# ./kafka-console-consumer.sh --bootstrap-server ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9092 --topic kafka-security-topic

Create Kafka Producer

[root@ip-172-31-39-218 bin]# ./kafka-console-producer.sh --broker-list ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9092 --topic kafka-security-topic

Detailed Setup of SSL Diagram present in link below

Kafka SSL Archi

SSL Encryption in java

We will setup a certificate authority setup a broker certificate sign a broker certificate setup a key store for kafka broker setup a trust store for kafka client reboot kafka broker with SSL mode (port 9093) test our setup using secure SSL producer and consumer

We are going to create a CA for issuing SSL Certificates we will create a certificate for our kafka broker we will verfiy the content of the SSL Certificates

Steps to create certificate authority,

Creating a private Certificate Authority

**we are requesting new keys with rsa encryption with key length 4096 with 365 days valid as subject with command name

we receive 2 files "ca-key" (private key of ca) ca-cert (public key) used to import on trust store later on openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Kafka-Security-CA" -keyout ca-key -out ca-cert -nodes

configure keystore and truststore for kafka broker

using certificate we have generated, we will setup kafka broker to use SSL on port 9093

once kafka is rebooted we will test the SSL certificates once again to make sure they are correctly applied.

Create kafka broker certificate

export SRVPASS=serversecret

[root@ip-172-31-39-218 ssl]# keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass $SRVPASS -keypass $SRVPASS -dname "CN=ec2-15-206-248-73.ap-south-1.compute.amazonaws.com" -storetype pkcs12

List the certificate preset in keystore keytool -list -v -keystore kafka.server.keystore.jks password = serversecret

Sign a Certificate is a 2 step process 1st we need send a signing request to CA cerficate authority and then send the file (in our case -cert-file)

Request CA to get a certificate signed - we get cert-file created keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass $SRVPASS -keypass $SRVPASS this command creates -> cert-file

-rw-r--r-- 1 root root 1513 Nov 21 10:09 cert-file

Send cert-file generated from previous command to CA and we get "cert-signed" file as output. openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:$SRVPASS

-rw-r--r-- 1 root root 2126 Nov 21 10:12 cert-signed

This is the signed file for our kafka broker

keytool -printcert -v -file cert-signed (shows details of our signed key, owner and who signed it)

[root@ip-172-31-39-218 ssl]# keytool -printcert -v -file cert-signed Owner: CN=ec2-15-206-248-73.ap-south-1.compute.amazonaws.com Issuer: CN=Kafka-Security-CA Serial number: a7ec8b8765ecc71b Valid from: Sat Nov 21 10:12:41 UTC 2020 until: Sun Nov 21 10:12:41 UTC 2021 Certificate fingerprints: MD5: 0C:BA:DA:DF:0D:20:6E:D3:CF:05:AD:12:7A:4C:2C:30 SHA1: B8:63:ED:4F:01:32:18:1A:22:BB:75:EE:C3:CE:1E:D7:95:4A:AF:C7 SHA256: A8:97:02:B8:E3:44:07:71:C4:C1:3C:AA:B0:B7:DE:3E:01:19:9E:6E:1D:BB:D0:3A:2B:BB:EF:40:D4:19:B7:1E Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit DSA key Version: 1

Now Create Trust Store on our kafka broker

import ca-cert into key store keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $SRVPASS -keypass $SRVPASS -noprompt Certificate was added to keystore

below file is created

-rw-r--r-- 1 root root 1358 Nov 21 10:22 kafka.server.truststore.jks

import cert-signed into key store

[root@ip-172-31-39-218 ssl]# keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass $SRVPASS -keypass $SRVPASS -noprompt Certificate reply was installed in keystore

Open /home/kafka/kafka_2.12-2.6.0/server.properties

Copy the SSL file for kafka in here and paste it in /home/kafka/kafka_2.12-2.6.0/server.properties Source code : https://courses.datacumulus.com/downloads/kafka-security-a42/

Change it to reflect ssl details

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093 advertised.listeners=PLAINTEXT://ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9092,SSL://ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9093 zookeeper.connect=ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:2181

ssl.keystore.location=/home/ssl/kafka.server.keystore.jks ssl.keystore.password=serversecret ssl.key.password=serversecret ssl.truststore.location=/home/ssl/kafka.server.truststore.jks ssl.truststore.password=serversecret

Start Zookeeper and kafka broker

[root@ip-172-31-39-218 config]# /home/kafka/kafka_2.12-2.6.0/bin/zookeeper-server-start.sh /home/kafka/kafka_2.12-2.6.0/config/zookeeper.properties [root@ip-172-31-39-218 bin]# /home/kafka/kafka_2.12-2.6.0/bin/kafka-server-start.sh /home/kafka/kafka_2.12-2.6.0/config/server.properties

Addtional Check :

sudo grep -i "EndPoint" /home/kafka/kafka_2.12-2.6.0/logs/server.log

[2020-11-21 10:42:53,263] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9092,SSL://ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9093, czxid (broker epoch): 343 (kafka.zk.KafkaZkClient)

so our kafka broker is listening to our SSL endpoint so we need to add in EC2 security group port 90923

Connect to Kafka Brokerr endpoint thru SSL - Yes

[root@ip-172-31-39-218 ssl]# openssl s_client -connect ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9093 CONNECTED(00000003) depth=1 CN = Kafka-Security-CA verify error:num=19:self signed certificate in certificate chain

Certificate chain 0 s:/CN=ec2-15-206-248-73.ap-south-1.compute.amazonaws.com i:/CN=Kafka-Security-CA 1 s:/CN=Kafka-Security-CA i:/CN=Kafka-Security-CA

Check whether we are able to reach our kafka broker thru SSL port

openssl s_client -connect ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9093

ca-cert and ca-key are from Certificate Authority in real world these wont be here

cert-file is kafka broker cert can be deleted ca-cert.srl related to signing of certificate

cert-signed - we need to be imported this to key store kafka.server.truststore.jks need as various brokerrof kafka

from sensitivity point - ca-key and kafka.server.keystore.jks we never have to distribute to anybody.

ca-cert and cert-singed - can be distributed to client so they can import it to their keystore to communicate with kafka broker thru SSL.

Configuration of Kafka Client

export CLIPASS=clientpass mkdir /home/client_ssl

Create Trust Store for our clients this is need to verfiy ssl cerficate that client gets from kafka broker

2 options

a. chain of trust - import ca-public certificate to trust store that means we trusting any server certificate signed by this CA b. importing server public certificate - we trust just this single server instance

copy CA public certicate to /home/client_ssl

[root@ip-172-31-39-218 client_ssl]# cp /home/ssl/ca-cert . [root@ip-172-31-39-218 client_ssl]# ls -ltr total 4 -rw-r--r-- 1 root root 1809 Nov 21 11:08 ca-cert

Create the KeyStore for Client

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $CLIPASS -keypass $CLIPASS -noprompt

print the details of kafka.client.truststore.jks keytool -list -v -keystore kafka.client.truststore.jks

Property file for console consumer and client

vi client.properties (you can find it in resource of this couse - D:\Kafka-Security-Udemy\code_v3\Kafka-Security-master\Setup-SSL\client) D:\Kafka-Security-Udemy\code_v3\Kafka-Security-master\Setup-SSL Source code : https://courses.datacumulus.com/downloads/kafka-security-a42/

Start our client

Create Kafka Consumer

cd /home/kafka/kafka_2.12-2.6.0/bin ./kafka-console-consumer.sh --bootstrap-server ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9093 --topic kafka-security-topic

Create Kafka Producer

cd /home/kafka/kafka_2.12-2.6.0/bin cd /home/kafka/kafka_2.12-2.6.0/bin> ./kafka-console-producer.sh --broker-list ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9093 --topic kafka-security-topic /home/client_ssl/client.properties

[2020-11-21 11:29:16,423] INFO [SocketServer brokerId=0] Failed authentication with /172.31.39.218 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

SSL Setup for our Kafka Client (Always refer the big picture which has CA, keystore,trust store, kafka server etc)

Setup SSL Chaneel between server and client thru SSL

Ref : https://sites.google.com/site/ownscratchpad/kafka/ssl

export CLIPASS=clientpass (non interactive to create trust store)

create truststore for our client - so that client can accept and verfiy the ssl certificate from the kafka end point.

copy ca public certificate to our client i.e "ca-cert" file.

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert pass $CLIPASS -keypass $CLIPASS -noprompt

check content of keystore

keytool -list -v -keystore kafka.client.truststore.jks

create client.properties with below contents

security.protocol=SSL ssl.truststore.location=/home/ssl/kafka.client.truststore.jks ssl.truststore.password=clientpass

start producer

[root@ip-172-31-39-218 bin]# ./kafka-console-producer.sh --broker-list ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9092 --topic kafka-security-topic --producer.config ~/client-ssl/client.properties

start consumer

[root@ip-172-31-39-218 bin]# ./kafka-console-consumer.sh --bootstrap-server ec2-15-206-248-73.ap-south-1.compute.amazonaws.com:9092 --topic kafka-security-topic --consumer.config ~/client-ssl/client.properties

Producer and Consumer would fail if we dont provide --consumer.config ~/client-ssl/client.properties