Kafka cluster - stanislawbartkowski/wikis GitHub Wiki
Introduction
Short description on how to set up minimalistic Kafka cluster
Zookeeper
Zookeeper hosts
At least three nodes to install HA Zookeeper cluster.
cat /etc/hosts
192.168.0.13 zoo1.sb.com zoo1
192.168.0.76 zoo2.sb.com zoo2
192.168.0.111 zoo3.sb.com zoo3
Download and install
https://zookeeper.apache.org/releases.html#download
Unpack in /opt directory
ls /opt/apache-zookeeper-3.7.0-bin/ -ltr
* bin
* README_packaging.md
* README.md
* NOTICE.txt
* LICENSE.txt
* docs
* lib
* conf
* logs
Starter
Create shortcut in /usr/local/bin
cat zkServer
cd /opt/apache-zookeeper-3.7.0-bin/bin
exec ./zkServer.sh $@
Configure cluster
On every host in the cluster create the corresponding myid tag file as "1", "2" and "3". Directory for myid file is defined by dataDir configuration parameter in zoo.cfg file.
mkdir /var/lib/zookeeper echo "1" > /var/lib/zookeeper/myid
cd /opt/apache-zookeeper-3.7.0-bin/conf cp cp zoo_sample.cfg zoo.cfg vi zoo.cfg
dataDir=/var/lib/zookeeper
server.1=zoo1.sb.com:2888:3888
server.2=zoo2.sb.com:2888:3888
server.3=zoo3.sb.com:2888:3888
Start
On all three nodes run.
zkServer start
After several seconds verify.
zkServer status
On server should report leader mode and two other follower modes.
zkServer status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
Kafka cluster
Kafka hosts
At least three hosts.
cat /etc/hosts
192.168.122.73 kafka1.sb.com kafka1
192.168.122.246 kafka2.sb.com kafka2
192.168.122.176 kafka3.sb.com kafka3
Download and install
https://kafka.apache.org/downloads
Untar in /opt directory
ls /opt/kafka_2.13-3.0.0/
* NOTICE
* LICENSE
* licenses
* configvi server.properties
* bin
* site-docs
* libs
Create starter
cd /usr/local/bin cat kafkaStart
cd /opt/kafka_2.13-3.0.0/
exec bin/kafka-server-start.sh config/server.properties
Configure
cd /opt/kafka_2.13-3.0.0/config
Parameter zookeeper.connect specifies Zookeeper cluster. Parameter broker.id should be different for every host in the cluster. For instance: 0,1 and 2.
vi server.properties
zookeeper.connect=zoo1.sb.com:2181,zoo2.sb.com:2181,zoo3.sb.com:2181
.........
broker.id=0
Start
kafkaStart
Basic test
Create topic
kafka-topics --create --partitions 1 --replication-factor 1 --topic quickstart-events --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092
Created topic quickstart-events.
List topics.
kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --list
quickstart-events
Publish messages to topic
kafka-console-producer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic quickstart-events
>first message
>second message
>third message
Publish messages as key/value
kafka-console-producer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test -property parse.key=true --property key.separator=,
key1,line1
key2,line2
key3,line3
Read messages for topic
kafka-console-consumer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic quickstart-events --from-beginning
first message
second message
third message
Read messages including key
kafka-console-consumer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test --from-beginning --property print.key=true
Useful commands
Detailed information about the topic.
kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --topic quickstart-events
Topic: quickstart-events TopicId: Ru_ak3wiSseoWMCHs7RVug PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: quickstart-events Partition: 0 Leader: 2 Replicas: 2 Isr: 2
(There is a single replica of the topic on the Kafka broker 2)
Delete the topic
kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --delete --topic quickstart-events
Create a topic with a replication factor
kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --create --partitions 1 --topic quickstart-events --replication-factor 3
kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --topic quickstart-events
Topic: quickstart-events TopicId: _UMqCSphShWF0NZiQEwndA PartitionCount: 1 ReplicationFactor: 3 Configs: segment.bytes=1073741824
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
(three replicas across three Kafka brokers)
Consumer groups
kafka-consumer-groups.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --all-groups
Offsets
kafka-get-offsets.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test