Kafka - vasanthsumanath/MyWIKI GitHub Wiki
Installing KAFKA in Ubuntu
sudo apt-get install -y wget net-tools netcat tar openjdk-8-jdk
wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
Refer to http://kafka.apache.org/downloads for the latest version of the Kafka
Note for Kafka, java version 8 is the pre requisite.
setting Java to point version 8,
https://serverfault.com/questions/814064/set-jdk-8-as-the-default-java-on-debian-8
Start Zookeeper
~/kafka/bin/zookeeper-server-start.sh -daemon ~/kafka/config/zookeeper.properties
command to check zookeeper is up, health check, should return imok
echo "ruok"|nc localhost 2181; echo
imok
If you are getting the below error
ubuntu@ip-172-31-37-175:~/kafka/bin$ echo "ruok"|nc localhost 2181 ; echo
ruok is not executed because it is not in the whitelist.
ensure set the below attribute in the zookeeper config (zookeeper.properties) file
4lw.commands.whitelist=*
ubuntu@ip-172-31-37-175:~/kafka/bin$ echo "stat"|nc localhost 2181; echo
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/127.0.0.1:51572[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 25
Sent: 24
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: standalone
Node count: 5
ubuntu@ip-172-31-37-175:~/kafka/bin$ echo "conf"|nc localhost 2181; echo
clientPort=2181
secureClientPort=-1
dataDir=/tmp/zookeeper/version-2
dataDirSize=424
dataLogDir=/tmp/zookeeper/version-2
dataLogSize=424
tickTime=3000
maxClientCnxns=0
minSessionTimeout=6000
maxSessionTimeout=60000
serverId=0
ubuntu@ip-172-31-37-175:~/kafka/bin$ echo "ruok"|nc localhost 2181; echo
imok
ubuntu@ip-172-31-37-175:~/kafka/bin$ echo "isro"|nc localhost 2181; echo
rw
ubuntu@ip-172-31-37-175:~/kafka/bin$ echo "mntr"|nc localhost 2181; echo
zk_version 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 32
zk_packets_sent 31
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state standalone
zk_znode_count 5
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 44
zk_open_file_descriptor_count 132
zk_max_file_descriptor_count 1048576
Setting up Zookeeper & Kafka as Service
sudo vi /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apche.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/home/ubuntu/kafka/bin/zookeeper-server-start.sh /home/ubuntu/kafka/config/zookeeper.properties
ExecStop=/home/ubuntu/kafka/bin/zookeeper-server-stop.sh
[Install]
WantedBy=multi-user.target
sudo vi /etc/systemd/system/kafka.service
sudo systemctl enable zookeeper
sudo systemctl enable kafka
sudo systemctl status zookeeper
sudo systemctl status kafka
sudo systemctl daemon-reload
sudo systemctl start kafka
sudo systemctl stop kafka
sudo systemctl stop zookeeper
sudo journalctl -u kafka
sudo journalctl -u zookeeper