Setup Kafka on cluster

We will setup Kafka on cluster of 3 nodes. Kafka must be implemented in cluster of odd nodes.

Prerequisites

• Java 8 must be installed. JAVA_HOME must be set correctly.
• Zookeeper must be installed. We suppose that default port of zookeeper (2181) was set. if you changed it, use appropriate port in Kafka configurations.
• We assume that we have 3 clusters which theirs name is:master, slave-1, slave-2.

Install Kafka

First download Kafka and extract it in a specific folder. Do this on all your nodes.

$ wget https://www-eu.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz
$ tar -xvf kafka_2.12-2.3.0.tgz
$ mv kafka_2.12-2.3.0 /opt/kafka
$ rm kafka_2.12-2.3.0.tgz

Configure Kafka

First of all, configure Kafka properties. There is a prepared configuration file inside /opt/kafka/config/server.properties. Configure it as below:

broker.id = 0 # it must be unique between servers
listeners = PLAINTEXT://0.0.0.0:9092
advertised.listeners = PLAINTEXT://your-host-name:9092
delete.topic.enable=true # delete a topic, automatically delete it from zookeeper
log.retention.hours=72 # old messages will be deleted after 72 hours
zookeeper.connect = master:2181 , slave-1:2181 , slave-2:2181

Do this configuration on all nodes of cluster.

Enable JMX Of Kafka

For activating status of Kafka, edit /opt/kafka/bin/kafka-run-class.sh file and change this line of code:

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=master -Djava.net.preferIPv4Stack=true"

then export JMX port inside opt/kafka/bin/kafka-server-start.sh before last line:

export JMX_PORT=9999

Start Kafka

To start Kafka, use this command:

/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

If you want to bring it up as service, use this command:

/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

Use jps command to know when it is started. It's process will be seen as Kafka process. If it is failed to start, look at logs to understand what is problem. it's probably because of incorrect configuration.

Restart Kafka

To restart kafk, you must stop it and after stopped successfully, start it again.

/opt/kafka/bin/kafka-server-stop.sh

Use jps command to know when it is stopped.

Check Kafka Cluster

Use command below to see status of Kafka cluster.

/opt/kafka/bin/zookeeper-shell.sh master:2181 ls /brokers/ids

You must see something like this when you run this command:

Connecting to master:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[0, 1, 2]

Note: The last line shows Kafka cluster ids which are connected to each other.

Test Kafka

Create a consumer and producer in different terminals and check you Kafka:

/opt/kafka/bin/kafka-console-consumer.sh --topic links --from-beginning --bootstrap-server master:9092,slave-1:9092,slave-2:9092

/opt/kafka/bin/kafka-console-producer.sh --broker-list master:9092,slave-1:9092,slave-2:9092 --topic links