[Archive 1.0] 6. Kafka - mata-elang-stable/MataElang-Platform GitHub Wiki

Back to HOME

• Prerequisite
• Install Confluent
• Configure Confluent
• Start Kafka

Prerequisite

• Ubuntu 18.04.6 LTS installed and updated by sudo apt update && sudo apt -y upgrade command.
• Time Zone and NTP already set.
• Docker 20.10.11 or later installed and ready.
• Docker Compose 1.17.1 or later installed by sudo apt -y install docker-compose command.

Install Confluent

1. Make clone of Mata-Elang-Stable/MataElang-confluent from GitHub.

cd ~

git clone https://github.com/Mata-Elang-Stable/MataElang-Confluent.git

2. Download "kafka-connect-mqtt" of lensesio/tream-reactor from GitHub.

cd ~/MataElang-Confluent

wget https://github.com/lensesio/stream-reactor/releases/download/2.1.3/kafka-connect-mqtt-2.1.3-2.5.0-all.tar.gz

tar -xvf kafka-connect-mqtt-2.1.3-2.5.0-all.tar.gz

Configure Confluent

3. Edit "docker-compose.yml".

nano docker-compose.yml

• Configuration

services:
  broker:
    environment:
      ...
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://<Kafka broker (e.g. 172.16.2.40)>:9092
      ...

4. Edit "connector.json".

nano connector.json

• Configuration

{
  "name" : "mqtt-source",
  "config" : {
    ...
    "connect.mqtt.hosts" : "tcp://<MQTT broker (e.g. 172.16.2.30)>:1883",
    ...
  }
}

Start Kafka

5. Start Kafka services and confirm the services are up.

cd ~/MataElang-Confluent

sudo docker-compose up -d

sudo docker-compose ps

• Result

     Name                    Command               State                        Ports                    
----------------------------------------------------------------------------------------------------------
broker            /etc/confluent/docker/run        Up       0.0.0.0:9092->9092/tcp,:::9092->9092/tcp, ...
connect           /etc/confluent/docker/run        Up       0.0.0.0:8083->8083/tcp,:::8083->8083/tcp, ...
connect-add       /entrypoint.sh curl -s -X  ...   Exit 7
control-center    /etc/confluent/docker/run        Up       0.0.0.0:9021->9021/tcp,:::9021->9021/tcp
ksqldb-cli        /bin/sh                          Up
ksqldb-server     /etc/confluent/docker/run        Up       0.0.0.0:8088->8088/tcp,:::8088->8088/tcp
rest-proxy        /etc/confluent/docker/run        Up       0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
schema-registry   /etc/confluent/docker/run        Up       0.0.0.0:8081->8081/tcp,:::8081->8081/tcp
zookeeper         /etc/confluent/docker/run        Up       0.0.0.0:2181->2181/tcp,:::2181->2181/tcp, ...

6. If the connect-add service exits with code 7, run the following command until the exit code is 0.

• You may need to repeat it several times until all services is up or exit with code 0.

sudo docker-compose start connect-add

sudo docker-compose ps

• Result

     Name                    Command               State                        Ports                    
----------------------------------------------------------------------------------------------------------
  ...
connect-add       /entrypoint.sh curl -s -X  ...   Exit 0
  ...

7. Check the connector settings with the following command whether it was deployed or not.

curl http://localhost:8083/connectors

• Result

["mqtt-source"]

8. Open the following URL in your browser to see the dashboard.

• URL: http://<Kafla Server (e.g. 172.16.2.40)>:9021/

9. You can check received messages by selecting "Topics" -> "snoqttv5"(MQTT Topic) -> "Messages" in the dashboard

Next Step >>