Kafka - mata-elang-stable/MataElang-Platform GitHub Wiki

Back to HOME

Contents

kafka

Example Environment

Item Value
Mosquitto IP address 172.16.2.30
Kafka IP address 172.16.2.40
MQTT user / password mataelang / mataelang

Prerequisite

✅ Ubuntu 20.04 LTS installed and updated with the following command.

sudo apt update && sudo apt -y upgrade

✅ Time Zone and NTP already set.

✅ Docker 20.10 or later installed with the following command.

sudo apt -y install docker.io

✅ Docker Compose 2.13 or later installed with the following command.

sudo curl -L "https://github.com/docker/compose/releases/download/v2.13.0/docker-compose-$(uname -s)-$(uname -m)"\
 -o /usr/bin/docker-compose && sudo chmod +x /usr/bin/docker-compose

Setup Kafka

1. Configure Kafka

▶️ Clone Mata-Elang-Stable/kafka-asset from GitHub to your server.

git clone https://github.com/mata-elang-stable/kafka-asset.git ~/kafka

▶️ Configure docker-compose.yaml.

sudo nano ~/kafka/docker-compose.yaml
Configuration

🔑 Change services.mqtt-source.environment.MQTT_HOST to specify the MQTT host. (e.g. 172.16.2.30)

🔑 Change a part of services.kafka.environment.KAFKA_ADVERTISED_LISTENERS to access the Kafka broker from the outside. (e.g. EXTERNAL://172.16.2.40:9093)

🔑 Set MQTT_USERNAME and MQTT_PASSWORD in services.mqtt-source.environment for MQTT authentication. (e.g. mataelang / mataelang)

services:
  mqtt-source:
    environment:
      MQTT_HOST: <MQTT_HOST_IP_OR_NAME>
      MQTT_PORT: 1883
      MQTT_USERNAME: <MQTT_USERNAME>
      MQTT_PASSWORD: <MQTT_PASSWORD>
      MQTT_TOPIC: mataelang/sensor/v3/+
      KAFKA_BOOSTRAP_SERVERS: kafka:9092
      KAFKA_PRODUCE_TOPIC: sensor_events

  kafka:
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://<KAFKA_SERVER_IP_OR_NAME>:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_SEGMENT_BYTES: 536870912   # 512MB
      KAFKA_LOG_RETENTION_BYTES: 536870912 # 512MB
      KAFKA_LOG_RETENTION_MS: 86400000     # 1 day

2. Start Kafka

▶️ Start Kafka service.

❗ The MQTT service must be started before running the sensor service.

sudo docker-compose -f ~/kafka/docker-compose.yaml up -d

✅ Confirm the containers are running.

sudo docker-compose -f ~/kafka/docker-compose.yaml ps -a
Result 

NAME                COMMAND                  SERVICE             STATUS              PORTS
control-center      "/bin/sh -c 'java $J…"   control-center      running             0.0.0.0:9021->8080/tcp, :::9021->8080/tcp
kafka-kafka-1       "/etc/confluent/dock…"   kafka               running             9092/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp
kafka-zookeeper-1   "/etc/confluent/dock…"   zookeeper           running             2181/tcp, 2888/tcp, 3888/tcp
mqtt-source         "/app/me-mqtt-source…"   mqtt-source         running

Admin Web UI

▶️ Open the following URL to see UI for Kafka.

  • URL: http://<KAFKA_SERVER_IP_OR_NAME (e.g. 172.16.2.40)>:9021/
Click to view screen image

kafka

Useful Commands

Click to show commands

Service Commands

✅ Show service status

sudo docker-compose -f ~/kafka/docker-compose.yaml ps -a
Result 

NAME                COMMAND                  SERVICE             STATUS              PORTS
control-center      "/bin/sh -c 'java $J…"   control-center      running             0.0.0.0:9021->8080/tcp, :::9021->8080/tcp
kafka-kafka-1       "/etc/confluent/dock…"   kafka               running             9092/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp
kafka-zookeeper-1   "/etc/confluent/dock…"   zookeeper           running             2181/tcp, 2888/tcp, 3888/tcp
mqtt-source         "/app/me-mqtt-source…"   mqtt-source         running

✅ Start services

sudo docker-compose -f ~/kafka/docker-compose.yaml up -d

✅ Stop services (and remove containers)

sudo docker-compose -f ~/kafka/docker-compose.yaml down

✅ Stop services (and keep containers)

sudo docker-compose -f ~/kafka/docker-compose.yaml stop

✅ Restart services

sudo docker-compose -f ~/kafka/docker-compose.yaml restart

Maintenance Commands

✅ Build kafka-mqtt-source image.

  • Please prepare another host to build the image.
# update packages and install docker
sudo apt update && sudo apt -y upgrade
sudo apt -y install docker.io

# install make and golang
sudo apt -y install make
sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt -y install golang

# download kafka-mqtt-source from Github
git clone https://github.com/mata-elang-stable/kafka-mqtt-source.git ~/kafka-mqtt-source

# build an image
cd ~/kafka-mqtt-source
make build
sudo make build-docker-image

# push the image to your Docker Hub
sudo docker login -u <USERNAME>
Password:
sudo docker push <REPOSITORY>/<IMAGE>[:TAG]

Configuration Commands

✅ Show environment variables

sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' kafka-kafka-1
sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' mqtt-source
sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' kafka-zookeeper-1
sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' control-center

Log Commands

✅ Show Kafka log

sudo docker-compose -f ~/kafka/docker-compose.yaml logs kafka
sudo docker-compose -f ~/kafka/docker-compose.yaml logs mqtt-source
sudo docker-compose -f ~/kafka/docker-compose.yaml logs zookeeper
sudo docker-compose -f ~/kafka/docker-compose.yaml logs control-center

Version Commands

✅ Show Kafka version

sudo docker-compose -f ~/kafka/docker-compose.yaml exec kafka /bin/kafka-configs --version

✅ Show Docker version

sudo docker version

✅ Show Docker Compose version

docker-compose version

✅ Show OS version

cat /etc/os-release

Next Step >>

⚠️ **GitHub.com Fallback** ⚠️