Confluent Kafka & Replicator Demo - CloudCommandos/JohnChan GitHub Wiki

Assumptions

• You have an Ubuntu node with at least 10GB RAM and 3 vCPUs
• Your node is assigned with IP 10.0.1.43

Install Docker and Docker-Compose

Install Docker

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
sudo docker run hello-world

Install Docker-Compose

sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

Get Confluent Kafka Demo

Git clone Confluent examples repository and checkout 5.5.1-post branch

git clone https://github.com/confluentinc/examples
cd examples
git checkout 5.5.1-post

You may need to install jq

apt install jq

Start the cluster setup

cd multi-datacenter
./start.sh

Test Confluent Kafka

Install Java dependency on test_producer, test_consumer1, and test_consumer2

apt install openjdk-8-jre-headless

Download Kafka binaries into test_producer, test_consumer1, and test_consumer2

cd ~
wget https://apachemirror.sg.wuchna.com/kafka/2.5.0/kafka_2.13-2.5.0.tgz
tar -zxvf kafka_2.13-2.5.0.tgz

For this demo, topics are already created. Stop the data generator for topic2 first so that we can test with our own data

docker stop datagen-dc1-topic2

Cross-Cluster Event (Topic)

Subscribe to the first broker

~/kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --property print.timestamp=true --bootstrap-server 10.0.1.43:9091 --topic topic2

From another SSH session, subscribe to the second broker

~/kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --property print.timestamp=true --bootstrap-server 10.0.1.43:9092 --topic topic2.replica

From another SSH session, start a producer connection and send test messages. Both the consumers should receive all messages.

~/kafka_2.13-2.5.0/bin/kafka-console-producer.sh --broker-list 10.0.1.43:9091 --topic topic2

Cross-Cluster Command (Queue)

Subscribe to the first broker with group id consumer-grp1

~/kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --property print.timestamp=true --bootstrap-server 10.0.1.43:9091 --topic topic2 --group consumer-grp1

From another SSH session, subscribe to the second broker with group id consumer-grp1

~/kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --property print.timestamp=true --bootstrap-server 10.0.1.43:9092 --topic topic2.replica --group consumer-grp1

From another SSH session, start a producer connection and send test messages.Both consumers receive the message, therefore this command (queue) message fails) (not a queue anymore because it's consumed more than once).

~/kafka_2.13-2.5.0/bin/kafka-console-producer.sh --broker-list 10.0.1.43:9091 --topic topic2