1.21 Kafka with Azure - amresh087/Question GitHub Wiki

• Go to the Azure Portal
• In the search bar, look for Virtual Machines and click on that
• You got landed on to Virtual Machines Dashboard
• Click on Create -> Virtual Machine
• Under Resource Group, click on Create new
• Define a name and Click on OK
• Define a name for your Virtual Machine
• Choose a location, let it be East US
• Under Image section we need to select OS, choose Ubuntu 18.04 LTS under that.˳
• If Ubuntu 18.04 LTS is not in the dropdown list, click on See All Images below the dropdown
• Search for Ubuntu 18.04 LTS Genand select the version with Canonical as a publisher
• Now, we need to define the size for the Virtual Machine, click on See All Sizes
• Select B2s from the list and click on Select button on bottom left corner
• Choose the authentication type as Password
• Define your username and password for the Virtual Machine
• Click on Review + Create
• Click on Create Button

• Open Terminal/CMD/Git Bash and run the below stated command

  ssh your_username@public_ip_of_vm

• If the above is not working, then follow this method to connect using putty.

  • Download Putty from this link https://the.earth.li/~sgtatham/putty/latest/w64/putty.exe
  • Open putty
  • In the hostname text box, put your_username@public_ip_of_vm
  • Click on Open followed by a click on Accept
  • Now provide the password, and there you go

• Switch to root user

  sudo su

• Update Repo List

  apt-get update

• Install JDK

  apt-get install default-jdk

• Verify Java Installation

  java --version

• Download Kafka Package

  wget https://mirrors.estointernet.in/apache/kafka/2.8.0/kafka-2.8.0-src.tgz

• Extract it

  tar -xzf kafka-2.8.0-src.tgz

• Change Directory

  cd kafka-2.8.0-src

• Compile it with Gradle

  ./gradlew jar

• As we compiled KAFKA using gradle in this session only, so we are in the KAFKA directory only

• Let's start zookeeper

  bin/zookeeper-server-start.sh config/zookeeper.properties

• Switch to root user

  sudo su

• Go to KAFKA directory

  cd kafka-2.8.0-src

• Let's start kafka broker

  bin/kafka-server-start.sh config/server.properties

• Switch to root user

  sudo su

• Go to KAFKA directory

  cd kafka-2.8.0-src

• Let's get the list of TOPICs in the kadka

  bin/kafka-topics.sh --list --bootstrap-server localhost:9092

• Creating a topic bin/kafka-topics.sh --create --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

• List the topics bin/kafka-topics.sh --list --bootstrap-server localhost:9092

• Desribe the topic created by you bin/kafka-topics.sh --describe --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

• To write some events bin/kafka-console-producer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

  After execution of this command, it will prompt you for messages, just send them and at the end once you are done, just Press CTRL + C to halt the process

• To read the events from beginning bin/kafka-console-consumer.sh --topic demo1 --bootstrap-server localhost:9092 --from-beginning

  Press CTRL + C to halt this

• To delete the topic bin/kafka-topics.sh --delete --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

• To create a topic, if it is not there bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic YOUR_TOPIC_NAME --if-not-exists

• To create a topic and specifying number of partitions bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic YOUR_TOPIC_NAME --if-not-exists --partitions NO_OF_PARTITIONS

• Produce messages bin/kafka-console-producer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

  After execution of this command, it will prompt you for messages, just send them and at the end once you are done, just Press CTRL + C to halt the process

• Consume Messages from the topic with multiple partitions bin/kafka-console-consumer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092 --from-beginning

  Press CTRL + C to halt this

• Consume message from a topic with a specific partition bin/kafka-console-consumer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092 --from-beginning --partition PARTITION_NO

• Altering the topic bin/kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic YOUR_TOPIC_NAME --partitions NO_OF_PARTITIONS

• Describe topic to verify change bin/kafka-topics.sh --describe --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

• Produce messages bin/kafka-console-producer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092

  After execution of this command, it will prompt you for messages, just send them and at the end once you are done, just Press CTRL + C to halt the process

• Consume Messages from the altered topic bin/kafka-console-consumer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092 --from-beginning

  Press CTRL + C to halt this

• Consume message from a topic with a specific partition bin/kafka-console-consumer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092 --from-beginning --partition PARTITION_NO Produce messages using key bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic YOUR_TOPIC_NAME --property parse.key=true --property key.separator=":"

  Messages should be in this order (Key:Msg) After execution of this command, it will prompt you for messages, just send them and at the end once you are done, just Press CTRL + C to halt the process

• Consume the messages bin/kafka-console-consumer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092 --from-beginning --partition PARTITION_NO

• Consume messages with key printed as well bin/kafka-console-consumer.sh --topic YOUR_TOPIC_NAME --bootstrap-server localhost:9092 --from-beginning --property print.key=true --partition PARTITION_NO

• Go to your VM page
• In the left pane, Click on Networking
• Click on Add Inbound Rule
• Under destination port put 8080 and click on Add Button

Install maven

apt-get install maven

Instructions

Switch to home directory cd /home/<user_name>

Get the code cloned git clone https://gitlab.com/synechron-elevate4/kafka-producer-lab

Change directory to code cd kafka-producer-lab

If you wish to change the topic name, then you need to edit the controller file (OPTIONAL) vi src/main/java/com/thinknyx/kafkademo2/controller/KafkaController.java

Run the app using ./mvnw spring-boot:run

Now access the application in browser

To produce message url will be

your_vm_ip:8080/kafka/produce?message=SomeMessage

Verify the sent messages via consumer using cli, if topic name is different change it bin/kafka-console-consumer.sh --topic demo --bootstrap-server localhost:9092 --from-beginning