Centralized logging using ELK - vidyasekaran/current_learning GitHub Wiki

Refer Part I and Part 2

In part 2 i was able to download and run zookeeper and kafka create a topic "test" and run a consumer. Refer Part 2 down for more details.

Currently Part 2- Microservices can publish logs to kafka topic but need to configure ELK to pick log from kafka is pending. I need to combine Part I with Part II so that Logstash points to kafka and could see logs in ELK

Refer this link for Logstash - Kafka integration - https://dzone.com/articles/deploying-kafka-with-the-elk-stack

Part I

https://howtodoinjava.com/microservices/elk-stack-tutorial-example/ https://www.youtube.com/watch?v=9g-h1biMn2E

downloaded elk in D:\softwares downloaded code and setup in STS D:\workspace\elk-example-spring-boot

  1. Download and start kibana D:\softwares\kibana-7.4.2-windows-x86_64\kibana-7.4.2-windows-x86_64\bin>kibana.bat

Access it http://localhost:5601/

Refer : Microservice | Centralize Logging using ELK - Stack| PART-6 | Javatechie (9:02) to Create Index in Kibana and point in logstash config

In Kibana - Click on Management - Dev Tools - in Console put below json to create index and put dummy document

PUT /javatechie { "settings":{ "index":{ "number_of_shards" :3, "number_of_replicas":2 } } }

Put dummy document to above index

POST /javatechie/default/ { "name":"event processing", "instructor":{ "firstName":"java", "lastName":"techie" } }

2.Make below config changes to incorporate index created above "/javatechie" with time "javatechie-%{+yyyy.MM.dd}" in logstash config D:\softwares\logstash-7.4.2\logstash-7.4.2\config\logstash.conf. If you have multiple microservices all microservices should write to same log file- you need to write log to same file if you have different microservices

input { file { type => "java" path => "E:/build_complete_microservices/elk-example-spring-boot/elk-example.log" codec => multiline { pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*" negate => "true" what => "previous" } } }

Sending properly parsed log events to elasticsearch

elasticsearch { hosts => ["localhost:9200"] index => "javatechie-%{+yyyy.MM.dd}" } }

Restart Logstash D:\softwares\logstash-7.4.2\logstash-7.4.2\bin>logstash -f logstash.conf

Start ElasticSearch D:\softwares\elasticsearch-7.4.2-windows-x86_64\elasticsearch-7.4.2\bin>elasticsearch.bat

==============================================================

Part 2

I downloaded code from https://examples.javacodegeeks.com/enterprise-java/logback/logback-kafka-appender-example/ Code uses logback points to "test" topic and on running - the kafka consumer receives log messages.

This link has details on starting kafka, zookeeper, kafka consumer https://dzone.com/articles/running-apache-kafka-on-windows-os

Kafka downloaded and setup in : In kafka config i had to point to zookeeper ip : 0.0.0.0
Zookeeper downloaded and setup :

Starting kafka server - D:\softwares\kafka_2.12-2.6.0>.\bin\windows\kafka-server-start.bat .\config\server.properties

Start kafka consumer : D:\softwares\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test

Starging zookeper - D:\softwares\apache-zookeeper-3.5.8-bin.tar\apache-zookeeper-3.5.8-bin\bin>zkserver

Microservices can publish logs to kafka topic but need to configure ELK to pick log from kafka Completed. I was able to combine Part I with Part II so that Logstash pick from kafka and write to "javatechie" index and was able to see log in kibana.

  1. First I created an index "javatechie" in ElasticSearch using Kibana - Refer steps above.

  2. I configured logstash.conf to pick logs from kafka and write to elasticsearch index "javatechie" -

D:\softwares\kafka_2.12-2.6.0\config\logstash.conf

Config to pick log from kafka server port and topic

input { kafka { bootstrap_servers => 'localhost:9092' topics => ["test"] codec => json {} } }

Config to write the log to elastic search "javatechie" index elastic search host port mentioned

output {

stdout { codec => rubydebug }

Sending properly parsed log events to elasticsearch

elasticsearch { hosts => ["localhost:9200"] index => "javatechie-%{+yyyy.MM.dd}" } }

  1. Now Start

Starting kafka server - D:\softwares\kafka_2.12-2.6.0>.\bin\windows\kafka-server-start.bat .\config\server.properties

Start kafka consumer : D:\softwares\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test

Staring zookeper - D:\softwares\apache-zookeeper-3.5.8-bin.tar\apache-zookeeper-3.5.8-bin\bin>zkserver

Staring Logstash -D:\softwares\logstash-7.4.2\logstash-7.4.2\bin>logstash -f logstash.conf

Start ElasticSearch -D:\softwares\elasticsearch-7.4.2-windows-x86_64\elasticsearch-7.4.2\bin>elasticsearch.bat

Start Kibana - D:\softwares\kibana-7.4.2-windows-x86_64\kibana-7.4.2-windows-x86_64\bin>kibana.bat Access it http://localhost:5601/

  1. run the program in local -> E:\build_complete_microservices\logbackkafka\logbackkafka\src\main\java\com\jcg\logbackKafka\LoggerRoot.java

that I downloaded code from https://examples.javacodegeeks.com/enterprise-java/logback/logback-kafka-appender-example/ Code uses logback points to "test" topic and on running - the kafka consumer receives log messages.

Refer this link for Logstash - Kafka integration - https://dzone.com/articles/deploying-kafka-with-the-elk-stack

Logstash config to Kafka below url helped fix : https://facingissuesonit.com/2017/05/06/integrate-logstash-with-kafka/

  1. Check whether kafka consumer is able to pick the message and

Start kafka consumer : D:\softwares\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test

  1. Check logs in kibana - Access it http://localhost:5601/

  2. Modify log message to log shanthi, jaiguru and search in kibana to see the messages

message:10:15:53.488 [main] INFO LoggerRoot - Shanthi:4 @version:1 @timestamp:2020-11-21T04:45:53.628Z tags:_jsonparsefailure, _grokparsefailure _id:L3of6XUBj1yRJjdOczng _type:_doc _index:javatechie-2020.11.21 _score:0 message:10:15:53.489 [main] INFO LoggerRoot - Shanthi:8 @version:1 @timestamp:2020-11-21T04:45:53.633Z tags:_jsonparsefailure, _grokparsefailure _id:MHof6XUBj1yRJjdOczng _type:_doc _index:javatechie-2020.11.21 _score:0