System Monitoring: ELK Installation and Configuration - AsrafulSamad/PoraLikha GitHub Wiki

Filebeat

Steps

Download and install filebeat-6.4.2-x86_64.rpm (In each VM where runs spring boot applications. Core(mfs-app1-6), Web(mfs-web1-3), Clearing(mfs-clr1-4), DMZ(mfs-ussd1-3) )
Change the configuration file(filebeat.yml) accordingly

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
 
  # Change to true to enable this input configuration.
  enabled: true
paths:
    - /var/log/mfs/*/logs/applogs/*mfs-app1-INST_1.log
    - /var/log/mfs/*/logs/applogs/*mfs-app1-INST_2.log
    - /var/log/mfs/*/logs/applogs/*mfs-app1-INST_3.log
    - /var/log/mfs/*/logs/applogs/*mfs-app1-INST_4.log
    - /var/log/mfs/*/logs/applogs/*mfs-app1-INST_5.log
#For USSD:
    - /var/log/mfs/*/logs/applogs/*mfs-ussd1-teletalk.log
    - /var/log/mfs/*/logs/applogs/*mfs-ussd1-robi.log
    - /var/log/mfs/*/logs/applogs/*mfs-ussd1-gp.log
    - /var/log/mfs/*/logs/applogs/*mfs-ussd1-banglalink.log
    - /var/log/mfs/*/logs/applogs/*mfs-ussd1-airtel.log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}\s-\s(((\[\w+-[0-9]{1,5}\])|(\[\w+-[0-9]{1,5}\s\w+-[0-9]{1,5}\])|(\[\w+-[0-9]{1,5}\s\w+\])))'
multiline.negate: true
multiline.match: after
  
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
 
 
#----------------------------- Logstash output --------------------------------
output.logstash:
   hosts: ["mfs-ops1:5044"]

Logstash

Steps

Download and install logstash-6.4.2.rpm in operation (mfs-ops1) server
Open port 5044 to get access from Core, Web, Clering and DMZ.
Check '/etc/logstash/pipelines.yml' file.

- pipeline.id: main
   path.config: "/etc/logstash/conf.d/*.conf"

Configure '/etc/logstash/conf.d/logstash-simple.conf'

input {
  beats {
    port => 5044
  }
}
filter {
     if [source] != "" {
        grok{
           match => { "source" => "\/var\/log\/mfs\/%{WORD:kpp.component}\/logs\/applogs\/%{WORD}-%{WORD}-%{WORD}-%{WORD:kpp.instance}\.log"}
       }
     }
 
    mutate {
        add_field => { "kpp.beat.host.name" => "" }
        add_field => { "kpp.beat.name" => "" }
        add_field => { "kpp.beat.version" => "" }
        }
 
     mutate {
            copy => { "[beat][hostname]" => "[kpp][beat][host][name]" }
            copy => { "[beat][name]" => "[kpp][beat][name]" }
            copy => { "[beat][version]" => "[kpp][beat][version]" }
        }
 
    grok {
        match => {
            "message" => "^%{TIMESTAMP_ISO8601:timestamp}%{TYPE_PATTERN}"
        }
    }
 
    if "_grokparsefailure" in [tags] {
        grok {
            match => {
                "message" => "^%{TIMESTAMP_ISO8601:timestamp}%{SHORT_PATTERN}"
            }
        }
    }else if([kpp.api.type] == "RES"){
        grok {
            match => {
                "kpp.logmessage" => "%{RES_PATTERN}"
            }
        }
    }else if([kpp.api.type] == "REQ"){
        grok {
            match => {
                "kpp.logmessage" => "%{REQ_PATTERN}"
            }
        }
             
    }

 
 date {
     match => ["timestamp", "ISO8601"]
     target => "@timestamp"
  }
}
 
output {
  elasticsearch {
    hosts => ["http://mfs-ops1:9200"]
    index => "mfs-app-logs"
  }
}

Create new pattern file '/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/mfspatterns'

MFSHEADER %{SPACE}-%{SPACE}\[%{GREEDYDATA:kpp.thread}\]%{SPACE}-%{SPACE}\[%{WORD:kpp.loglevel}%{SPACE}\]%{SPACE}-%{SPACE}%{JAVACLASS:kpp.logger}%{SPACE}-%{SPACE}CID\[%{GREEDYDATA:kpp.cid}\]%{SPACE}-%{SPACE}UID\[%{GREEDYDATA:kpp.uid}\]%{SPACE}-%{SPACE}\[%{WORD:kpp.api.type}\]%{GREEDYDATA:kpp.logmessage}
 
TYPE_PATTERN %{SPACE}-%{SPACE}\[%{GREEDYDATA:kpp.thread}\]%{SPACE}-%{SPACE}\[%{WORD:kpp.loglevel}%{SPACE}\]%{SPACE}-%{SPACE}%{JAVACLASS:kpp.logger}%{SPACE}-%{SPACE}CID\[%{GREEDYDATA:kpp.cid}\]%{SPACE}-%{SPACE}UID\[%{GREEDYDATA:kpp.uid}\]%{SPACE}-%{SPACE}\[%{WORD:kpp.api.type}\]%{GREEDYDATA:kpp.logmessage}
 
SHORT_PATTERN %{SPACE}-%{SPACE}\[%{GREEDYDATA:kpp.thread}\]%{SPACE}-%{SPACE}\[%{WORD:kpp.loglevel}%{SPACE}\]%{SPACE}-%{SPACE}%{JAVACLASS:kpp.logger}%{SPACE}-%{SPACE}CID\[%{GREEDYDATA:kpp.cid}\]%{SPACE}-%{SPACE}UID\[%{GREEDYDATA:kpp.uid}\]%{SPACE}-%{SPACE}%{GREEDYDATA:kpp.logmessage}
 
RES_PATTERN %{SPACE}%{WORD:kpp.api.method}%{SPACE}%{PATH:kpp.api.path}(\?%{GREEDYDATA:kpp.api.requestparams}|%{GREEDYDATA:kpp.api.requestparams})%{SPACE}%{NUMBER:kpp.api.response.status}%{SPACE}%{NUMBER:kpp.api.response.time:int}ms%{GREEDYDATA:kpp.logmessage}
 
REQ_PATTERN %{SPACE}%{WORD:kpp.api.method}%{SPACE}%{PATH:kpp.api.path}%{SPACE}%{GREEDYDATA:kpp.logmessage}

Elasticsearch

Steps

Download and install elasticsearch-6.4.2.rpm in operation (mfs-ops1) server
Configure elasticsearch.yml file (Dir: /etc/elasticsearch/)

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0

Kibana

Download and install kibana-6.4.2-x86_64.rpm in operation (mfs-ops1) server
Configure kibana.yml file (Dir: /etc/kibana/)

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
logging.dest: /var/log/kibana/kibana-app-log.log

Open port 5601 to access kibana