use Logstash & Filebeat - SeoJeongHan/managed_log GitHub Wiki

Filebeat - 6.1.1

• filebeat 실행전 /filebeat/data/registry 파일을 삭제 후 실행해야 한다.
• stop : kill -9 'filebeat PID'

[irteam@hostname 14:57 오후~/apps]$wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.1.1-linux-x86_64.tar.gz
[irteam@hostname 14:57 오후~/apps]$tar xvzf filebeat-6.1.1-linux-x86_64.tar.gz
[irteam@hostname 14:57 오후~/apps]$ln -s filebeat-6.1.1-linux-x86_64 filebeat
[irteam@hostname 14:57 오후~/apps/filebeat]$mv ./filebeat.yml ./filebeat.yml.org
[irteam@hostname 14:57 오후~/apps/filebeat]$vim filebeat.yml
     
filebeat.prospectors:
 
- type: log
  enabled: true
  paths:
    - /xxx/xxx/xxx/A/*/xxx*
  document_type: A
 
output.logstash:
  hosts: ["127.0.0.1:5044"] //logstash install server IP : port
 
 
[[email protected] 04:53 오후~/apps/filebeat]$chmod 600 ./filebeat.yml
[[email protected] 04:53 오후~/apps/filebeat]$./filebeat -e -c filebeat.yml -d publish &

Logstash - 6.1.1

• logstash JDBC output plugin : https://github.com/theangryangel/logstash-output-jdbc
• stop : kill -9 'logstash PID'

[irteam@hostname 14:57 오후~/apps]$wget https://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz
[irteam@hostname 14:57 오후~/apps]$tar xvzf logstash-6.1.1.tar.gz
[irteam@hostname 14:57 오후~/apps/logstash/config]$vim A_log.conf
 
    input {
        beat{
            poat=> 5044
            type=> "A"
        }
    }
    filter {
      grok { //log file parsing filter regex
        match => { "message" => "%{TIMESTAMP_ISO8601:date} \[%{DATA:xxx}\] %{WORD:xxx}  %{WORD:xxx} %{USER:xxx} %{USER:xxx}\    u241B(?:A=%{USER:A})\u241B(?:B=%{DATA:B})\u241B(?:%{DATA:etc})"}
      }
    }
    output {
        jdbc {
            driver_class => "com.mysql.jdbc.Driver"
            connection_string => "jdbc:mysql://localhost:13306/A?user=admin01&password=1234" //DB Connection Info
            statement => [ "INSERT INTO table_name (date, A, B) VALUES(?, ?, ?)", "date", "A", "B"]
        }
    }
 
[irteam@hostname 02:38 오후~/apps/logstash]$nohup ./bin/logstash -f ./config/xxx_log.conf --config.reload.automatic > /dev/null 2>&1 &