[Learning Note] [ELK] Configure Filebeat to send log lines to Logstash - Gukie/building-recommend GitHub Wiki

• refer: https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html#configuring-grok-filter

outline

• Filebeat collect log, and then output to Logstash
• A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write the data to a destination

Tips about the configuration

FileBeat configuration

1. log prospector should be enabled, otherwise, log will not collected, and result in nothing output in Logstash.

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  #enabled: false
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- d:\\data\\elk\\logstach\\logstash-tutorial.log
    - d:\data\elk\logstach\logstash-tutorial.log

2. comment out Elasticsearch output, since there is only one output should be used at one time.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
  #hosts: ["0.0.0.0:5044"]
  #hosts: ["localhost:9600"]

logstash pipeline configuration

1. the input port should be the same as the output configured in filebeat.yml in Filebeat.

This port is 5043 in the refer doc, which is not correct, it should be 5044

2. filebeat data cache

• Filebeat data will be cached under {filebeat.install.dir}/data after collecting.
• If the data is cached, the data will not be output to Logstash.
• If you want to make the data to output in Logstash again, clear the cached data by removing the registry file under {filebeat.install.dir}/data