Elasticsearch and Kibana - CloudCommandos/JohnChan GitHub Wiki

Elasticsearch serves as a data catalog to index and store Fluentd logs. Kibana enables the visualization of the stored data in Elasticsearch. This document will guide you in setting up Elasticsearch and Kibana to work with your Fluentd instance(s).

You already have a running Fluentd setup.

Download and install GPG key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Install prerequisite

sudo apt-get install apt-transport-https

Add repository

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update repository and install Elasticsearch

sudo apt-get update && sudo apt-get install elasticsearch kibana

Enable and start Elasticsearch and Kibana

sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
sudo systemctl enable kibana.service
sudo systemctl start kibana.service

Check that Elasticsearch responds to HTTP Get request from localhost. The default port is 9200.

curl localhost:9200

You should see something like this

{
  "name" : "hostname",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "SARHlClNQFO20BPwSAS1Ow",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Edit the network.host and 'discovery.seedhost' properties according to your setup

nano /etc/elasticsearch/elasticsearch.yml

...
network.host: 0.0.0.0
...
discovery.seed_hosts: ["10.0.1.77"]
...

Restart Elasticsearch

sudo systemctl restart elasticsearch.service

Now Elasticsearch should respond to external HTTP GET requests

Edit the server.host and 'elasticsearch.hosts' properties according to your setup

nano /etc/elasticsearch/elasticsearch.yml

...
server.host: "0.0.0.0"
...
elasticsearch.hosts: ["http://localhost:9200"]
...

Restart Kibana

sudo systemctl restart kibana.service

Default port for Kibana is 5601. You can access Kibana's web portal via: http://localhost:5601/app/kibana

You should add an output plugin configuration to your Fluentd instance(s) to forward logs to Elasticsearch. Change the host value according to your setup.

nano /etc/td-agent/td-agent.conf

...
## Output plugin: this is to send logs tagged with 'memory_log' to an Elasticsearch instance
<match memory_log>
  @type elasticsearch
  host localhost
  port 9200
  include_tag_key true
  tag_key @log_name
  logstash_format true
  flush_interval 10s
</match>
...

Reload Fluentd

sudo systemctl reload td-agent.service