Fluentd & Fluentbit - CloudCommandos/JohnChan GitHub Wiki

Fluentbit is designed to be a log collector/forwarder while Fluentd is designed to be a log aggregator. A typical setup is to install Fluentbit in edge nodes to forward logs to a central logger such as a node running Fluentd. This guide demonstrates how to setup such an architecture.

Host1 has ip 10.0.1.77/24 and serves as our log aggregator node. Host2 has ip 10.0.1.78/24 and serves as our edge node. All hosts are running with Ubuntu 18.04.

Download and install Fluentd

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh

Enable and start Fluentd

sudo systemctl enable td-agent.service
sudo systemctl start td-agent.service

Check that Fluentd is running

sudo systemctl status td-agent.service

Add into Fluentd config

sudo nano /etc/td-agent/td-agent.conf

...
## Input plugin: This is to receive logs forwarded by Fluentbit
<source>
  @type forward
  @id fluentbit_forward
  port 24284
  bind 0.0.0.0
</source>

## Input plugin: This is to receive logs via HTTP POST
<source>
  @type http
  @id input_http
  port 8888
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

## Output plugin: This is to output received logs tagged with 'test', 'test.*', 'test.*.*' etc.
## to the path specified 
<match test.**>
  @type file
  @id output_file_test
  path /var/log/td-agent/${tag}/%Y%m%d.%H%M
  append true
  <buffer tag,time>
    timekey 1m
    timekey_wait 10s
  </buffer>
</match>
...

Reload Fluentd

sudo systemctl reload td-agent.service

Test the HTTP listener

curl -X POST -d 'json={"foo":"bar"}' http://localhost:8888/test
curl -X POST -d 'json={"foo":"bar"}' http://localhost:8888/test.hi1
curl -X POST -d 'json={"foo":"bar"}' http://localhost:8888/test.hi1.sub1

Wait for timekey + timekey_wait amount of time then check for the logs inside /var/log/td-agent/

Add apt-key

wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

Update /etc/apt/sources.list

nano /etc/apt/sources.list

...
deb https://packages.fluentbit.io/ubuntu/bionic bionic main
...

Update repository

sudo apt-get update

Install Fluentbit

sudo apt-get install td-agent-bit

Enable and start Fluentbit

sudo systemctl enable td-agent-bit.service
sudo systemctl start td-agent-bit.service

Check that Fluentbit is running

sudo systemctl status td-agent-bit.service

Add into Fluentbit config

sudo nano /etc/td-agent-bit/td-agent-bit.conf

## Input plugin: This is to read in the latest changes of the specified file
## based on the offset stored in the specified db file
[INPUT]
    Name tail
    Tag  test.1
    Path /var/log/syslog
    Refresh_Interval 60s
    DB /etc/td-agent-bit/DB/test.1.db

## Output plugin: This is to forward logs that are tagged with 'test', 'test.*', 'test.*.*' etc.
## to the specified Host:Port
## Target Host:Port should be running Fluentd's or Fluentbit's forward input plugin.
[OUTPUT]
    Name  forward
    Match test.**
    Host          10.0.1.77
    Port          24284
    Self_Hostname fluentbit_Host2

Make directory for the tail plugin's db file. This is optional depending on your db file path.

sudo mkdir /etc/td-agent-bit/DB

Restart Fluentbit (Reload is not available)

sudo systemctl restart td-agent-bit.service

Now Host2's syslog should be forwarded to Host1's Fluentd periodically and stored under /var/log/td-agent/ or as specified in the output plugin config.

Fluentbit's binary can be found at /opt/td-agent-bit/bin/td-agent-bit

If Host2 is running Docker containers, you can expose container log files through local volume mounts then use Fluentbit daemon to access their contents.

Install and run Fluentbit as part of the container.

Store container log files in a mounted shared volume (e.g. NFS), then
a). Run Fluentbit container accessing the log files by mounting them from shared volume
OR
b). Mount log files to VM through shared volume and access them with Fluentbit daemon.