Lab 3.2 Configuring Filebeat - Hsanokklis/2023-2024-Tech-journal GitHub Wiki

Helpful Info

The Public IPv4 address will change every new session. Current IPv4 address in use: 54.159.208.77

to access your instance ssh -i hannelore-elk-key.pem ubuntu@public key

Private IPv4 address is : 172.31.87.23

When next login your system make sure to start everything again

Start in this order: Elasticsearch, Logstash, Kibana

To get to the kibana dashboard type http://public-ip:5601 into your browser

Helpful Vocab

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing. As with our Metricbeat lab - we will be forwarding them directly to Elasticsearch.

Lightweight Shipper *

Step 1: Configure Filebeat

sudo apt-get install filebeat

The filebeat config file is filebeat.yml. It should be in /etc/filebeat

Edit filebeat.yml

In the Kibana section -->
- uncomment hosts
- change localhost to the IP of your ELK server
In the Output-Elasticsearch section -->
- change the localhost to the IP of your ELK server

Filebeat also uses modules. In this case, the modules will identify and send to data from logs for different services. In our lab, we will again use "system" which sends "system" logs from /var/log including auth.log and syslog. And "apache" which sends the Apache log data from /var/log/apache2Filebeat also uses modules.

In this case, the modules will identify and send to data from logs for different services. In our lab, we will again use "system" which sends "system" logs from /var/log including auth.log and syslog. And "apache" which sends the Apache log data from /var/log/apache2

sudo filebeat modules list will show all the supported modules.
sudo filebeat modules enable command to specify apache and system

sudo filebeat test config -e shows you config info and output should end with "Config OK"
sudo filebeat setup -e to load the recommended index template for writing to Elasticsearch and deploy the sample dashboards for visualizing the data in Kibana.

Step 2: Starting Filebeat

Start Filebeat

sudo service metricbeat start

In the side navigation, click Discover. To see the filebeat data make sure the predefined filebeat- index pattern is selected.* You should see logs from the filebeat Agent showing up.

`SUBMIT Screenshot of filebeat data`

In the side navigation, click Dashboard

Find one relevent to apache

`Submit a screenshot`

[Filebeat Apache] Access and error logs ECS

Find one relevant to System

`Submit a screenshot`

[Filebeat System] SSH login attempts ECS

Step 3: Filebeat Analysis

Open a new terminal on your workstation/laptop. Use ssh to login and logout to your Ubuntu server. Include some bad passwords to generate failed logins
Open a browser on your workstation and make some connections to the Ubuntu web server (via http://public-ip:80)
Go back to Discover and view the Filebeat logs
Click on the Search bar and explore options to find the successful and failed SSH logins.

Create a search filter that displays log results for failed logins

`Submit a screenshot`

event.action: ssh_login AND event.outcome: "failure" AND user.name:"hannelore"

Failures from using the username hannelore when trying to ssh instead of ubuntu

Create another search filter that lets you find the apache logs from your browser connections based on your workstation IP

`Submit a screenshot`

I found that I could search by message or source IP address which was (184.171.147.147)

Go back to dashboard

Find a dashboard that displays details on the SSH logins