Lab 9.1: Setting Up Metric-Based, Event Monitoring Systems, Prometheus and Grafana

We want to set up Prometheus and Grafana on our system so that we can view detailed metrics on a GUI

Step 1: Configure Prometheus

1. Download and Run Prometheus

Run this command to download the tar file (It should be a single line)

wget https://github.com/prometheus/prometheus/releases/download/v2.53.4/prometheus-2.53.4.linux-amd64.tar.gz

Unpack the tar file and move into the extracted directory:

tar -xvf prometheus-2.53.4.linux-amd64.tar.gz

cd prometheus-2.53.4.linux-amd64

Once in the prometheus directory we can review the configuration file, prometheus.yml

Close the configuration file and start Prometheus with the following command

sudo ./prometheus --config.file=prometheus.yml

You can access the Prometheus UI by going to this URL
http://13.75.177.126:9090

2. Create a Systemd Service for Prometheus

To manage Prometheus as a service, create a systemd unit file

From your Prometheus directory, copy Prometheus files to

/usr/local/bin/prometheus

Create a service file at

/etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus/prometheus \
--config.file=/usr/local/bin/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/data \
--web.console.templates=/usr/local/bin/prometheus/consoles
RestartSec=5s

[Install]
WantedBy=multi-user.target

Creating a writable directory for Prometheus data

By default, it uses ./data/ (relative to where you run it).
Create the directory in

/var/lib/prometheus
sudo mkdir -p /var/lib/prometheus/data

Make sure you set appropriate ownership

sudo chown -R prometheus:prometheus /var/lib/prometheus

Reload systemd, start Prometheus

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus

Use this command to then check the status of Prometheus

sudo service prometheus status

3. View Prometheus Web Interface

To access the Prometheus dashboard, go to

http://13.75.177.126:9090/targets?search=

In the above image we can see the target but in order to see data about the target we need to go to 'Graph' from the navbar then we can select either 'Table' and enter something like

promhttp_metric_handler_requests_total

in the text box then select 'Execute' to see a summary of collected data for the selected metric

Selecting the Graph tab will plot how this metric has changed over time

Step 2: Configure Node_Exporter

For Prometheus to get data from a target the target needs a service that provides data on request, you can find this in the /metrics URL. To create the service that supplies the data we'll use Node Exporter.

1. We can download Node Exporter using this command all on one line

wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz

2. Extract the downloaded tar file

Node_exporter needs a service file to manage the exporter service, it is very similar to the prometheus one and is located in

/etc/systemd/system/node_exporter.service

3. After the service file has been created the system need to reload and restart. Enter the following commands

sudo systemctl daemon-reload
sudo systemctl start node_exporter

After a restart it can be tested using the command

sudo systemctl status node_exporter

4. You can then go to

http://13.75.177.126:9090/targets?search=

If you look under Status then Targets you should see both systems running

5. Adding the Rule Groups to Your Prometheus Configuration

In the directory with the prometheus.yml file

/usr/local/bin/
Create two new files, rule1.yml and rule2.yml

Add Rule Group One to Rule1.yml

Add Rule Group 2 to Rule2.yml
I got stuck here because I didn't indent the 'alert' line twice

To check the rules are running correctly run the command below, you need to direct the command to the prometheus folder and then promtool

sudo ./prometheus/promtool check rules rule1.yml

sudo ./prometheus/promtool check rules rule2.yml

Step 3: Setting up Grafana

Grafana is a dashboard view that helps us to visualize metrics data

1: Install Grafana

First, install required dependencies and Grafana.

sudo apt-get install -y apt-transport-https software-properties-common

Add Grafana GPG key

sudo wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

2. Add Grafana APT repository

sudo echo "deb https://packages.grafana.com/oss/deb stable main" \
| sudo tee -a /etc/apt/sources.list.d/grafana.list

3. Update package list

sudo apt-get update

4. Install Grafana

sudo apt-get install grafana
sudo systemctl start grafana-server

Start and Enable Grafana

Enable and start the Grafana server to run it immediately and on boot.

sudo systemctl enable --now grafana-server
sudo systemctl start grafana-server

sudo systemctl status grafana-server
:sudo ufw allow 3000/tcp

Grafana will now be accessible at

http://13.75.177.126:3000/?orgId=1&from=now-6h&to=now&timezone=browser

Configure Grafana with Prometheus and add Prometheus Data Source

After logging into grafana with the default password select 'Add your first data source'

Then select 'Prometheus'

Set the server IP to be the backup server IP

http://13.75.177.126:9090

Click 'Save and Test' at the bottom of the page

Go to the home page and select the + to the right of the screen, then click Import Dashboard

Enter the dahsboard ID

Select Prometheus as the data source

Then click import

Check that Node_exporter is up at

http://13.75.177.126:9090/targets?search=

You should now see information on the Grafana dashboard

I had some issues getting information to show up on the dashboard I had put the rules files outside the Prometheus folder