#268: Implement Prometheus and Grafana Monitoring System - Jenrite/OE2-project-group-B GitHub Wiki

All actions done on backup machine

Download and unpack Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.53.4/prometheus-2.53.4.linux-amd64.tar.gz
tar -xvf prometheus-2.53.4.linux-amd64.tar.gz
cd prometheus-2.53.4.linux-amd64

image

Start Prometheus

sudo ./prometheus --config.file=prometheus.yml

image

Create Systemd Service

sudo cp -r . /usr/local/bin/prometheus

Then create prometheus.service file and add the following

image

Create Prometheus and data directory, create Prometheus user and give correct permissions to directory

image

Systemctl can now be used to control Prometheus

image

Node Exporter Setup

wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz

After unpacked, cd in and run the following:

cd node_exporter-1.9.1.linux-amd64/
./node_exporter

While this is running, we can see the exporter provides data at http://13.75.176.89:9090/

Copy executable to bin/

cp node_exporter /usr/local/bin/

Same as with prometheus service, we need a node-exporter user, a /var/lib/node-exporter/data directory, and a node-exporter.service file in /etc/systemd/system. The following does this:

sudo useradd --no-create-home --shell /bin/false node-exporter
sudo mkdir -p /var/lib/node-exporter/data
sudo chown -R node-exporter:node-exporter/var/lib/node-exporter
cd /etc/systemd/system
sudo nano node-exporter.service

and add the following

[Unit]
Description=Node Exporter
Documentation=https://prometheus.io/docs/guides/node-exporter/
Wants=network-online.target
After=network-online.target

[Service]
User=node-exporter
Group=node-exporter
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:9100

[Install]
WantedBy=multi-user.target

image

Current issue is, both services are running, but Prometheus is not seeing the target node-exporter. I can access both services via the browser.

I was editing the wrong .yml file, i was editing in the intial directory, not the copied directory. So i have added

  - job_name: "node-exporter"
    static_configs:
      - targets: ["localhost:9100"]

to the prometheus.yml file in /usr/local/bin/prometheus/

Adding Rules

image

image

image

Grafana Setup

# Install dependencies
sudo apt-get install -y apt-transport-https software-properties-common
# Add Grafana GPG key
sudo wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
# Add Grafana APT repository
sudo echo "deb https://packages.grafana.com/oss/deb stable main" \ | sudo tee -a /etc/apt/sources.list.d/grafana.list
# Update package list
sudo apt-get update
# Install Grafana
sudo apt-get install grafana
sudo systemctl start grafana-server

From here I have not been able to access it from the browser at http://13.75.176.89:3000/. I have adjusted the following in the /etc/grafana/grafana.ini:

# The HTTP address to bind to
http_addr = 0.0.0.0  # This ensures Grafana listens on all interfaces.
http_port = 3000     # The port to listen on (default is 3000)

The fix (got from Ola) was adjusting the firewall settings with sudo ufw allow 3000

image

username: admin

password: same as backup-b password

Grafana Dashboard Config

  • Menu -> Connections -> Data Sources -> Add New Data Source

  • Select Prometheus

  • url = http://13.75.176.89:9090

  • Save and Test

  • Click + icon and select import

  • Enter dashboard id (1860 for Node Explorer Full)

  • Select Prometheus data source

  • Click Import

image