Alerting rules (Prometheus) - toge510/homelab GitHub Wiki

Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service.

Defining alerting rules

groups:
- name: node
  rules:
    - alert: DiskSpaceAlert
      expr: 100 - ((node_filesystem_avail_bytes{mountpoint="/"} * 100) / node_filesystem_size_bytes{mountpoint="/"}) > 15
      for: 10s  # Trigger alert if condition met for 10s 
      labels:
        severity: critical
        team: homelab
      annotations:
        summary: "Instance {{ $labels.instance }} : High Disk Space Usage Instance"
        description: "Instance {{ $labels.instance }} of job {{ $labels.job }} : Disk space usage on root filesystem is above 15%"
        message: "Instance {{ $labels.instance }} is seeting high disk usage, currently used disk space: {{.Value}}%"

• for: causes Prometheus to wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element. Elements that are active, but not firing yet, are in the pending state.
• labels: allows specifying a set of additional labels to be attached to the alert.
• annotations : specifies a set of informational labels that can be used to store longer additional information such as alert descriptions.

The $labels variable holds the label key/value pairs of an alert instance. The $value variable holds the evaluated value of an alert instance. In this case,

• $labels: map[device:/dev/mapper/ubuntu--vg-ubuntu--lv fstype:ext4 instance:node_exporter:9100 job:node mountpoint:/]
• $value: value of expr

ALERTING RULES