Configuring PrometheusExporter for scraping Linux sensors metrics - IBM/ibm-spectrum-scale-bridge-for-grafana GitHub Wiki

By default, the IBM Storage Scale bridge for Grafana PrometheusExporter plugin is configured to scrape GPFS sensor metrics. Nevertheless, the IBM Storage Scale Performance Monitoring Tool (ZiMon) also collects Linux operating system metrics-such as CPU utilization and memory usage-on every node in the cluster. You can add the endpoints for Linux sensors to the bridge configuration manually and let Prometheus pull them as well. However, please note that it is generally recommended to use the standard Prometheus NodeExporter to collect Linux-specific OS metrics instead.

Example configuration for adding CPU and memory endpoints to prometheus_endpoints.json

By default, prometheus_endpoints.json lists only GPFS sensors (e.g. GPFSFilesystem, GPFSDisk, ...)

Two additional endpoints are available for Linux OS metrics:

Endpoint	Sensor	Description
`/metrics_cpu`	`CPU`	Per-node CPU utilisation metrics collected by ZiMon
`/metrics_memory`	`Memory`	Per-node memory usage metrics collected by ZiMon

Prometheus scrape job configuration

Append the following jobs to the scrape_configs section of your prometheus.yml. Replace <grafana_bridge_ip> with the hostname or IP address of the host running the grafana-bridge.

Recommended scrape interval: 10s
A shorter interval may cause Prometheus to run into performance issues during data retrieval because each scrape issues a live ZiMon query.

HTTP (with Basic Authentication)

  - job_name: 'CPU'
    scrape_interval: 10s
    honor_timestamps: true
    metrics_path: '/metrics_cpu'
    scheme: http
    basic_auth:
      username: 'scale_admin'
      password: 'TXlWZXJ5U3Ryb25nUGFzc3cwcmQhCg=='
    static_configs:
    - targets: ['<grafana_bridge_ip>:9250']

  - job_name: 'Memory'
    scrape_interval: 10s
    honor_timestamps: true
    metrics_path: '/metrics_memory'
    scheme: http
    basic_auth:
      username: 'scale_admin'
      password: 'TXlWZXJ5U3Ryb25nUGFzc3cwcmQhCg=='
    static_configs:
    - targets: ['<grafana_bridge_ip>:9250']

HTTPS (with TLS and Basic Authentication)

  - job_name: 'CPU'
    scrape_interval: 10s
    honor_timestamps: true
    metrics_path: '/metrics_cpu'
    scheme: https
    tls_config:
      cert_file: /etc/prometheus/certs/cert.pem
      key_file: /etc/prometheus/certs/privkey.pem
      insecure_skip_verify: true
    basic_auth:
      username: 'scale_admin'
      password_file: /etc/prometheus/certs/basic_auth
    static_configs:
    - targets: ['<grafana_bridge_ip>:9250']

  - job_name: 'Memory'
    scrape_interval: 10s
    honor_timestamps: true
    metrics_path: '/metrics_memory'
    scheme: https
    tls_config:
      cert_file: /etc/prometheus/certs/cert.pem
      key_file: /etc/prometheus/certs/privkey.pem
      insecure_skip_verify: true
    basic_auth:
      username: 'scale_admin'
      password_file: /etc/prometheus/certs/basic_auth
    static_configs:
    - targets: ['<grafana_bridge_ip>:9250']

Restart the Prometheus service after modifying the prometheus.yml file:

systemctl restart prometheus

Verify the endpoints are reachable

After configuring Prometheus you can verify that the grafana-bridge exposes the new endpoints by querying them directly:

# HTTP
curl -u scale_admin:<password> http://<grafana_bridge_ip>:9250/metrics_cpu
curl -u scale_admin:<password> http://<grafana_bridge_ip>:9250/metrics_memory

# HTTPS
curl -k -u scale_admin:<password> https://<grafana_bridge_ip>:9250/metrics_cpu
curl -k -u scale_admin:<password> https://<grafana_bridge_ip>:9250/metrics_memory

A successful response will return Prometheus-format text lines beginning with # HELP and # TYPE

You can also list all registered PrometheusExporter endpoints via the REST API:

curl -u scale_admin:<password> http://<grafana_bridge_ip>:9250/exporter_metrics_endpoints