Generate Prometheus scrape configs for multiple systems - IBM/ibm-spectrum-scale-bridge-for-grafana GitHub Wiki

When monitoring multiple IBM Storage Scale clusters (e.g., storage cluster + client/compute clusters, or multi-site deployments), or when integrating IBM Storage Scale metrics into an existing enterprise Prometheus server, you should not overwrite the central prometheus.yml configuration file.

Instead, use the Grafana Bridge GET /scrape_configs.yml endpoint combined with the Prometheus scrape_config_files feature (available in Prometheus 2.43 and later).

The /scrape_configs.yml endpoint generates a YAML snippet containing only the scrape_configs list corresponding to active Storage Scale sensors and optional bridge telemetry.

On your Prometheus server, edit /etc/prometheus/prometheus.yml to include scrape_config_files and out_of_order_time_window:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

storage:
  tsdb:
    out_of_order_time_window: 2d   # Required for historical/internal metrics

scrape_configs:
  # Central Prometheus self-monitoring
  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

# Include modular scrape configuration files for all Storage Scale clusters
scrape_config_files:
  - /etc/prometheus/scrape_configs/*.yml

Create the directory for external scrape configurations:

mkdir -p /etc/prometheus/scrape_configs

For each IBM Storage Scale cluster, fetch its scrape configuration using curl with a unique job_suffix.

# 1. Fetch scrape jobs for Cluster A (Storage Cluster)
curl -sf "http://10.0.100.11:9250/scrape_configs.yml?job_suffix=_clusterA&internal_metrics=true" \
     -u scale_admin:PasswordA! \
     -o /etc/prometheus/scrape_configs/scale_clusterA.yml.tmp \
  && mv /etc/prometheus/scrape_configs/scale_clusterA.yml.tmp \
        /etc/prometheus/scrape_configs/scale_clusterA.yml

# 2. Fetch scrape jobs for Cluster B (Client Cluster)
curl -sf "http://10.0.200.22:9250/scrape_configs.yml?job_suffix=_clusterB&internal_metrics=true" \
     -u scale_admin:PasswordB! \
     -o /etc/prometheus/scrape_configs/scale_clusterB.yml.tmp \
  && mv /etc/prometheus/scrape_configs/scale_clusterB.yml.tmp \
        /etc/prometheus/scrape_configs/scale_clusterB.yml

Containerized Bridges (Podman / Docker): If a bridge runs inside a container with published ports, add &target=<container-host-ip> to the curl request so Prometheus scrapes the reachable container host IP rather than the internal container virtual IP:

curl -sf "http://10.0.100.11:9250/scrape_configs.yml?target=10.0.100.11&job_suffix=_clusterA&internal_metrics=true" \
     -u scale_admin:PasswordA! \
     -o /etc/prometheus/scrape_configs/scale_clusterA.yml.tmp \
  && mv /etc/prometheus/scrape_configs/scale_clusterA.yml.tmp \
        /etc/prometheus/scrape_configs/scale_clusterA.yml

Tip – Atomic File Replacement: Notice the use of .tmp and mv. The atomic mv operation prevents Prometheus from reading a partially downloaded or incomplete configuration file during a scrape reload.

Validate the combined configuration with promtool and reload Prometheus without restarting:

promtool check config /etc/prometheus/prometheus.yml \
  && curl -sf -XPOST http://localhost:9090/-/reload

Below is an example of what /etc/prometheus/scrape_configs/scale_clusterA.yml contains:

# Generated by IBM Storage Scale bridge for Grafana.
# Do not edit, changes will be lost on the next update.
# Reference this file in prometheus.yml under scrape_config_files.
scrape_configs:
- basic_auth:
    password: PasswordA!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSFilesystem_clusterA
  metrics_path: /metrics_gpfs_filesystem
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: PasswordA!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSNSDDisk_clusterA
  metrics_path: /metrics_gpfs_nsddisk
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: PasswordA!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSNode_clusterA
  metrics_path: /metrics_gpfs_node
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: PasswordA!
    username: scale_admin
  honor_timestamps: true
  job_name: GrafanaBridgeHttpMetrics_clusterA
  metrics_path: /http_metrics
  scheme: http
  scrape_interval: 60s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: PasswordA!
    username: scale_admin
  honor_timestamps: true
  job_name: GrafanaBridgeInternalStats_clusterA
  metrics_path: /internal_stats
  scheme: http
  scrape_interval: 60s
  static_configs:
  - targets:
    - 10.0.100.11:9250

When cluster sensors change (e.g. new sensors activated or sensor periods altered), the scrape configurations can be refreshed automatically via a cron job or systemd timer.

# Refresh scrape configurations daily at 02:00
0 2 * * * root /usr/local/bin/refresh_scale_scrape_configs.sh >/var/log/refresh_scale_scrape.log 2>&1

#!/bin/bash
set -e

CONFIG_DIR="/etc/prometheus/scrape_configs"
mkdir -p "$CONFIG_DIR"

# Cluster A
curl -sf "http://10.0.100.11:9250/scrape_configs.yml?job_suffix=_clusterA&internal_metrics=true" \
     -u scale_admin:PasswordA! \
     -o "$CONFIG_DIR/scale_clusterA.yml.tmp"
mv "$CONFIG_DIR/scale_clusterA.yml.tmp" "$CONFIG_DIR/scale_clusterA.yml"

# Cluster B
curl -sf "http://10.0.200.22:9250/scrape_configs.yml?job_suffix=_clusterB&internal_metrics=true" \
     -u scale_admin:PasswordB! \
     -o "$CONFIG_DIR/scale_clusterB.yml.tmp"
mv "$CONFIG_DIR/scale_clusterB.yml.tmp" "$CONFIG_DIR/scale_clusterB.yml"

# Validate and reload
promtool check config /etc/prometheus/prometheus.yml
curl -sf -XPOST http://localhost:9090/-/reload
echo "Scrape configurations updated successfully at $(date)"

• Prometheus configuration structure
• Generate prometheus.yml for a single system
• HTTP REST API endpoint: /scrape_configs.yml
• HTTP REST API endpoint: /prometheus.yml