Generate Prometheus config.yml flile content using grafana bridge - IBM/ibm-spectrum-scale-bridge-for-grafana GitHub Wiki

The IBM Storage Scale Bridge for Grafana (grafana-bridge) can automatically pre-generate a complete, ready-to-use Prometheus configuration file (prometheus.yml) in YAML format based on the discovered IBM Storage Scale cluster configuration and active sensors.

This is the recommended approach when you are setting up Prometheus to monitor a single IBM Storage Scale system and want a turnkey configuration file.

Prerequisites

1. PyYAML Python Package

The bridge uses the PyYAML package to generate the YAML output. Verify if PyYAML is installed on the host running grafana-bridge:

pip list | grep -i PyYAML

If not installed, install it and restart the grafana-bridge service:

pip install PyYAML
systemctl restart grafana-bridge

2. Prometheus Exporter Port & Authentication

By default, the bridge's Prometheus exporter listens on port 9250. Ensure you have the credentials (username and password or API key) configured in config.ini / zk.cfg if basic authentication is enabled.

Query Parameters

The GET /prometheus.yml endpoint accepts the following optional query parameters:

Parameter Type / Format Default Description
job_suffix String (^[A-Za-z0-9_.:-]{1,64}$) (empty) Appends a literal suffix to every bridge-generated job_name (e.g. GPFSFilesystem_scale01). The static prometheus self-scrape job is never suffixed.
internal_metrics Boolean (true | false) false When set to true, appends two additional scrape jobs for the bridge's internal telemetry: GrafanaBridgeHttpMetrics (/http_metrics) and GrafanaBridgeInternalStats (/internal_stats), scraped every 60 s. Requires http_metrics_enabled in analytics.py.
target Host or Host:Port (auto-detected) Overrides the scrape target host or host:port in static_configs. When omitted, the bridge detects its outbound IP address. Required when running in a container (Podman/Docker) to replace the internal container IP with the reachable host IP/FQDN.

Query Parameter Examples:

# Add a cluster suffix
GET /prometheus.yml?job_suffix=_clusterA

# Include bridge internal observability metrics
GET /prometheus.yml?internal_metrics=true

# Combine suffix, internal metrics, and custom target host
GET /prometheus.yml?job_suffix=_clusterA&internal_metrics=true&target=10.0.100.11:9250

How to Retrieve the Configuration

Method 1: Using a Web Browser

Enter the prometheus.yml endpoint in the address bar of a browser using the Prometheus Exporter plugin port:

http(s)://<grafana-bridge-ip>:<prometheus-exporter-port>/prometheus.yml

/images/prometheus_endpoint.png

Method 2: Using curl REST API Call

HTTP with Basic Authentication:

curl -v http://<grafana-bridge-ip>:9250/prometheus.yml \
     -u scale_admin:MyStrongPassword!

Bridge Running in a Container (Podman / Docker):

When the bridge runs inside a container with port publishing (e.g. -p 9250:9250), pass ?target=<container-host-ip> so that the generated target points to the container host rather than the private internal container IP:

curl -sf "http://<container-host-ip>:9250/prometheus.yml?target=<container-host-ip>" \
     -u scale_admin:MyStrongPassword!

HTTPS with TLS Certificates:

curl -v --cacert /etc/prometheus/certs/ca.pem \
     -u scale_admin:MyStrongPassword! \
     https://<grafana-bridge-ip>:9250/prometheus.yml

Step-by-Step: Deploying prometheus.yml to the Prometheus Server

Follow these steps on your Prometheus server host:

Step 1: Download and Save the Configuration

Fetch the configuration directly into /etc/prometheus/prometheus.yml:

curl -sf http://<grafana-bridge-ip>:9250/prometheus.yml \
     -u scale_admin:MyStrongPassword! \
     -o /etc/prometheus/prometheus.yml

Step 2: Validate the Configuration

Use promtool to verify that the generated YAML file is syntactically valid:

promtool check config /etc/prometheus/prometheus.yml

Expected output:

Checking /etc/prometheus/prometheus.yml
  SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Step 3: Restart or Reload Prometheus

Apply the new configuration by restarting the service or triggering a hot reload:

# Restart via systemd
systemctl restart prometheus

# Or trigger hot reload (if --web.enable-lifecycle is enabled)
curl -XPOST http://localhost:9090/-/reload

Example Generated Output

Below is an example of the YAML configuration generated by the bridge:

global:
  evaluation_interval: 15s
  query_log_file: /var/log/prometheus/query.log
  scrape_interval: 15s

storage:
  tsdb:
    out_of_order_time_window: 2d

alerting:
  alertmanagers:
  - static_configs:
    - targets: null

rule_files: null

scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090

- basic_auth:
    password: MyStrongPassword!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSFilesystem
  metrics_path: /metrics_gpfs_filesystem
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: MyStrongPassword!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSNSDDisk
  metrics_path: /metrics_gpfs_nsddisk
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: MyStrongPassword!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSVFSX
  metrics_path: /metrics_gpfs_vfsx
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

- basic_auth:
    password: MyStrongPassword!
    username: scale_admin
  honor_timestamps: true
  job_name: GPFSNode
  metrics_path: /metrics_gpfs_node
  scheme: http
  scrape_interval: 10s
  static_configs:
  - targets:
    - 10.0.100.11:9250

Related Documentation