Prometheus configuration structure - IBM/ibm-spectrum-scale-bridge-for-grafana GitHub Wiki

This article explains the structure and key sections of the Prometheus configuration file (prometheus.yml) when scraping performance metrics from the IBM Storage Scale Bridge for Grafana (grafana-bridge).

Overview

Prometheus operates via a pull model, periodically scraping HTTP REST endpoints exposed by metric exporters. The IBM Storage Scale Bridge for Grafana acts as a Prometheus exporter by querying IBM Storage Scale performance monitoring collectors (ZIMon) and translating sensor data into standard Prometheus metrics.

The prometheus.yml file defines:

  1. Global engine settings (default scrape intervals, rule evaluation intervals).
  2. Storage engine settings (out-of-order sample handling).
  3. Alertmanager and rule definitions (optional alert forwarding and recording rules).
  4. Scrape jobs (scrape_configs or scrape_config_files) specifying each Storage Scale sensor endpoint, target address, authentication, TLS certificates, and scrape interval.

Key Sections of prometheus.yml

1. global Section

The global block specifies default parameters applied to all scrape jobs and alerting rules unless overridden at the job level.

global:
  scrape_interval: 15s      # Default scrape interval for jobs that do not set one
  evaluation_interval: 15s  # How frequently evaluation rules are calculated
  query_log_file: /var/log/prometheus/query.log
  • scrape_interval: Default scraping frequency. Bridge sensor jobs define their own specific scrape_interval based on their sensor collection periods, but this global value serves as a fallback.
  • evaluation_interval: Evaluation frequency for alerting and recording rules.
  • query_log_file: Path to the log file where Prometheus records executed queries.

2. storage.tsdb Section

storage:
  tsdb:
    out_of_order_time_window: 2d

Important: The out_of_order_time_window setting is strongly recommended when monitoring IBM Storage Scale.

Why is this needed?

  • The bridge preserves original timestamps from IBM Storage Scale sensors (honor_timestamps: true).
  • The bridge's own internal metrics endpoint (/http_metrics) re-emits historical timestamped samples upon each scrape.
  • Setting out_of_order_time_window: 2d allows Prometheus TSDB to ingest metrics with slightly delayed or re-emitted timestamps without dropping them as out-of-order errors.

3. alerting and rule_files Sections

alerting:
  alertmanagers:
    - static_configs:
        - targets: null

rule_files: null
  • alertmanagers: Specifies Alertmanager instances where firing alerts are routed. If no Alertmanager is configured, targets can be set to null or omitted.
  • rule_files: Specifies paths to YAML files containing Prometheus recording or alerting rules.

4. scrape_configs Section

The scrape_configs section is a list of individual jobs, each targeting a specific IBM Storage Scale sensor endpoint or internal bridge metric endpoint.

Anatomy of a Bridge Scrape Job

Each sensor job is structured as follows:

scrape_configs:
  # 1. Prometheus self-monitoring job
  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

  # 2. IBM Storage Scale sensor scrape job
  - job_name: GPFSFilesystem
    metrics_path: /metrics_gpfs_filesystem
    scrape_interval: 10s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

Job Configuration Options Explained:

Field Description
job_name Unique identifier for the scrape job. When scraping multiple clusters or bridge instances, append a unique suffix (e.g. GPFSFilesystem_clusterA) to prevent collisions.
metrics_path HTTP REST endpoint path corresponding to the IBM Storage Scale sensor (e.g. /metrics_gpfs_filesystem, /metrics_gpfs_nsddisk, /metrics_gpfs_vfsx).
scrape_interval Scrape frequency for this sensor. Calculated by the bridge as max(sensor_period, 10s) (minimum 10 seconds).
honor_timestamps Must be set to true. Ensures Prometheus stores the exact timestamps provided by the Storage Scale performance monitor rather than the scrape time.
scheme Protocol used to connect to the bridge (http or https).
static_configs.targets List containing <bridge_ip>:<prometheus_port>.
basic_auth Basic authentication credentials required by the bridge (username and password or password_file).
tls_config Required when scheme: https. Specifies cert_file, key_file, and insecure_skip_verify.

HTTPS / TLS Configuration Example:

  - job_name: GPFSFilesystem
    metrics_path: /metrics_gpfs_filesystem
    scrape_interval: 10s
    honor_timestamps: true
    scheme: https
    tls_config:
      cert_file: /etc/prometheus/certs/cert.pem
      key_file: /etc/prometheus/certs/key.pem
      insecure_skip_verify: true
    basic_auth:
      username: scale_admin
      password_file: /etc/prometheus/secrets/bridge_password
    static_configs:
      - targets:
          - 10.0.100.11:9250

5. scrape_config_files Section (Prometheus >= 2.43)

In modular or multi-system environments, instead of inlining all scrape jobs in prometheus.yml, you can reference external scrape configuration files:

scrape_config_files:
  - /etc/prometheus/scrape_configs/*.yml

Each referenced file contains only a scrape_configs: list (generated by the bridge /scrape_configs.yml endpoint). This separates the global Prometheus server settings from cluster-specific scrape jobs.

Sensor Periods and Scrape Intervals

In IBM Storage Scale, sensors collect metrics at configured periods (e.g. 1s, 5s, 10s, 30s, 300s).

  • Minimum Scrape Interval (10s): High-frequency sensors (e.g., period 1s or 5s) are capped at a minimum scrape interval of 10s to prevent excessive query load on the bridge and pmcollector.
  • Longer Periods: Sensors with longer periods (e.g., 30s, 300s) use their configured period as the scrape_interval.
  • Inactive Sensors: Sensors configured with a period of 0 (disabled) are omitted from the generated configuration.

Complete Example prometheus.yml

Below is a complete, standalone prometheus.yml configuration monitoring an IBM Storage Scale cluster:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  query_log_file: /var/log/prometheus/query.log

storage:
  tsdb:
    out_of_order_time_window: 2d

alerting:
  alertmanagers:
    - static_configs:
        - targets: null

rule_files: null

scrape_configs:
  # Prometheus self-monitoring
  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

  # IBM Storage Scale Sensor Scrape Jobs
  - job_name: GPFSFilesystem
    metrics_path: /metrics_gpfs_filesystem
    scrape_interval: 10s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

  - job_name: GPFSNSDDisk
    metrics_path: /metrics_gpfs_nsddisk
    scrape_interval: 10s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

  - job_name: GPFSVFSX
    metrics_path: /metrics_gpfs_vfsx
    scrape_interval: 10s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

  - job_name: GPFSNode
    metrics_path: /metrics_gpfs_node
    scrape_interval: 10s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

  # Bridge Internal Metrics Jobs (Optional)
  - job_name: GrafanaBridgeHttpMetrics
    metrics_path: /http_metrics
    scrape_interval: 60s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

  - job_name: GrafanaBridgeInternalStats
    metrics_path: /internal_stats
    scrape_interval: 60s
    honor_timestamps: true
    scheme: http
    static_configs:
      - targets:
          - 10.0.100.11:9250
    basic_auth:
      username: scale_admin
      password: MyStrongPassword!

Related Documentation

Visit the IBM Storage Scale Knowledge Center for getting more info about the latest product updates

⚠️ **GitHub.com Fallback** ⚠️