Setting Up Grafana & Prometheus for Multi‐Tenant Environments Storage and Client Clusters - IBM/ibm-spectrum-scale-bridge-for-grafana GitHub Wiki

This guide walks through end-to-end setup of the IBM Storage Scale Bridge for Grafana with Prometheus in a multi-tenant environment - where a storage cluster and one or more client clusters expose metrics independently and Prometheus on the storage node scrapes and merges both.

Overview

           ┌─────────────────────────────┐
           │       pmcollector node      │
           │  ┌───────────────────────┐  │
           │  │   grafana-bridge      │  │
           │  │   port: 9250          │  │
           │  └───────────────────────┘  │
           └─────────────┬───────────────┘
                         │ 
            ┌────────────┴────────────┐
            ▼                         ▼
   ┌─────────────────┐       ┌─────────────────┐
   │  Client Cluster │       │ Storage Cluster │
   └─────────────────┘       └────────┬────────┘
                                      │
                                      │
                                      ▼
                       ┌──────────────┌───────────────┐
                       │  Prometheus  │   Grafana     │
                       │  port: 9090  │   port: 3000  │
                       └──────────────└───────────────┘

Note: The grafana-bridge must be installed on the pmcollector node of each cluster. Prometheus scrapes metrics from both clusters; Grafana reads from Prometheus as its datasource.

Step 1 - Set Up Grafana Bridge on Both Clusters

Grafana Bridge must be installed, configured, and running on both the storage cluster and the client cluster. Install the bridge only on a pmcollector node of each cluster - the bridge reads performance data collected by the pmcollector daemon.

For RPM installation - yum install gpfs.grafana-bridge
For Ansible toolkit installation - Install Grafana Bridge with installation toolkit
For manual installation - Setup the grafana bridge on a classic IBM Storage Scale cluster

Key configuration files on every node where the bridge is installed:

File	Purpose
`/etc/grafanabridge/config.ini`	Bridge configuration (ports, authentication, logs, ...)
`/etc/systemd/system/grafana-bridge.service`	systemd unit that manages the bridge process

Step 2 - Install Grafana on the Storage Cluster

Download grafana.sh from this wiki repository and run it:

curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-bridge-for-grafana/master/wiki/scripts/grafana.sh
chmod +x grafana.sh
./grafana.sh

The script prompts for a Grafana version (default 12.0.0) and port (default 3000), detects the OS (RHEL/CentOS/Fedora, Ubuntu/Debian, SLES), installs the package, updates grafana.ini, opens the firewall port, and enables the systemd service.

Verify Grafana is running:

curl http://127.0.0.1:3000/api/health

Step 3 - Install Prometheus on the Storage Cluster

Download prometheus.sh from this wiki repository and run it:

curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-bridge-for-grafana/master/wiki/scripts/prometheus.sh
chmod +x prometheus.sh
./prometheus.sh

Common YAML parsing error

If promtool reports:

FAILED: parsing YAML file /tmp/prometheus.yml: yaml: unmarshal errors:
  line 281: cannot unmarshal !!seq into config.plain

The storage.tsdb block uses list syntax instead of map syntax. Fix it:

# Wrong - list syntax
storage:
  tsdb:
    - out_of_order_time_window: 2d

# Correct - map syntax
storage:
  tsdb:
    out_of_order_time_window: 2d

Validate after editing:

promtool check config /etc/prometheus/prometheus.yml
# SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Step 4 - Open Firewall Ports

Open firewall port 3000 (Grafana) & 9090 (Prometheus) on storage cluster unidirectional. Firewall port 9250 on both storage & all client clusters.

Step 5 - Configure Prometheus to Scrape Both Clusters

Prometheus natively supports loading scrape configs from separate files via scrape_config_files

5.1 Fetch each cluster's scrape config

The bridge returns a modular Prometheus scrape configuration. To fetch, configure, and automatically merge scrape configs for multiple clusters, follow the step-by-step instructions in Generate Prometheus scrape configs for multiple systems.

5.2 Validate and reload

promtool check config /etc/prometheus/prometheus.yml && systemctl restart prometheus

Step 6 - Verify Endpoints and Metrics Collection

Bridge endpoints

# Storage bridge
curl -k https://<storage_cluster_ip>:9250/endpoints
curl -k https://<storage_cluster_ip>:9250/metadata/time

# Client bridge
curl -k https://<client_cluster_ip>:9250/endpoints
curl -s -k https://<client_cluster_ip>:9250/metadata/sensormetrics | python3 -m json.tool

Note: Open the Grafana UI at http://<storage_cluster_ip>:3000, add your Prometheus datasource pointing to http://<storage_cluster_ip>:9090 and start building dashboards to see all the storage and client metrics in a same dashboard. Use the clusterName label in PromQL to filter by cluster:
gpfs_health_status{clusterName="storage-cluster"}
gpfs_health_status{clusterName="client-cluster"}