Exposing IBM Storage Scale metrics to Prometheus and Grafana multiple clusters and multitenancy - IBM/ibm-spectrum-scale-bridge-for-grafana GitHub Wiki

This article explains the concepts behind monitoring several IBM Storage Scale clusters with one Prometheus server and one Grafana instance, and how to give a tenant of an IBM Storage Scale System (ESS) a view of their own metrics. Step-by-step instructions are in separate articles, linked in each section.

Common building blocks

Component	Role
`pmsensors` and `pmcollector`	Collect the performance data inside a Storage Scale cluster. The bridge queries the collector on REST port 9980.
grafana bridge	Translates the collected data for Prometheus (port 9250) and for the Grafana OpenTSDB data source (port 4242). One bridge talks to one `pmcollector`.
Prometheus	Scrapes the bridges.
Grafana	Queries Prometheus and shows the Storage Scale dashboards.

Two principles apply to every setup in this article:

One bridge per pmcollector. To monitor N clusters, run N bridges, each with the credentials of its own cluster.
Unique job names. Prometheus identifies a data source by its job name. Every bridge generates the same default job names, so a suffix per bridge keeps them apart. The bridge endpoint /scrape_configs.yml generates the scrape jobs for you, with a job_suffix parameter for this purpose. It requires Prometheus 2.43 or later, which supports scrape_config_files.

Concept 1: Several Storage Scale clusters, one Prometheus

/images/scale_prometheus_grafana_overview.png

An admin wants one monitoring stack for several clusters, for example a storage cluster, compute clusters, or clusters at several sites.

Prerequisite: The admin has full administrative access to all of these clusters. This is needed to install and configure a bridge on each cluster's pmcollector node and to use the cluster's own API key or credentials. If the admin cannot get such access to a cluster, for example because it belongs to another party, see Concept 2.

Each cluster has its own bridge, connected to that cluster's pmcollector.
One central Prometheus server scrapes all bridges. Each bridge's scrape jobs are stored in a separate file and included through scrape_config_files, so the central prometheus.yml is never overwritten. This also fits an existing enterprise Prometheus server.
Grafana uses the central Prometheus as its data source. The job name (suffix) tells the clusters apart in dashboards.

Step-by-step: Generate scrape configurations for multiple systems.

Concept 2: Multitenancy with a storage cluster (ESS) and client clusters

In a multitenant setup, tenant client clusters mount file systems from an IBM Storage Scale System (ESS), the storage cluster run by the provider. For the multitenancy concept, see the IBM multitenancy document.

The metrics of the ESS can contain data of several tenants. Access to them is controlled with API keys on the ESS pmcollector. Depending on who operates the monitoring stack, there are two ways to organize it:

	Provider view	Tenant view
Who operates the stack	one party owns the ESS, the tenant clusters and the monitoring stack	the client cluster admin of one tenant, with the ESS run by a provider
Prometheus and Grafana run on	the provider side	the tenant's client cluster side
Bridges	one per cluster, each against its own `pmcollector`. The ESS bridge uses an API key with provider access.	one on the client cluster, plus a second one that queries the ESS `pmcollector` with the tenant API key
Tenant API keys	not needed	required. They limit the tenant to the metrics related to that tenant.
Result	one stack shows the ESS and all tenant clusters	the tenant sees the client cluster and their own part of the ESS in their own Grafana
Step-by-step	Monitor multi-tenant environments from the provider side	Monitor the tenant view of an ESS from a client cluster

Provider view

/images/scale_multitenancy_provider_view.png

One party operates the whole stack and monitors the ESS and all tenant clusters that mount from it:

Each tenant cluster and the ESS has its own bridge, connected to that cluster's pmcollector. The ESS bridge uses an API key with provider access, which is not restricted to one tenant. Because one party owns everything, no per-tenant API key is needed.
One Prometheus server (2.43 or later) scrapes all bridges. Each bridge has its own scrape file with a unique job_suffix, for example _ess, _tenantA, _tenantB and _tenantC, referenced through scrape_config_files.
Grafana uses Prometheus as its data source. The provider dashboards show the ESS and all tenants. Optionally, use a folder or an organization per tenant.

Tenant view

/images/scale_multitenancy_client_view.png

The admin of a tenant's client cluster monitors the client cluster and only the tenant's part of the ESS, in their own stack:

The ESS provider admin creates a dedicated API key for each tenant and hands it over securely. Other tenants use their own keys and see only their own metrics.
The client cluster admin runs bridge 1 against the client cluster's own pmcollector, and a second bridge against the ESS pmcollector with the tenant API key. The second bridge runs on another host or port.
Prometheus scrapes both bridges, with job_suffix values such as _client and _ess_tenant. Grafana shows both views side by side.

Which one to choose

Choose the provider view when one party operates the whole stack and is allowed to see all tenant data. Choose the tenant view when the tenant operates their own stack and should see only the tenant-related ESS metrics.