Multi‐cluster Thanos Setup - datnguyendv/monitoring_tools GitHub Wiki

In this guide, we will simulate a multi-cluster monitoring setup using Thanos by emulating two clusters in a single Minikube environment. Each simulated cluster will be represented by a separate namespace:

singapore: represents a production Kubernetes cluster in the Singapore region
observability: represents a centralized monitoring cluster
please apply storage config step in this doc: https://github.com/datnguyendv/monitoring_tools/wiki/Integration-With-Thanos. Service account must have storage admin of one bucket.

Cluster: singapore

1. Install Prometheus

We install Prometheus in the singapore namespace using the Helm chart:

helm upgrade --install singapore-prom prometheus-community/kube-prometheus-stack \
  --namespace singapore \
  -f scrape-prometheus/helm/kps-mcls-singapore.yaml

2. Install Thanos Sidecar and Thanos Query

In the singapore namespace, we will only install the Thanos Sidecar (alongside Prometheus) and Thanos Query.

helm upgrade --install singapore-ts bitnami/thanos \
  --namespace singapore \
  -f thanos/thanos-mcls-singapore.yaml

This thanos-mcls-singapore.yaml should contain configurations only for query. Components like storegateway, compactor are omitted here.

Cluster: observability

We now simulate the observability cluster in the observability namespace.

1. Install Prometheus

Same as before, using Helm:

helm upgrade --install obser-prom prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --create-namespace \
  -f scrape-promethues/helm/kps-mcls-observer.yaml

2. Install Full Thanos Stack

In this namespace, we will deploy the full set of Thanos components:

Thanos Query (centralized)
Store Gateway
Compactor
Optional: Ruler, Receive, Bucket Web UI

helm upgrade --install obser-ts bitnami/thanos \
  --namespace observability \
  -f thanos/thanos-mcls-observer.yaml

In this setup, the centralized thanos-query in observability is configured to connect to the thanos-query in the singapore namespace using its gRPC endpoint.

Make sure thanos-query in observability includes the external store configuration for singapore:

query:
  stores:
    - singapore-thanos-query.singapore.svc.cluster.local:10901

This setup enables centralized querying and long-term metric storage across multiple simulated clusters, ideal for testing multi-cluster observability scenarios.

End of guide