This document provides a comprehensive step-by-step guide for integrating Thanos with Consul as a service discovery mechanism to monitor multiple Kubernetes clusters using Prometheus, Thanos Sidecars, and Thanos Query components.

This setup is useful for multi-cluster monitoring scenarios, where services (like Prometheus, Store Gateway, and Thanos Query) dynamically register to Consul, allowing other clusters to discover and query them in real-time.

🚀 Deployment Steps

Step 1: Deploy Consul Using Helm

We begin by deploying a Consul cluster to be used as a central service registry.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm upgrade --install consul hashicorp/consul \
  -f discovery-consul/helm/consul-server.yml \
  --namespace consul 

📝 The consul-server.yml should include configuration such as service annotations, storage class, replica count, and ACLs (if needed).

Step 2: Register Prometheus (Singapore Cluster) in Consul

To enable Thanos Query to discover Prometheus via Consul, register the Prometheus service manually using a predefined JSON file

curl --request PUT \
  --data @example/prometheus-singapore.json \
  http://<CONSUL_ADDRESS>:8500/v1/agent/service/register

Replace <CONSUL_ADDRESS> with the actual IP or DNS address of your Consul server.

##Step 3: Configure Consul Template via ConfigMap This step sets up a ConfigMap that holds the template and config for the Consul Template agent. This allows dynamic rendering of Thanos Store or Sidecar endpoints.

kubectl apply -f thanos/k8s/consul-config.yaml -n singapore
kubectl apply -f thanos/k8s/consul-tpl.yaml -n singapore

• consul-config.yaml: Defines how the Consul Template connects to Consul and where to render the output.
• consul-tpl.yaml: Contains the Go-Template file that dynamically generates the list of Thanos Store endpoints.

Step 4: Deploy Thanos Sidecar with Consul Integration

Next, deploy Thanos Sidecar configured to consume the rendered configuration from the Consul Template.

helm upgrade --install thanos-sidecar-singapore \
  -f thanos/helm/thn-mcls-singapore-cs.yaml \
  bitnami/thanos \
  --namespace singapore

Step 5: Register Thanos Query in Consul

After the Thanos Query component is updated to use the Consul template output, register it into Consul:

curl --request PUT \
  --data @example/thanos-query-singapore.json \
  http://<CONSUL_ADDRESS>:8500/v1/agent/service/register

Step 6: Repeat for Observer Cluster

Perform similar steps for the observer cluster to register both its Prometheus and Thanos Store Gateway services:

Register Prometheus in Observer

curl --request PUT \
  --data @example/prometheus-observer.json \
  http://<CONSUL_ADDRESS>:8500/v1/agent/service/register

Register Thanos Storage Gateway for Observer

curl --request PUT \
  --data @example/thanos-store-observer.json \
  http://<CONSUL_ADDRESS>:8500/v1/agent/service/register

Configure Consul Template via ConfigMap

kubectl apply -f thanos/k8s/consul-config.yaml -n observer
kubectl apply -f thanos/k8s/consul-tpl.yaml -n observer

Deploy Thanos Sidecar with Consul Integration

Next, deploy Thanos Sidecar configured to consume the rendered configuration from the Consul Template.

helm upgrade --install thanos-sidecar-observer \
  -f thanos/helm/thn-mcls-observer-cs.yaml \
  bitnami/thanos \
  --namespace singapore

🔍 Validation & Observability

After deployment:

1. Access Thanos Query UI at http://<QUERY_SERVICE_IP>:9090.
2. Check the "Stores" page to see if Prometheus and Store Gateway endpoints from both clusters are discovered.
3. Run PromQL queries to confirm metrics aggregation across clusters.

🧠 Tips & Best Practices

• Use health checks in the Consul service registration JSON to ensure reliability.
• Periodically clean up stale service registrations to avoid broken queries.
• For security: enable ACLs and TLS for Consul communication in production.
• Consider deploying Consul Template as a sidecar container or daemonSet depending on the topology.

End of guide.