Monitoring a Single Kubernetes Cluster - datnguyendv/monitoring_tools GitHub Wiki

This tutorial demonstrates how to monitor a single Kubernetes cluster using Prometheus (via kube-prometheus-stack) with custom exporters for both node metrics and MongoDB metrics. For this example, we’ll simulate the setup in Minikube, and assume the cluster name is singapore. It will be the name of helm.

For reference architecture, see this diagram.

Prepare Namespaces

We will create two namespaces:

kubectl create namespace monitoring
kubectl create namespace mongodb

Install Prometheus with Helm

We’ll use the official kube-prometheus-stack Helm chart to deploy Prometheus and related components into the monitoring namespace.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install singapore prometheus-community/kube-prometheus-stack -f scrape-prometheus/helm/kps-scls.yaml -n monitoring

After installation, Prometheus will automatically collect node-level metrics and Kubernetes control plane metrics.

Access Prometheus and Verify Setup

Port-forward Prometheus UI to your localhost:

kubectl port-forward -n monitoring svc/singapore-kube-prometheus-prometheus 9090

Then open http://localhost:9090/ in your browser.

To verify that node metrics are being collected, try running this query:

node_cpu_seconds_total

You should see time series data from all nodes in your Minikube cluster.

Deploy Grafana visualization

helm install singapore-graf grafana/grafana -f visualization-grafana/helm/grafana.yaml -n monitoring

Access Grafana and Verify Setup

Port-forward Prometheus UI to your localhost:

kubectl port-forward -n monitoring svc/singapore-graf-grafana 9090

Then open http://localhost:9090/ in your browser.

Add prometheus as a datasource.

To verify that node metrics are being collected, try running this query:

node_cpu_seconds_total

Deploy MongoDB Example

Let’s now deploy MongoDB into the mongodb namespace:

kubectl -n mongodb create deployment mongodb --image=mongodb/mongodb-community-server:latest
kubectl -n mongodb expose deployment mongodb --port=27017 --target-port=27017

Ensure that the MongoDB pod and service are running correctly:

kubectl get pods -n mongodb
kubectl get svc -n mongodb

Create mongodb user for mongdb-exporter service:

db.createUser({
  user: "exporter",
  pwd: "exporterpass",
  roles: [
    { role: "clusterMonitor", db: "admin" },           // required for serverStatus
    { role: "readAnyDatabase", db: "admin" }
  ]
})

Add MongoDB Exporter

Install a MongoDB exporter (e.g., [bitnami/mongodb-exporter](https://github.com/bitnami/bitnami-docker-mongodb-exporter)) in the same namespace.

Or using a manifest-based setup, deploy it like this:

kubectl -n mongodb apply -f metrics-exporter/mongodb/k8s/deployment.yaml
kubectl -n mongodb apply -f metrics-exporter/mongodb/k8s/service.yaml

Make sure the MongoDB exporter is pointing to your MongoDB instance via environment variables or connection string.

Add a ServiceMonitor for MongoDB Exporter

ServiceMonitor is custom CRD of prometheus operator. When apply ServiceMonitor, prometheus will scrape resource automaticaly. Ensure namespaceSelector.matchNames and metadata.labels.release includes:

...
metadata:
  labels:
    release: singapore  <--- name of release of prometheus
...
  namespaceSelector:
    matchNames:
      - mongodb <--- it must be namespace of exporter deployment
...

Apply ServiceMonitor:

kubectl -n monitoring apply -f scrape-prometheus/helm/service_monitor/mongo-exporter.yaml

Verify MongoDB Metrics in Prometheus

Wait a few seconds, then go back to Prometheus UI and try this query:

mongodb_up

Or explore metrics such as:

mongodb_ss_mem_resident_bytes
mongodb_mongod_op_counters_total

If data appears, your setup is working!

End of guide.