Home - jvcastaneda/DevopsToolingConfig GitHub Wiki

Welcome to the DevopsToolingConfig wiki!

If the prometheus-node-exporter is running, integrating it with Prometheus involves ensuring Prometheus is configured to scrape the metrics provided by the node-exporter. Here's how you can set it up:

Check if the node-exporter service is deployed and accessible:

kubectl get svc -n monitoring | grep node-exporter

You should see a service like this:

prometheus-node-exporter   ClusterIP   10.0.0.1   <none>   9100/TCP

The kube-prometheus-stack uses ServiceMonitor resources to tell Prometheus how to scrape metrics. Check if one exists for node-exporter:

kubectl get servicemonitor -n monitoring | grep node-exporter

If it does not exist, create one. Here’s an example ServiceMonitor for node-exporter:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-node-exporter
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-node-exporter
  endpoints:
    - port: metrics
      interval: 15s

Apply this configuration:

kubectl apply -f node-exporter-servicemonitor.yaml

Prometheus needs to be configured to discover ServiceMonitor resources. Verify that the Prometheus instance is set to select the appropriate ServiceMonitor:

Check your Prometheus custom resource:

kubectl describe prometheus -n monitoring

Look for the serviceMonitorSelector field. It should include the label matching the ServiceMonitor for node-exporter (e.g., release: kube-prometheus-stack).

Prometheus automatically discovers targets through the ServiceMonitor. Verify this in the Prometheus UI:

1. Port-forward the Prometheus service:

   kubectl port-forward -n monitoring svc/<prometheus-service-name> 9090

2. Open the Prometheus UI: Navigate to http://localhost:9090.

3. Check the Targets: Go to Status > Targets and confirm that the node-exporter endpoints are listed and in the "UP" state.

Once the node-exporter targets are discovered and scraped, you can query metrics in Prometheus. Some common queries include:

• CPU Usage:

  rate(node_cpu_seconds_total{mode!="idle"}[5m])
  

• Memory Usage:

  node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
  

• Disk Usage:

  node_filesystem_free_bytes / node_filesystem_size_bytes * 100
  

If you do not see metrics in Prometheus:

1. Check Logs of node-exporter:

   kubectl logs -n monitoring daemonset/prometheus-node-exporter

2. Inspect ServiceMonitor: Ensure the labels on the ServiceMonitor match those on the node-exporter service.

3. Check Prometheus Logs:

   kubectl logs -n monitoring <prometheus-pod-name>

4. Manually Scrape Node Exporter: Access the metrics endpoint directly to ensure it is serving metrics:

   kubectl port-forward -n monitoring svc/prometheus-node-exporter 9100

   Visit http://localhost:9100/metrics in your browser.

This setup should ensure that prometheus-node-exporter metrics are scraped and ingested into Prometheus.