Prometheus Grafana Project Setup - pranavkumarpk01/MD-DevOps GitHub Wiki

📊 Jenkins Monitoring with Prometheus & Grafana

🚀 Overview

Monitoring Jenkins helps in keeping track of the health of Jenkins servers and jobs. Integrating Prometheus and Grafana into Jenkins allows you to monitor metrics like job status, build duration, failure rate, and server health.

❓ Why Integrate Prometheus and Grafana with Jenkins?

  • Real-time monitoring of Jenkins builds and jobs
  • Visualize Jenkins metrics (e.g., build times, queue length)
  • Set up alerts for failing jobs or build issues
  • Centralized monitoring across multiple Jenkins instances
  • Customizable dashboards to get a deep understanding of Jenkins performance

🧠 Key Components for Jenkins Monitoring Setup

  1. Jenkins Server: The CI/CD tool generating the metrics.
  2. Prometheus: Collects metrics from Jenkins.
  3. Grafana: Visualizes Jenkins metrics in custom dashboards.

🔧 Step-by-Step Setup

1️⃣ Install Jenkins Prometheus Plugin

  • Go to Jenkins Dashboard → Manage Jenkins → Manage Plugins.
  • Under the Available tab, search for Prometheus plugin.
  • Install the Prometheus Metrics Plugin.

This plugin exposes Jenkins metrics in the Prometheus format on a given HTTP endpoint.

2️⃣ Expose Jenkins Metrics Endpoint

Once the plugin is installed, Jenkins will expose metrics on the following endpoint:

http://<jenkins-server>:8080/metrics

You can access Jenkins server metrics directly from this endpoint.

  • Go to Jenkins → Manage Jenkins → Configure System → Prometheus Metrics.
  • Make sure the endpoint is accessible, and configure security (e.g., username/password or API tokens) if needed.

3️⃣ Install Prometheus

Install Prometheus on your monitoring server.

3.1 Install Prometheus (Linux):

wget https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.linux-amd64.tar.gz
tar -xvzf prometheus-2.43.0.linux-amd64.tar.gz
cd prometheus-2.43.0.linux-amd64
./prometheus --config.file=prometheus.yml

3.2 Configure Prometheus to Scrape Jenkins Metrics

Edit the prometheus.yml configuration file to add Jenkins as a target for scraping metrics.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'jenkins'
    static_configs:
      - targets: ['<jenkins-server>:8080']

Make sure to replace <jenkins-server> with the actual Jenkins server address.

3.3 Start Prometheus

./prometheus --config.file=prometheus.yml

Prometheus will now start scraping the Jenkins metrics.

You can view Prometheus at http://localhost:9090.

4️⃣ Install Grafana

Install Grafana on your monitoring server.

4.1 Install Grafana (Linux):

wget https://dl.grafana.com/oss/release/grafana-10.2.0.linux-amd64.tar.gz
tar -zxvf grafana-10.2.0.linux-amd64.tar.gz
cd grafana-10.2.0
./bin/grafana-server web

You can access Grafana at http://localhost:3000.

4.2 Configure Prometheus as Data Source in Grafana

  1. Open Grafana in the browser at http://localhost:3000.
  2. Login using default credentials (admin/admin).
  3. Go to Configuration → Data Sources → Add Data Source.
  4. Select Prometheus as the data source type.
  5. Set the URL to http://localhost:9090 (Prometheus server address).
  6. Click Save & Test to confirm the connection.

5️⃣ Import Pre-built Jenkins Dashboard into Grafana

Grafana provides pre-built Jenkins dashboards for monitoring Jenkins metrics.

5.1 Import Dashboard

  1. Go to + (Create) → Import in Grafana.

  2. Enter the Dashboard ID: 12239 (for Jenkins Monitoring).

  3. Select your Prometheus data source and click Import.

    This will automatically import a pre-configured dashboard for Jenkins.

6️⃣ Create Custom Dashboards

6.1 Sample Metrics Queries in Grafana

Here are a few example Prometheus queries for Jenkins that you can use to visualize data in Grafana:

  1. Job Duration Over Time

    • Query: avg_over_time(jenkins_job_duration_seconds_sum{job=~".*"}[1h])
    • This will show the average duration of Jenkins jobs over the past hour.

  2. Jenkins Job Success/Failure Rate

    • Query: rate(jenkins_job_builds{result="SUCCESS"}[5m]) / rate(jenkins_job_builds_total[5m])
    • Shows the success rate of Jenkins jobs over the past 5 minutes.

  3. Build Queue Length

    • Query: jenkins_build_queue_length
    • Displays the length of the Jenkins build queue.

  4. Active Executors

    • Query: jenkins_node_count{state="idle"}
    • Shows the number of idle executors on the Jenkins node.

  5. Jenkins Job Failures

    • Query: rate(jenkins_job_builds_total{result="FAILURE"}[5m])
    • Displays the failure rate of Jenkins jobs over the last 5 minutes.

6.2 Create Panels in Grafana

For each of the above queries:

  1. Create a new panel in Grafana by clicking the + sign on the dashboard.
  2. Set the query for the panel and choose an appropriate visualization (e.g., graph, gauge, bar chart, etc.).
  3. Add any additional thresholds or alerts based on your requirements.

7️⃣ Set up Alerts in Grafana

Grafana supports alerting on Jenkins metrics. For example, you can set up alerts if Jenkins build times are too high or if job failures exceed a certain threshold.

7.1 Create Alert for Failed Jobs

  1. Go to a panel that displays the failure rate of Jenkins jobs.
  2. Click the Alert tab in the panel settings.
  3. Set a condition, e.g.,:
    • If job failure rate exceeds 5% for 10 minutes.
  4. Set a notification channel (e.g., Slack, Email) to get alerts when the condition is met.

📚 Resources