prometheus istio - ghdrako/doc_snipets GitHub Wiki

Istio's Default Prometheus Scraping:

  • By default, Istio injects a sidecar container with Envoy proxy into your application pods. This sidecar exposes various metrics on port 15090 by default (configurable).
  • Istio also automatically creates a ServiceMonitor resource for each service in your mesh. This resource instructs Istio's Prometheus server to scrape metrics from the sidecar containers' port 15090.

Controlling Prometheus Scraping:

Metrics merging

To collect both application and Istio metrics in a single Prometheus scrape, enable Prometheus scraping for your application pods and set meshConfig.enablePrometheusMerge=true during Istio installation using Helm:

helm install istio istiod --set meshConfig.enablePrometheusMerge=true

This approach simplifies monitoring by providing a combined view of metrics.

The metrics merging is more of a convenience feature that allows you to get the application metrics without setting up a dedicated Prometheus job (since the Prometheus job for scraping the 15020 “merged” port of the Envoy proxy is provided by the Istio installation).

When enabled, appropriate prometheus.io annotations will be added to all data plane pods to set up scraping. If these annotations already exist, they will be overwritten. With this option, the Envoy sidecar will merge Istio’s metrics with the application metrics. The merged metrics will be scraped from :15020/stats/prometheus.

This option exposes all the metrics in plain text. Not support TLS . If required, this feature can be disabled per workload by adding a prometheus.istio.io/merge-metrics: "false" annotation on a pod.

The enablePrometheusMerge will automatically add the annotations:

prometheus.io path: /stats/prometheus
prometheus.io port: "15020"
prometheus.io scrape: "true"
Port Protocol Description
15020 HTTP Merged Prometheus telemetry from Istio agent, Envoy, and application
15090 HTTP Envoy Prometheus telemetry

In istio-proxy

- name: ISTIO_META_POD_PORTS
  value: [  {"name":"http","containerPort":9538,"protocol":"TCP"}  ]
- name: ISTIO_PROMETHEUS_ANNOTATIONS
  value: '{"scrape":"true","path":"/metrics","port":"9538"}'

Customized scraping configurations

you would have two Prometheus jobs instead of one, one for the Istio Envoy metrics and one for the application metrics, and you can fully control how the metrics are scraped for each one independently.

If you prefer to scrape application metrics separately from Istio metrics, disable the automatic sidecar injection of Prometheus scraping annotations:

apiVersion: networking.istio.io/v1alpha3
kind: SidecarInjection
spec:
  webhookMode: (SET_WEBHOOK | DISABLE_WEBHOOK)  # Choose based on your setup
  injectedAnnotations:
    prometheus.io/scrape: "false"  # Disable automatic injection

Then, configure your own Prometheus scraping for your application pods.