prometheus service discovery relabeling - ghdrako/doc_snipets GitHub Wiki

• Prometheus Service Discovery code base: https://github.com/prometheus/prometheus/tree/v2.46.0/discovery
• Prometheus HTTP service discovery documentation: https://prometheus.io/docs/prometheus/latest/http_sd/
• Pull Request to Prometheus implementing the http_sd provider: https://github.com/prometheus/prometheus/pull/8839

Each service discovery provider can be configured within a scrape job within a section named in the pattern of _sd_configs – for example, kubernetes_sd_configs or azure_sd_configs.

• https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs

metrics in prometheus are identified by

• name
• ley/value pair (labels)

haproxy_http_responses_total{
  env="acc",
  job="haproxy"
  ....
}

Note: Prometheus creates the label __name__ for every metric with the name of the metric as the value. Metric name is also label.

{
  __name__="haproxy_http_responses_total"
  env="acc",
  job="haproxy"
  ....
}

Metrics are scraped as plaintext lines from every target in the format:

metric_name{label_name=label_value,...} metric_value [timestamp]

metric_name always exists, there can be 0 or more label_name/label_value pairs, metric_value is always a number, and there can be an optional timestamp.

• https://relabeler.promlabs.com/ - relabeling visalizer
• https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/
• https://www.youtube.com/watch?v=QufG6rjbsOM

After relabeling, metrics must be unique

Source of labels

Section of relabeling in config

# A list of scrape configurations.
scrape_configs:
    - job_name: "some scrape job"
    ...
 
    # List of target relabel configurations.
    relabel_configs:
    [ - <relabel_config> ... ]

    # List of metric relabel configurations.
    metric_relabel_configs:
    [ - <relabel_config> ... ]

# Settings related to the remote write.
remote_write:
    url: https://remote-write-endpoint.com/api/v1/push 
    ...

    # List of remote write relabel configurations.
    write_relabel_configs:
    [ - <relabel_config> ... ]

#
alerting:
  alert_relabeling_config:

Processing labels

Metalables start with __ belong to 2 categories. First how target will be scraped and manipulate them to change behavior of scrape target. The second group is informetion and depend on service discovery mehanism. Metalabels will be lost or replace to target normal labels to use farther.

Any target labels that start with two underscores get dropped after the relabel step. Any remaining labels are added to every metric scraped from the target.

Relabeling rules can be found in multiple parts of a Prometheus config file.

• First off, the relabel_configs key can be found as part of a scrape job definition. These relabeling steps are applied before the scrape occurs and only have access to labels added by Prometheus’ Service Discovery. They allow us to filter the targets returned by our SD mechanism, as well as manipulate the labels it sets.

• Once the targets have been defined, the metric_relabel_configs steps are applied after the scrape and allow us to select which series we would like to ingest into Prometheus’ storage.

• Finally, the write_relabel_configs block applies relabeling rules to the data just before it’s sent to a remote endpoint. This can be used to filter metrics with high cardinality or route metrics to specific remote_write targets.

relabel_configs is an array of configuration settings applied to every target discovered by Prometheus before the target's metrics are scraped.

The relabel_configs and metric_relabel_configs blocks is where most of the magic happens in tailoring Prometheus to your monitoring needs. It offers the flexibility to filter, modify, or otherwise manipulate targets and labels before they are scraped.

A <relabel_config> consists of seven fields. These are:

• source_labels

• separator (default = ;)

• target_label - defines to which label the replacement should be written

• regex (default = (.*)) - match everything

• modulus

• replacement (default = $1)

• action (default = replace)

relabel_configs:
  - source_labels: [label_name,...]
    separator: ;
    regex: (.+)
    target_label: label_name
    replacement: $1
    action: replace

1. source_labels: These are the input labels to the transformation. The values of these labels are concatenated using the separator and matched against the regex.
2. separator: A string to join multiple source label values together. Default value is ;.
3. regex: The regular expression against which the source_labels are matched. Default value is (.*), which matches everything
4. arget_label: The label to be updated or created based on the replacement.
5. replacement: The value to use to replace the value of the target_label. $1 refers to the first matching group from the regex. Default value is $1.
6. action: The operation to be performed. There are a few actions. Common ones are replace (default), keep, drop, and labelmap. See the documentation for more options.

Not all of the fields are required, and most of them have a default setting. Fields that are not changed from a default value simply left out.

The two sections relabel_configs and metric_relabel_configs have the same syntax, so they are often explained at the same time. However they cover two different things.

• relabel_configs lets you manipulate targets. So you can drop specific targets and rearrange the targets labels before they are ingested into Prometheus.
• metric_relabel_configs lets you manipulate individual metrics and can let you drop, or metrics and metric labels before they are saved.

• keep/drop - allow us to filter out targets and metrics based on whether our label values match the provided regex.

my_custom_counter_total{server="webserver01",subsystem="kata"} 192  1644075074000
my_custom_counter_total{server="sqldatabase",subsystem="kata"} 14700  1644075074000

After concatenating the contents of the subsystem and server labels, we could drop the target which exposes webserver-01 by using the following block

- source_labels: [subsystem, server]
  separator: "@"
  regex: "kata@webserver"
  action: "drop"

keep specific metric names

- source_labels: [__name__]
  regex: “my_custom_counter_total|my_custom_counter_sum|my_custom_gauge”
  action: keep

- source_labels: [__meta_kubernetes_namespace]
  regex: “testing|staging”
  action: drop

- source_labels: [__meta_kubernetes_pod_container_name]
  action: keep
  regex: 'istio-proxy'

Prometheus bedzie skrapowal tylko pody z kontenerem istio-proxy

- source_labels: [__address__]
  action: replace
  regex: ([^:]+):.*
  replacement: $1:15020
  target_label: __address__

Prometheus always scrape pod only on port 15020

- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)

- action: keep
  source_labels: [__name__]
  regexp: '(api_|http_).*'

Filtrujemy metryki i zachowujemy tylko jesli maja w nazwie prefix api_ lub http_

• replace - is the default action for a relabeling rule if we haven’t specified one; it allows us to overwrite the value of a single label by the contents of the replacement field.

- action: replace
  source_labels: [__meta_kubernetes_pod_name,__meta_kubernetes_pod_container_port_number]
  separator: ":"
  target_label: address

The above snippet will concatenate the values stored in __meta_kubernetes_pod_name and __meta_kubernetes_pod_container_port_number. The extracted string would then be set written out to the target_label and might result in {address="podname:8080}.

In Prometheus, there are two types of relabeling: standard relabeling and metric relabeling. They both serve distinct purposes and occur at different times in the process of performing a scrape. Metric relabeling happens after a scrape occurs, but before the time series are stored in their chunks. This is when time series can be dropped or modified based on various criteria. We’ll cover this in more detail later when we discuss how to address issues of data cardinality within Prometheus. Standard relabeling occurs before a target is scraped and is where we perform additional configuration of our service discovery providers. It performs relabeling on discovered scrape targets to add additional labels and filter the targets we want to scrape.

Avaliable labels: source: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpoints

ServiceMonitor filters for a port named http-metrics in discovered endpoints. We can see this take place in relabeling by using the __meta_kubernetes_endpoint_port_name metadata label, which is only available during relabeling:

- source_labels: [__meta_kubernetes_endpoint_port_name]
regex: http-metrics
action: keep

By using relabel_config like this, we can explicitly keep endpoints with a port that matches http-metrics. Implicitly, this means that any discovered endpoints with a port name that doesn’t match will be dropped.

Relabeling helps add labels to discovered targets.

Label is overwritten during relabeling. We can see this occurring here:

- source_labels: [__meta_kubernetes_service_label_jobLabel]
regex: (.+)
target_label: job
replacement: ${1}
action: replace

During this stage of relabeling, Prometheus takes the value of the jobLabel label on the discovered service and places it in the job label on the Prometheus scrape target.

relabel_configs:
  - source_labels: [env]
    action: keep
    regex: production

block with default values

relabel_configs:
  - source_labels: [env]
    action: keep
    separator: ;
    regex: production
    target_label: label_name
    replacement: $1

relabel_configs:
  - source_labels: [path]
    target_label: __metrics_path__
    replacement: /$1/metrics

block with default values

relabel_configs:
  - source_labels: [path]
    action: replace
    regex: (.+)
    target_label: __metrics_path__
    replacement: /$1/metrics

So if our target had a label path=/mypath/scrape, the actual path Prometheus would scrape would be /mypath/scrape/metrics.

you have a label instance and you want to rename it to node

relabel_configs:
  - source_labels: [instance]
    target_label: node

block with default values

relabel_configs:
  - source_labels: [instance]
    action: replace
    regex: (.+)
    target_label: node
    replacement: $1