Prometheus - kamialie/knowledge_corner GitHub Wiki

Contents

Architecture

  • scheduler (fetches metrics)
  • time-series database (TSBD)
  • HTTP API
  • web UI
  • alerting system

Prometheus uses a pull model - applications expose metrics and Prometheus fetches them.

All metrics are stored in time series database - everything is recorded with a timestamp.

Client libraries.

Node Exporter

Runs a process on host, which provides the HTTP endpoint that returns all the current metric values (/metrics).

Useful metrics:

  • node_disk_io_time_seconds_total
  • node_cpu_seconds_total
  • node_filesystem_avail_bytes
  • node_uname_info

Server

Important startup logs:

  • TSDB started (time series database).
  • Loading configuration file=path/to/file.yaml
  • Server is ready to receive web requests.

By default Prometheus stores its database in ./data local directory. To set different path use --storage.tsdb.path option.

Default port is 9090. Exposes its own metrics at /metrics path.

Start up options:

  • --config.file=path/to/config.yaml
  • --web.enable-lifecycle - gives http endpoint to call to force Prometheus to reload config file; reload configuration with the following endpoint 
    $ curl -X POST http://host:9090/-/reload

UI

Status -> Configuration - active configuration.

  • /service_discovery - fetched services and labels

Installation

Configuration

$ kubectl create ns monitor
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install prometheus prometheus-community/kube-prometheus-stack -n monitor

For clean deletion also delete CRDs:

$ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
$ kubectl delete crd alertmanagers.monitoring.coreos.com
$ kubectl delete crd podmonitors.monitoring.coreos.com
$ kubectl delete crd probes.monitoring.coreos.com
$ kubectl delete crd prometheuses.monitoring.coreos.com
$ kubectl delete crd prometheusrules.monitoring.coreos.com
$ kubectl delete crd servicemonitors.monitoring.coreos.com
$ kubectl delete crd thanosrulers.monitoring.coreos.com

Configuration

Docs

Common fields:

  • scrape_interval (default 1m)
  • scrape_timeout (default 10s) - time to wait a scrape to complete successfully
  • metrics_path (default /metrics)
  • scheme (default http)
  • job_name is added as a lable to fetched metrics

Simple example with static targets.

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'linux'
    static_configs:
	  - targets: ['host_one:9100']

  - job_name: 'windows'
    scrape_timeout: 10s
	metrics_path: /metrics
	scheme: http
    static_configs:
	  - targets: ['host_two:9100']

Check configuration file:

$ promtool check config configuration.yaml

Targets

Information about targets is metadata. Prometheus considers all target metadata as not sensetive, so no secrets should be saved there.

Jobs can override global settings, but scrape_interval generally isn't changed, otherwise metrics will have different ages.

Targets can be specified statically inside Prometheus configuration (would require server restart to apply changes) or through service discovery.

Static configuration;

static_configs:
  - targets: ['host_two:9100']

Service discovery methods:

File based

Targets are specified in a separate file (yaml or json). Prometheus looks for changes and automatically updates targets based on file contents.

file_sd_configs:
  - files:
	- 'file.json'

[
	{
		"targets": ["host_three:9182:]
	},
	{
		"targets": ["host_four:8080:]
	}
]

file_sd_configs:
  - files:
	- 'file.yaml'

- labels:
    job: job-name
  targets:
    - host:port

Kubernetes SD

Kubernetes service discovery roles:

  • node - uses kubelet port by default; node address is one the following (in the order of precedence:
    • internal IP
    • external IP
    • legacy host IP
    • hostname cAdvisor metrics are not scraped by default - available at /metrics/cadvisor (need a separate job with explicit path)
  • service - fits closed box monitoring (closer to user) to answer questions like "if service is alive"; the address is service DNS name and port
  • pod
  • endpoints
  • ingress - similar to service

Service discovery

Service discovery (SD) is automatica detection of services and hosts. Service discover option. Aim of SD is to dump all possible targets. Filtering can be used to discover desired targets (also for performance issues).

DNS

SRV (service locator record)

dsn_sd_configs:
  - names:
    - metrics.web
    type: SRV

# name         TTL  port  target
metrics.web    300  9132  box1.web
metrics.web    300  9132  box2.web
metrics.web    300  9132  box3.web

A record (doesn't specify port, thus all targets must listen on the same port)

dsn_sd_configs:
  - names:
    - metrics.web
    type: A
	port: 8080

# name       address
metric1.web  172.17.143.1
metric2.web  172.17.143.2

Metrics

Best practice is to use standard base units, like bytes and seconds. More usable values can be projected via queries.

Metric types

Counter always increases. Typical metrics are number of http requests or cpu time. Together with timestamps and Prometheus functions, like offset, trends can be formed.

Gauge is a snapshot of changing value. F.e. currently serving http requests or size of memory allocated.

Summary is used for recording an avarage size. F.e. time taken for a calculation. Expressed via count (number of events) and sum (total size).

calculation_seconds_count 3
calculation_seconds_sum 150

Histogram records data in buckets - each buckets contains metrics with the corresponding value. Example below splits calculations based on how much time it took. Histograms are cumulative - all data is contained inside bucket with less than 60 seconds. This also tells that there has been only 1 calculation over 20 seconds. Default bucket with +Inf label includes all values.

calculation_seconds_bucket{le="1"} 0
calculation_seconds_bucket{le="5"} 3
calculation_seconds_bucket{le="10"} 6
calculation_seconds_bucket{le="20"} 9
calculation_seconds_bucket{le="60"} 10

# Default total
# calculation_seconds_bucket{le="+Inf"}

Usefule metrics

  • up - value of 1 means Prometheus was able to reach it, 0 otherwise
  • scrape_duration_seconds
  • scrape_samples_scraped - number of metrics
  • prometheus_config_last_reload_success_timestamp_seconds - Unix timestamp when configuration was last reloaded

Labels

Label are used to track more granular metrics, f.e. successful and failed requests and request paths. Prometheus query functions can include or exclude labels.

http_request_total{code="200", path="/"} 800
http_request_total{code="500", path="/p1"} 1200

Prometheus also automatically adds some labels (which can be configured), like hostname, region, OS, etc. Job label helps to distinguish same metrics of different services. Instance label identifies multiple particular host of a service is running on multiple targets.

Static labels

Static labels are set in static_configs or file discovery section of Prometheus configuration.

- job_name: 'name'
  static_configs:
    - targets: ['host_one:9100']
	  labels:
	    os: linux
		runtime: vm

Target labels

Target relabelling (when target is being scraped) can be used to modify or filter based labels based on their values. Most useful with service discovery, since it provides temporary labels (they start with __) - aren't delivered to Prometheus. However, temporary labels can be copied to permanent ones.

Each time runtime metric is met and its value is container, it is changed to docker.

- job_name: 'name'
  file_sd_configs:
    - files:
	    - 'linux.json'
  relabel_configs:
    - source_labels: [runtime]
	  regex: container
	  target_label: runtime
	  replacement: docker

Metric labels

Metric relabelling is similar to target relabelling, but it doesn't have access to discovery labels, as it happens later in the process. Useful for applying consistency - if same metrics have different labels, they can be modified for easy querying. Can be used to edit or drop labels or even drop entire metrics.

Go metrics are removed, while Windows volume label is modified and copied as device.

- job_name: 'name'
  file_sd_configs:
    - files:
	    - 'linux.json'
  metric_relabel_configs:
    - source_labels: [__name__]
	  regex: 'go_.*'
	  action: drop
    - source_labels: [__name__, volume]
	  regex: 'windows_logical_disk_free_bytes;(.*)'
	  replacement: '${1}'
	  target_label: device

Query

Prometheus uses its own PromQL language for queries.

General form.

operator(function(metric_name{selector}[range]))

Queries are called expressions and the simplest form is just the name of metric - returns most resent entires (equivalent to SELECT TOP in SQL) with sample values and labels.

metric_name

Expression can be filtered using a selector, which works on labels (similar to WHERE in SQL). Specify one or multple labels and values. Regular expression can be used as a value - specify ~ after equal sign and pass a regex.

metric_name
  {label_one="i1"}

metric_name
  {label_two="batch, label_one=~"i.*"}

Specify range of results with brackets and time parameter. Ranges and selectors can also be combined.

metric_name[3m]

metric_name
  {label_one="i1"}
  [3m]

Operators

Operators include simple arithmetic (f.e. adding or dividing) and aggregation operators (f.e. min, max, sum, average).

sum(metric_name)

Exclude labels in aggregation using without clause.

sum without(label_one)
(metric_name)

Function

Functions manipulate raw data. F.e. delta computes the difference between first and last value in a given range. Returns a result for each series with a unique label combination. Use avg to compute aggregation across all labels.

delta(metric_name[1h])

avg(delta(metric_name[1h]))

API

host:port/api/v1/query?query=metric_name

References