Add Prometheus and Grafana - NormB/sipnab GitHub Wiki

Prometheus collects numbers over time:

• how many calls are up,
• how many failed,
• how busy each process is.

It reads them from each program's metrics endpoint every few seconds. It keeps the history. Grafana draws that history as dashboards and alerts on it.

This guide:

• stands up both beside an OpenSIPS proxy,
• has OpenSIPS publish its statistics,
• shows a test call in the numbers.

With Kamailio instead, one section replaces the OpenSIPS steps. This guide does not use sipnab. When you have this working, Add sipnab's metrics to Prometheus adds sipnab's own.

The parts, and what each one does:

Part	Role
Prometheus	Scrapes each metrics endpoint every 5 seconds and stores the series, on port 9090.
Grafana	Dashboards over what Prometheus stored, on port 3000.
OpenSIPS	Your SIP proxy. Its prometheus module publishes its statistics over its httpd module, on 127.0.0.1:8888.

Everything below runs on one machine. The last section says what changes when they are apart.

Tested on

Every block on this page ran as written, in order, on 2026-09-28. It ran on clean x86_64 virtual machines with 2 cores and 3 GB of memory:

• Debian 13 (kernel 6.12.63)
• Ubuntu 24.04.5 (kernel 6.8.0)

On each, it ran:

• With OpenSIPS built from source on a machine with nothing installed.
• With the OpenSIPS 4.0 packages.
• With Kamailio, and beside OpenSIPS on one machine.
• On different machines, with Prometheus on its own machine.

On Debian 13, causing each fault under When something does not work produced what it describes. The commands pin the components to the versions below.

Software	Version or commit
Docker Engine / Compose	from the Docker apt repository
Prometheus	prom/prometheus:v3.15.0
Grafana	grafana/grafana:13.2.2
OpenSIPS, built here	f46ef9337b, master, 4.1.0-dev
OpenSIPS, from packages	4.0.2, installed as the OpenSIPS guide installs it
Kamailio	6.1.4, installed as the Kamailio guide installs it
SIPp (for the test call)	the distribution's sip-tester

The examples use 192.0.2.10 as the machine's address. Replace it with yours everywhere it appears.

Before you start: your SIP server

Find your case, and follow the steps it names:

• No SIP server yet. Follow every step.
  • The Install OpenSIPS's metrics modules step builds OpenSIPS for you.
  • You may rather run the OpenSIPS 4.0 packages. Then install them with the Install OpenSIPS step of the OpenSIPS guide. After that, take the packages' route in the Install OpenSIPS's metrics modules step.
• OpenSIPS already runs, from the packages or built from source.
  • The Install OpenSIPS's metrics modules step adds the two modules the metrics need.
  • The Configure OpenSIPS to publish its statistics step finds your configuration file and replaces it.
  • On a machine whose script you want to keep, add the lines that step marks to your own script instead.
• Kamailio already runs, or you want Kamailio. Follow the Install Docker step. Then follow With Kamailio, which takes you through the rest.
• Both OpenSIPS and Kamailio on one machine. Set up OpenSIPS as above. Then see With Kamailio, which covers both.

Install Docker

Prometheus and Grafana run as containers. Install Docker Engine and the Compose plugin from the Docker repository:

# Run all of these, in order.
sudo apt-get update
sudo apt-get install -y ca-certificates curl git
sudo install -m 0755 -d /etc/apt/keyrings
. /etc/os-release
sudo curl -fsSL "https://download.docker.com/linux/$ID/gpg" -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/$ID $VERSION_CODENAME stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker "$USER"

Log out and back in, so that your account's new docker group takes effect. Then check:

docker compose version

Install OpenSIPS's metrics modules

OpenSIPS publishes its statistics with its prometheus module, over HTTP served by its httpd module.

With the OpenSIPS packages, the two modules come in two more packages:

sudo apt-get install -y opensips-http-modules opensips-prometheus-module

Or build OpenSIPS with them. Skip this if you installed the packages.

• The prometheus module is part of OpenSIPS's default build.
• The httpd module it publishes through is not. It needs libmicrohttpd. Install that and build httpd too.
• You may have already built OpenSIPS from source by one of these guides. Then the block reuses that tree, /usr/local/src/voice/opensips, and rebuilds it with httpd.
• With a tree of your own, change /usr/local/src/voice in the block to the directory that holds it.
• OpenSIPS master also builds with compiler optimizations turned off.
  • That is right for OpenSIPS's own developers.
  • It is wrong for a proxy carrying calls.
  • So turn them back on:

# Run all of these, in order.
sudo apt-get install -y --no-install-recommends build-essential bison flex uuid-dev pkg-config libncurses-dev libssl-dev libmicrohttpd-dev
sudo mkdir -p /usr/local/src/voice && sudo chown "$USER": /usr/local/src/voice
cd /usr/local/src/voice
[ -d opensips ] || git clone https://github.com/OpenSIPS/opensips.git
cd opensips
git checkout f46ef9337b
make Makefile.conf
sed -i 's/^DEFS+= -DCC_O0/#DEFS+= -DCC_O0/' Makefile.conf
make -j2 all include_modules="httpd"
sudo make install include_modules="httpd"
/usr/local/sbin/opensips -V | head -2

Leave DBG_MALLOC as it is. With both it and CC_O0 switched off, this commit of master does not compile:

• net/tcp_conn_defs.h calls get_ticks() without including the header that declares it.
• Only DBG_MALLOC's headers happen to supply it.

Configure OpenSIPS to publish its statistics

This configuration is a minimal proxy with the metrics endpoint added. The metrics part is the block of loadmodule and modparam lines marked below.

Write the configuration

The packages and the source build keep their configuration and their modules in different places. The first two lines find them:

• C is your configuration file.
• M is the directory your install loads modules from. The script's mpath names it.

# Run all of these, in order.
for f in /etc/opensips/opensips.cfg /usr/local/etc/opensips/opensips.cfg; do sudo test -f "$f" && C=$f && break; done
for d in /usr/lib/*/opensips/modules /usr/local/lib64/opensips/modules; do [ -f "$d/tm.so" ] && M=$d && break; done
echo "configuration: $C   modules: $M"
sudo tee "$C" >/dev/null <<'EOF'
# OpenSIPS as a SIP proxy that publishes its statistics for Prometheus.
log_level=3
stderror_enabled=no
syslog_enabled=yes
syslog_facility=LOG_LOCAL0
udp_workers=2
open_files_limit=4096

socket=udp:192.0.2.10:5060   # the address your phones and carriers reach

mpath="MODULES/"

loadmodule "proto_udp.so"   # built into the core, but still loaded by name
loadmodule "signaling.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "sipmsgops.so"
loadmodule "dialog.so"

loadmodule "mi_fifo.so"
modparam("mi_fifo", "fifo_name", "/run/opensips/opensips_fifo")

# --- metrics start here ---
loadmodule "httpd.so"
modparam("httpd", "ip", "127.0.0.1")
modparam("httpd", "port", 8888)
loadmodule "prometheus.so"
modparam("prometheus", "statistics", "core: dialog: tm: shmem: load:")
# --- metrics end here ---

route {
	if (!mf_process_maxfwd_header(10)) {
		send_reply(483, "Too Many Hops");
		exit;
	}

	if (has_totag()) {
		if (is_method("ACK") && t_check_trans()) {
			t_relay();
			exit;
		}
		if (!loose_route()) {
			send_reply(404, "Not here");
			exit;
		}
		t_relay();
		exit;
	}

	if (is_method("CANCEL")) {
		if (t_check_trans())
			t_relay();
		exit;
	}
	t_check_trans();

	if (!is_method("INVITE")) {
		send_reply(405, "Method Not Allowed");
		exit;
	}

	record_route();
	create_dialog();

	# Where the call goes. Here, a test callee on this machine; in your
	# stack, lookup("location"), dispatcher or a carrier.
	$du = "sip:127.0.0.1:5070";
	t_relay();
}
EOF
sudo sed -i "s|^mpath=\"MODULES/\"|mpath=\"$M/\"|" "$C"
sudo grep '^mpath=' "$C"

What the metrics lines do:

• httpd serves HTTP on 127.0.0.1:8888, the loopback address. So only this machine can read the statistics.
• prometheus publishes them at /metrics on that server.
  • It publishes nothing until statistics names what to publish.
  • Here, that is the core, dialog, transaction, memory and load groups.
  • Each statistic becomes a series named opensips_ and the statistic's name, such as opensips_processed_dialogs.
• create_dialog() makes OpenSIPS track each call. That is what the dialog statistics count.

If you built OpenSIPS while installing its metrics modules: give it a user and a unit

The packages come with an opensips user and a systemd unit. Skip to Check the configuration and start OpenSIPS in either case:

• you installed them,
• your source build already runs as a service.

A fresh build has neither. Run OpenSIPS as its own user, under systemd:

# Run all of these, in order.
sudo useradd --system --home-dir /run/opensips --shell /usr/sbin/nologin opensips
sudo chown root:opensips /usr/local/etc/opensips
sudo chmod 750 /usr/local/etc/opensips
sudo tee /etc/systemd/system/opensips.service >/dev/null <<'EOF'
[Unit]
Description=OpenSIPS SIP server
After=network.target

[Service]
Type=forking
User=opensips
Group=opensips
RuntimeDirectory=opensips
RuntimeDirectoryMode=775
PIDFile=/run/opensips/opensips.pid
ExecStart=/usr/local/sbin/opensips -P /run/opensips/opensips.pid -f /usr/local/etc/opensips/opensips.cfg -m 64 -M 8
Restart=always
TimeoutStopSec=30s
LimitNOFILE=262144

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload

Check the configuration and start OpenSIPS

# Run all of these, in order.
for f in /etc/opensips/opensips.cfg /usr/local/etc/opensips/opensips.cfg; do sudo test -f "$f" && C=$f && break; done
sudo chown root:opensips "$C"
sudo chmod 640 "$C"
sudo opensips -C -f "$C"
sudo systemctl enable opensips
sudo systemctl restart opensips
systemctl is-active opensips
curl -s 127.0.0.1:8888/metrics | awk '/^opensips_/ && n++ < 3'

The last line prints the first three series OpenSIPS publishes.

Start Prometheus and Grafana

Both run with the host's own network. That way, Prometheus can reach OpenSIPS's endpoint on the loopback address. Choose a Grafana admin password. Put it in .env, where Compose reads it:

# Run all of these, in order.
sudo mkdir -p /opt/monitoring && sudo chown "$USER": /opt/monitoring
cd /opt/monitoring
echo "GF_PASS=$(openssl rand -hex 12)" > .env
chmod 600 .env
mkdir -p grafana/datasources
cat > prometheus.yml <<'EOF'
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: opensips
    static_configs:
      - targets: ["127.0.0.1:8888"]
EOF
cat > grafana/datasources/prometheus.yml <<'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://127.0.0.1:9090
    isDefault: true
EOF
cat > docker-compose.yml <<'EOF'
services:
  prometheus:
    image: prom/prometheus:v3.15.0
    network_mode: host
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.2.2
    network_mode: host
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GF_PASS}
    volumes:
      - ./grafana/datasources:/etc/grafana/provisioning/datasources:ro
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-data:
EOF
docker compose up -d
until curl -fs -o /dev/null localhost:9090/-/ready; do sleep 2; done
until curl -fs -o /dev/null localhost:3000/api/health; do sleep 2; done
echo ready

--storage.tsdb.retention.time=15d is how long Prometheus keeps the history.

Check that Prometheus reaches OpenSIPS:

curl -s localhost:9090/api/v1/targets | python3 -c 'import sys,json; [print(t["labels"]["job"], t["health"]) for t in json.load(sys.stdin)["data"]["activeTargets"]]'

It prints opensips up. Open Grafana at http://192.0.2.10:3000. Log in as admin with the password in /opt/monitoring/.env. The Prometheus data source is already there.

Place a test call and watch the counter

SIPp plays both ends: a callee on this machine, and a caller that dials through OpenSIPS.

• SIPp's built-in caller ignores the Record-Route header OpenSIPS adds.
• So its BYE would miss the proxy and draw 404 Not here.
• The two sed lines make it honor the route set, the way a real phone does:

# Run all of these, in order.
sudo apt-get install -y sip-tester
mkdir -p ~/sipp && cd ~/sipp
sipp -sd uac > uac_rr.xml
sed -i 's|<recv response="200" rtd="true">|<recv response="200" rtd="true" rrs="true">|' uac_rr.xml
sed -i -E 's#^( *)(ACK|BYE) sip:\[service\]@\[remote_ip\]:\[remote_port\] SIP/2.0#\1\2 [next_url] SIP/2.0\n\1[routes]#' uac_rr.xml
sipp -sn uas -i 127.0.0.1 -p 5070 -m 1 -bg
sipp -sf uac_rr.xml 192.0.2.10:5060 -i 192.0.2.10 -p 5080 -m 1 -d 1000 -timeout 20s

Prometheus scrapes every 5 seconds. Wait for one scrape. Then ask it how many dialogs OpenSIPS has processed:

# Run all of these, in order.
sleep 6
curl -s localhost:9090/api/v1/query --data-urlencode 'query=opensips_processed_dialogs' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["result"][0]["value"][1])'

It prints 1, the test call.

Operate it

Check health and read the logs.

# Run all of these, in order.
cd /opt/monitoring && docker compose ps
docker compose logs --tail 20 prometheus
systemctl is-active opensips

Restart. Prometheus misses the scrapes while it is down, and shows a gap.

# Run all of these, in order.
cd /opt/monitoring && docker compose restart
sudo systemctl restart opensips

A restart of OpenSIPS starts its counters again from zero.

Uninstall. docker compose down -v removes the containers and their volumes. The volumes hold the stored history and Grafana's settings:

# Run all of these, in order.
cd /opt/monitoring && docker compose down -v
sudo systemctl disable --now opensips

With Kamailio

Kamailio publishes its statistics with its xhttp_prom module. It serves them over HTTP with its xhttp module. Both come with the kamailio package.

• Set Kamailio up as the Kamailio guide does, through its Configure Kamailio as a proxy step.
• Follow the Install Docker step here for Docker.
• In place of the Install OpenSIPS's metrics modules step and the Configure OpenSIPS to publish its statistics step, add the metrics endpoint to Kamailio's configuration.

The lines marked metrics are the ones to add to your own script:

# Run all of these, in order.
sudo tee /etc/kamailio/kamailio.cfg >/dev/null <<'EOF'
#!KAMAILIO
# Kamailio as a SIP proxy that publishes its statistics for Prometheus.
debug=2
log_stderror=no
log_facility=LOG_LOCAL0
children=2

listen=udp:192.0.2.10:5060   # the address your phones and carriers reach
listen=tcp:127.0.0.1:8888    # metrics: the endpoint, on the loopback address
tcp_accept_no_cl=yes         # metrics: HTTP requests carry no Content-Length

loadmodule "tm.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "siputils.so"
loadmodule "textops.so"
loadmodule "pv.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "ctl.so"

# metrics
loadmodule "xhttp.so"
loadmodule "xhttp_prom.so"
modparam("xhttp_prom", "xhttp_prom_stats", "all")

request_route {
	if (!mf_process_maxfwd_header("10")) {
		sl_send_reply("483", "Too Many Hops");
		exit;
	}

	if (has_totag()) {
		if (loose_route()) {
			t_relay();
			exit;
		}
		if (is_method("ACK") && t_check_trans()) {
			t_relay();
		}
		exit;
	}

	if (is_method("CANCEL")) {
		if (t_check_trans()) {
			t_relay();
		}
		exit;
	}
	t_check_trans();

	if (!is_method("INVITE")) {
		sl_send_reply("405", "Method Not Allowed");
		exit;
	}

	record_route();

	# Where the call goes. Here, a test callee on this machine; in your
	# stack, lookup("location"), dispatcher or a carrier.
	$du = "sip:127.0.0.1:5070";
	t_relay();
}

# metrics: answer HTTP requests for /metrics
event_route[xhttp:request] {
	if ($hu =~ "^/metrics") {
		prom_dispatch();
		exit;
	}
	xhttp_reply("404", "Not Found", "text/plain", "not found\n");
}
EOF
sudo kamailio -c -f /etc/kamailio/kamailio.cfg
sudo systemctl enable kamailio
sudo systemctl restart kamailio
systemctl is-active kamailio
curl -s 127.0.0.1:8888/metrics | awk '/^kamailio_/ && n++ < 3'

What the metrics lines do:

• listen=tcp:127.0.0.1:8888 gives Kamailio a TCP socket on the loopback address. xhttp answers the HTTP requests that arrive on it. Only this machine can read the statistics.
• xhttp_prom_stats "all" publishes every statistic Kamailio keeps. Each becomes a series named kamailio_, the group and the statistic's name, such as kamailio_core_rcv_requests_invite.
• event_route[xhttp:request] is where Kamailio handles each HTTP request. prom_dispatch() answers the ones for /metrics.

Then follow the Start Prometheus and Grafana step. Name the scrape job after Kamailio:

# Run all of these, in order.
cd /opt/monitoring
sed -i 's/job_name: opensips/job_name: kamailio/' prometheus.yml
docker compose restart prometheus
until curl -fs -o /dev/null localhost:9090/-/ready; do sleep 2; done

Place the test call from the Place a test call and watch the counter step. Then ask Prometheus how many INVITE requests Kamailio has received:

# Run all of these, in order.
sleep 6
curl -s localhost:9090/api/v1/query --data-urlencode 'query=kamailio_core_rcv_requests_invite{job="kamailio"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["result"][0]["value"][1])'

It prints 1, the test call. The query names the kamailio job for two reasons:

• For the few minutes before the rename, Prometheus scraped the same endpoint as opensips.
• A query still finds that series for up to five minutes after its last sample.

The Operate it step then works unchanged, with kamailio in place of opensips in the systemctl commands.

With OpenSIPS on the same machine, OpenSIPS already serves its metrics on 127.0.0.1:8888. Kamailio listens for SIP on 5062, as OpenSIPS and Kamailio on one machine sets it up. Give Kamailio's endpoint its own port too. In the configuration above:

• change 5060 to 5062 in the first listen line,
• change 8888 to 8889 in the second listen line and in the curl line at the end.

Then keep the opensips scrape job and add one for Kamailio, in place of renaming it:

# Run all of these, in order.
cd /opt/monitoring
cat >> prometheus.yml <<'EOF'
  - job_name: kamailio
    static_configs:
      - targets: ["127.0.0.1:8889"]
EOF
docker compose restart prometheus
until curl -fs -o /dev/null localhost:9090/-/ready; do sleep 2; done

Prometheus then shows both targets up. Each proxy's series appear under its own name: opensips_ and kamailio_.

Put the parts on different machines

Prometheus on its own machine. The proxy's metrics endpoint has no authentication. So keep it on an internal address that only Prometheus can reach. Here, that is the proxy machine's own 192.0.2.10. The first line finds the proxy's configuration: the OpenSIPS packages', a source build's, or Kamailio's:

# Run all of these, in order.
for f in /etc/opensips/opensips.cfg /usr/local/etc/opensips/opensips.cfg /etc/kamailio/kamailio.cfg; do sudo test -f "$f" && C=$f && break; done
sudo sed -i -e 's|modparam("httpd", "ip", "127.0.0.1")|modparam("httpd", "ip", "192.0.2.10")|' \
  -e 's|^listen=tcp:127.0.0.1:8888|listen=tcp:192.0.2.10:8888|' "$C"
sudo grep -nE 'httpd", "ip"|^listen=tcp' "$C"
case "$C" in */kamailio/*) sudo systemctl restart kamailio;; *) sudo systemctl restart opensips;; esac
curl -s 192.0.2.10:8888/metrics | awk '/^(opensips|kamailio)_/ && n++ < 3'

Allow port 8888 from the Prometheus machine only. On the Prometheus machine, here 192.0.2.20:

• follow the Install Docker step and the Start Prometheus and Grafana step there,
• then scrape the proxy's address in place of the loopback one:

# Run all of these, in order.
cd /opt/monitoring
sed -i 's|targets: \["127.0.0.1:8888"\]|targets: ["192.0.2.10:8888"]|' prometheus.yml
grep -n 'targets' prometheus.yml
docker compose restart prometheus
until curl -fs -o /dev/null localhost:9090/-/ready; do sleep 2; done
for i in $(seq 1 15); do curl -s localhost:9090/api/v1/targets | grep -qE '"health":"(up|down)"' && break; sleep 2; done
curl -s localhost:9090/api/v1/targets | python3 -c 'import sys,json; [print(t["labels"]["job"], t["health"]) for t in json.load(sys.stdin)["data"]["activeTargets"]]'

Right after the restart, Prometheus lists no target. Then it lists one that reads unknown until its first scrape. So the loop waits up to 30 seconds for an up or a down. It prints the proxy's job, up. With Kamailio, name the job as With Kamailio does.

Several SIP servers. Add each one to the targets list, or give each its own job. Prometheus labels every series with the target it came from. OpenSIPS and Kamailio on one machine is that case. With Kamailio sets it up.

When something does not work

• curl 127.0.0.1:8888/metrics prints no opensips_ lines. The statistics parameter is missing. The prometheus module publishes nothing until it names what to publish.
• OpenSIPS does not start: failed to load module 'httpd.so'. The module is not installed.
  • With the packages, install opensips-http-modules, as in the Install OpenSIPS's metrics modules step.
  • With a source build, build with include_modules="httpd". That needs libmicrohttpd-dev.
• The target shows down. Check curl 127.0.0.1:8888/metrics on the OpenSIPS machine. Then check that the target address in prometheus.yml matches.