Prometheus Metrics - NormB/sipnab GitHub Wiki
sipnab exposes Prometheus-compatible metrics, enabled with the api feature
flag (served on the REST API's port) or the metrics feature (a standalone
server on --metrics). Both emit the same families.
This is a different surface from the REST API, which returns what sipnab SAW โ dialogs, streams, messages. These are numbers ABOUT sipnab: what it captured, what it lost, and what it could not read. They also authenticate differently โ see Authentication.
It is also easy to confuse with HEP, the Homer Encapsulation Protocol, because
both are how sipnab fits into a monitoring estate โ and they answer opposite
questions. HEP is a TRANSPORT FOR SIP MESSAGES: sipnab receives them with
--hep-listen or forwards them with
--hep-send, and Homer stores the signaling itself. Prometheus carries no SIP
at all. A scrape tells you sipnab dropped 4,000 packets. It never tells you which
call. Wire both โ they are complements, not alternatives.
Prometheus-compatible metrics endpoint. Returns metrics in the Prometheus text exposition format (text/plain; version=0.0.4).
curl:
curl -s -H "Authorization: Bearer $SIPNAB_API_KEY" \
http://127.0.0.1:8080/metricsPython:
import requests
resp = requests.get(
"http://127.0.0.1:8080/metrics",
headers={"Authorization": "Bearer my-secret-token"},
)
print(resp.text) # Prometheus text formatGo:
req, _ := http.NewRequest("GET", "http://127.0.0.1:8080/metrics", nil)
req.Header.Set("Authorization", "Bearer my-secret-token")
resp, _ := http.DefaultClient.Do(req)
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
fmt.Println(string(body))JavaScript (Node.js):
const resp = await fetch("http://127.0.0.1:8080/metrics", {
headers: { Authorization: "Bearer my-secret-token" },
});
console.log(await resp.text()); // Prometheus text formatResponse (text/plain):
# HELP sipnab_dialogs_total Total dialogs by state
# TYPE sipnab_dialogs_total counter
sipnab_dialogs_total{state="completed"} 1180
sipnab_dialogs_total{state="failed"} 32
sipnab_dialogs_total{state="incall"} 23
# HELP sipnab_rtp_streams_total RTP streams by status
# TYPE sipnab_rtp_streams_total counter
sipnab_rtp_streams_total{status="established"} 43
sipnab_rtp_streams_total{status="orphaned"} 3
...
Metric names emitted by src/output/prometheus.rs:
| Metric | Type | Notes |
|---|---|---|
sipnab_dialogs_total{state} |
counter | Tracked dialogs grouped by DialogState (Trying, Ringing, InCall, Completed, Canceled, Failed, Redirected, Registered, Expired, Pending, Active, Terminated, Transferring). The --api server emits state values lowercased; the standalone --metrics server emits them as-cased โ pick the right form for your queries. |
sipnab_dialogs_active |
gauge | Dialogs in one of six active states: Trying, Ringing, InCall, Transferring, Pending, Active. Two of those six are SUBSCRIBE dialogs carrying no media, so this is not a count of calls โ a box serving presence traffic reports a non-zero value here with nothing on the phone. Graph it to see load on the dialog store, and alert on sipnab_calls_active instead. Only the standalone --metrics server counts it โ see Which server fills which series. |
sipnab_calls_active |
gauge | Calls that are up right now: dialogs in InCall, and nothing else. A dialog enters InCall on the 200 OK to its INVITE and leaves on the BYE, so this is the concurrent-call figure โ channels in use, and the number to compare against a carrier's simultaneous-call limit. By construction never greater than sipnab_dialogs_active, and the gap is calls still in setup plus subscriptions. Only the standalone --metrics server counts it, so an alert rule aimed at an --api scrape target reads a flat 0 and never fires. |
sipnab_messages_total{method} |
counter | SIP messages by method (INVITE, REGISTER, โฆ). |
sipnab_responses_total{code} |
counter | SIP responses in the tracked dialogs, grouped by class: 1xx, 2xx, 3xx, 4xx, 5xx, 6xx. Every class appears on every scrape, at 0 where the capture saw none, so a rule watching for the first 5xx reads zero instead of no-data. |
sipnab_rtp_streams_active |
gauge | The two servers count different things under this one name. The --api server counts streams a dialog claims, however long ago the last packet arrived; the standalone --metrics server counts streams whose last packet arrived within the previous 30 seconds, whatever their dialog association. A call whose media died five minutes ago is still counted by --api and is not counted by --metrics โ an alert threshold tuned on one scrape target does not carry over to the other. |
sipnab_rtp_streams_total{status} |
counter | RTP streams by status: orphaned when no dialog claims the stream, established when one does. The stream's dialog association decides the label on every scrape, with no age threshold in front of it, so a stream unclaimed for two seconds counts as orphaned exactly as one unclaimed for an hour does. --api only. The standalone --metrics server never populates the map, and an empty family drops out rather than reporting zero, so on --metrics the series does not exist at all โ a panel built on it stays permanently blank. |
sipnab_kill_responses_sent_total{mode} |
counter | Scanner-kill responses sent, by source mode: raw (source-spoofed via a raw socket) or ephemeral (sipnab's own port). Alert on unexpected ephemeral to catch a silent spoof fallback. |
sipnab_capture_packets_total |
counter | Packets the capture handed to the processing pipeline since the process started. Counted before parsing, so a frame sipnab cannot parse still counts โ it arrived. This series says nothing about whether sipnab understood any of it; a capture on a link type with no decoder climbs this counter exactly like a clean one. Pair it with sipnab_capture_undecoded_fraction before reading any zero elsewhere in the scrape as a finding. One process-wide total covering every input (-I files, live devices, HEP) and every worker of the parallel pipeline, identical on both servers. A line that stops climbing means packets stopped arriving. |
sipnab_reassembly_timeouts_total |
counter | IP fragments whose datagram never completed, plus TCP streams that went idle, dropped once older than the 30-second reassembly TTL. Capacity evictions stay out of this number: those say the entry cap is too small, not that a peer stopped sending. |
sipnab_capture_kernel_dropped_packets_total |
counter | Packets the kernel discarded because the capture ring buffer was full when they arrived (ps_drop). Non-zero means the analysis is incomplete: dialogs may be missing messages, and RTP loss figures overstate what was on the wire. The remedy is a larger -B/--buffer, a narrower BPF filter, or a smaller --snaplen. Always zero for a -I file replay, which has no ring. |
sipnab_capture_interface_dropped_packets_total |
counter | Packets the interface or its driver discarded before libpcap ever saw them (ps_ifdrop). Counted apart from the kernel drops because a larger -B cannot recover these โ the link is delivering faster than the host accepts, so the answer is at the NIC, the driver, or the mirror. Alerting on a sum of the two drop counters points the operator at the wrong remedy half the time. |
sipnab_capture_invalid_timestamps_total |
counter | Packets whose pcap timestamp did not parse, which stamped with the wall clock instead. No packet goes missing, so the counts elsewhere in the scrape stay right โ but every timing figure derived from the run is not: post-dial delay, RFC 3550 jitter, MOS and call duration all read from a substituted clock. |
sipnab_capture_undecodable_frames_total{reason} |
counter | Frames that reached the parser and produced no packet at all, grouped by why. The reason label carries the number, because the number is the whole deliverable: unsupported_link_type_0 says the file is DLT_NULL and editcap -T ether converts it, where a bare "unsupported link type" names no format an operator can act on. Labels are unsupported_link_type_<dlt>, not_ip_ethertype_0x<hhhh>, no_transport_ip_protocol_<n>, truncated_frame, decode_error, the _unrecorded variants of the two numbered EtherType/protocol labels (the decoder did not hand the number out), and reason_not_retained for frames beyond the tally's slot cap โ that last one exists so sum() over the family always equals the true total. Emitted by both the --api /metrics route and the standalone --metrics server. Absent entirely on a capture that decoded cleanly, so alert on the fraction below, not on this. |
sipnab_capture_snapped_frames_total |
counter | Frames the capture's own --snaplen cut short (caplen < origlen). Neither loss nor a decode failure: the frames arrived, most of them decoded, and what is missing is payload โ which is exactly what a signaling-only capture sets out to discard. It matters because the --snaplen warnings fire once per run and cannot say how MUCH of a capture came in truncated: a run that decoded every packet and snapped 94% of them is not a clean capture, and no other series says so. Raise --snaplen if you need RTP payload, audio export, or a faithful -O re-emit. |
sipnab_nat_unanswered_requests |
gauge | STUN and TURN transactions that went out and never came back โ the signal behind a one-way-audio complaint. The only capture-quality series about the network rather than about the capture: these frames arrived perfectly, and the reply to them did not. A gauge and not a counter because a late answer removes one, which a monotonic counter could never record. An endpoint that cannot learn its reflexive address advertises its private address in SDP, and the far end then sends media to an address the internet cannot route, while signaling looks healthy. Silence rather than a refusal points at something in the path discarding the packets โ on school, campus and corporate networks most often a security appliance (web filter, secure web gateway, firewall or IPS) dropping UDP it does not recognize. A refusal counts as answered: the server was reachable and said no, which is a different fault. |
sipnab_nat_lapsed_turn_allocations |
gauge | TURN allocations still carrying traffic past the lifetime the server last granted them, with no Refresh seen in between. The second capture-quality series about the NETWORK rather than about the capture, and the only condition sipnab reports that has no other symptom anywhere: a relay tears an allocation down the moment its lifetime lapses, the relayed media stops with it mid-call, and no SIP message says why โ the signaling shows a healthy call that went quiet. A gauge and not a counter because a Refresh seen later moves the expiry and unsays it, which a monotonic counter could never record. The wording is "no Refresh SEEN" rather than "none sent", because a capture that started late or lost a packet cannot tell those apart. A deliberate release (a Refresh with LIFETIME 0) is never counted โ the client asked for the teardown. |
sipnab_nat_lapsed_turn_allocation_streams |
gauge | Media streams observed crossing a TURN allocation that had already lapsed. The scale beside sipnab_nat_lapsed_turn_allocations, and the reason that series is worth paging on: an allocation that lapsed with nothing on it cost nobody a call, while one carrying four streams cut off four conversations mid-sentence โ and without relayed media attributable to the allocation that carried it, the two render identically. Counted by distinct SSRC seen inside the ChannelData frames on the allocation's channels, so RTCP does not inflate it. A gauge for the same reason the series above is: a Refresh seen later moves the expiry and takes the streams back out of this number. |
sipnab_nat_ice_role_conflicts |
gauge | Candidate pairs where both ICE agents claimed the same role, or where one answered 487 Role Conflict (RFC 8445 section 7.3.1.1). The third capture-quality series about the NETWORK rather than about the capture. ICE resolves this itself โ the losing agent switches role and repeats every check it had already sent โ so a single conflict is not always fatal, which is exactly why it belongs on a dashboard rather than only in an alert: a fleet where it happens constantly is misconfigured whether or not any individual call survived it. Where no pair between the two was ever nominated, the conflict is a candidate cause of media that never started. The usual source is two endpoints configured with the same role, or a B2BUA relaying one side's role attribute to the other. A gauge because a later nomination on the same pair unsays the severity of it. |
sipnab_capture_undecoded_fraction |
gauge | Share of captured frames sipnab could not decode, 0โ1, emitted on every scrape including clean ones. This is the series that separates "this capture holds no SIP" from "sipnab could not read this capture" โ both of which otherwise show sipnab_messages_total at zero and look identical. At 1 nothing in the rest of the scrape describes traffic sipnab read. A non-zero but small value is normal: ARP and other non-IP background is undecodable by definition on any Ethernet link, which is exactly why this is a proportion and not a flag. |
sipnab_capture_quality_degraded |
gauge |
1 when any of the three loss counters above exceeds zero, 0 otherwise. Undecodable frames are deliberately not folded in: ARP makes them non-zero on nearly every capture, so a flag including them would always be 1 and carry no information โ use sipnab_capture_undecoded_fraction with a threshold instead. The one series to put on a dashboard or an alert rule to know whether the rest of the scrape describes the whole capture. 0 means nothing surfaced as wrong, not that the capture provably saw every packet: loss upstream of the capture point โ an oversubscribed SPAN port, a tap mirroring one direction, a filter that excluded the traffic โ is invisible to all three counters. |
sipnab_capture_queue_depth_packets |
gauge | Packets currently queued between the capture reader and the processing thread (standalone --metrics server). |
sipnab_capture_backpressure_blocks_total |
counter | Times the capture reader blocked on a full queue (standalone --metrics server). |
sipnab_diagnosis_total{type} |
counter | Tracked dialogs whose media diagnosis raises each finding. A dialog with two findings counts under both. Seven label values, in three groups an alert rule has to keep apart. one_way_audio, nat_mismatch and no_media appear on every scrape, at 0 where nothing raises them, so a rule watching for the first one reads zero rather than no-data. private_media_address โ the SDP c= line offered an address the far end cannot route back to โ and private_media_address_confirmed โ the subset where STUN evidence also shows nothing rewrote that address โ appear only once a dialog raises them, so both series stay absent until the first one does. The ratio of the second to the first is how much of the fleet's warning the capture confirms. dead_air and clipping are the third group: they are amplitude measurements on the decoded audio, so they tick only on a run started with --retain-audio and stay absent otherwise. Absent means the run kept no samples, never that the calls were fine โ an alert rule that treats a missing dead_air series as a healthy fleet is reading a capture setting as a finding. Both servers run the diagnosis during the scrape, so scrape cost grows with the number of tracked dialogs (capped by -l/--limit). |
sipnab_security_alerts_total{type} |
counter | Security alerts by the detector that fired: scanner, fraud, digest, reg_flood. Only types that have fired appear, so the family is absent before the first alert. A rule name can come from packet data, so sipnab caps the family at 128 distinct names and folds everything past the cap into type="other" โ the total stays right even where the breakdown cannot. |
sipnab_mcp_tool_calls_total{tool,outcome} |
counter | MCP tool calls, by the tool asked for and how the call ended: ok, tool_error (the tool answered and said the answer is an error), or refused (the rate limit, the concurrency cap, a scope check, unknown arguments, or a name no tool answers to). The name comes from the CLIENT, so a name no tool answers to still counts โ probing is what this series is for โ and sipnab caps the distinct names at 256 (MAX_TOOLS), well above the tool surface, folding everything past the cap into tool="other" so a peer looping over invented names cannot grow the series set. Absent until the first tool call, like sipnab_security_alerts_total. sipnab counts at the same point it writes the mcp_audit log line, so the two always agree. |
sipnab_mcp_tool_duration_seconds{tool} |
histogram | How long each MCP tool takes to answer, in seconds, from the same instant the audit line's elapsed_ms measures. Fixed boundaries at 1/5/10/25/50/100/250/500ms and 1/2.5/5/10s โ unlike the four capture histograms these have no operator threshold to derive from, because they measure sipnab answering rather than the calls it observed. Emits _bucket{tool,le}, _count{tool}, _sum{tool}. |
sipnab_mcp_tool_response_bytes_total{tool} |
counter | Text content bytes each MCP tool has returned. The payload an agent reads, not the framed wire size: the JSON-RPC envelope and its escaping are outside this figure. Divide by sipnab_mcp_tool_duration_seconds_count{tool} for the average answer size, which is the number to look at when a client's context window is the constraint โ see --mcp-max-rows and --mcp-max-body-bytes. A refused call returns no content, so it moves the call counter and leaves this one where it was. |
sipnab_pdd_seconds |
histogram | Post-dial delay distribution. Boundaries come from [diagnosis] post_dial_delay_secs rather than a fixed list: a resolution ladder at 0.5/1/2/3/5s, then the threshold itself and twice it (11s and 22s at the shipped setting), so a query here reproduces the finding sipnab raises. Emits sipnab_pdd_seconds_bucket{le}, _count, _sum. --api only, as with the three quality histograms below it โ see Which server fills which series. |
sipnab_mos |
histogram | RTP MOS distribution. Half-steps up the 1-5 scale, plus [quality] mos_warn and mos_bad (4 and 3 at the shipped settings). --api only.
|
sipnab_jitter_ms |
histogram | RTP jitter distribution. A ladder at 5/10/20ms, plus [quality] jitter_warn_ms and jitter_bad_ms and multiples above them (30/50/100/200ms at the shipped settings). --api only.
|
sipnab_loss_percent |
histogram | RTP packet-loss distribution. A ladder at 0.1/0.5%, plus [quality] loss_warn_pct and loss_bad_pct and multiples above them (1/2/5/10/20% at the shipped settings). --api only.
|
Two shapes of counter share that table, and an alert rule has to know which one it reads. sipnab_capture_packets_total, sipnab_reassembly_timeouts_total, sipnab_kill_responses_sent_total, sipnab_capture_backpressure_blocks_total, sipnab_capture_undecodable_frames_total, sipnab_capture_snapped_frames_total, the two MCP counters (sipnab_mcp_tool_calls_total, sipnab_mcp_tool_response_bytes_total) and the three capture-quality counters (sipnab_capture_kernel_dropped_packets_total, sipnab_capture_interface_dropped_packets_total, sipnab_capture_invalid_timestamps_total) count events since the process started and only ever climb, so rate() and increase() over them mean what they say. The rest โ dialogs, messages, responses, streams, diagnosis findings, and the four NAT gauges โ describe what sipnab tracks right now, and every one of them falls as well as rises: dialogs and streams age out of their stores, a late STUN answer clears an unanswered request, a TURN Refresh unsays a lapsed allocation. Alert on the current value or on a ratio there, never on increase().
sipnab_security_alerts_total{type} reads differently from the rest, and the difference matters to an alert rule. AlertEngine::fire records each alert under its rule name, so the family carries only the types that have actually fired and stays absent from the scrape entirely until the first one does. An absent series therefore means "no alert of that type has fired since this process started", not "the metric is unavailable". firing_an_alert_moves_the_metric in tests/metrics_alert_wiring_test.rs holds the recording call to that behavior.
sipnab publishes these numbers two ways, and the two do not carry the same
series. The api feature serves /metrics on the REST API's port. The
metrics feature runs a standalone server on --metrics. A scalar gauge or
counter always prints, at 0 where the server behind it never fills the value,
so a panel aimed at the wrong target renders a flat line rather than an error.
Both targets publish the same series. Scrape either one. A rule tuned
against --api holds against --metrics, and a dashboard needs only one of
them.
That is new as of 0.5.158. The two doors each assembled the scrape themselves, and the six differences that produced are worth naming, because a reader who tuned a rule against the old behavior needs to know what moved:
| Series | What changed |
|---|---|
sipnab_dialogs_total{state} |
The standalone server published Completed, the REST door completed. Both now publish lowercase, which is what the dashboards in contrib/ query โ that panel read empty against --metrics, which looks exactly like an idle switch. |
sipnab_rtp_streams_active |
Two populations under one name: streams a dialog claims, or streams whose last packet arrived inside 30 seconds. Both now count streams a dialog claims, matching the established label on the counter beside it. |
sipnab_rtp_streams_total{status} |
The standalone server never filled it, and an empty family drops out rather than reading zero, so a panel built on it stayed blank. Both fill it now. |
sipnab_dialogs_active, sipnab_calls_active
|
The REST door published a flat 0. Both read the store's own accessors now. |
sipnab_pdd_seconds, sipnab_mos, sipnab_jitter_ms, sipnab_loss_percent
|
The standalone server published empty buckets. Both fill them now. |
sipnab_capture_queue_depth_packets, sipnab_capture_backpressure_blocks_total
|
The REST door published a flat 0 โ "the queue is clear" on a box whose queue was full. Both read the capture meter now. |
both_scrape_doors_publish_identical_exposition compares the formatted text
the two produce for one capture, so a future difference fails the build rather
than reaching a dashboard.
# prometheus.yml
scrape_configs:
- job_name: sipnab
bearer_token: your-api-key
static_configs:
- targets: ['127.0.0.1:8080']
scrape_interval: 15sThe metrics endpoint is lightweight and suitable for 5โ15 second scrape intervals. A sample Grafana dashboard JSON ships in the repo at contrib/grafana/sipnab-dashboard.json.