Rtpengine Relays - NormB/sipnab GitHub Wiki

You installed sipnab on a media relay, ran a capture, and got a screen full of orphaned RTP streams. Every packet is there and none of it says which call it belongs to. This page fixes that.

The short version. A standalone relay carries no SIP, so a capture there has media and nothing to name it with. rtpengine knows the name — its ng control protocol carries the Call-ID and the ports it allocated — and it can already mirror that control plane to a Homer collector. sipnab reads that mirror and ties the media to the call. A call that was already up when the capture opened left no control message for the mirror to carry, so --rtpengine-control asks the relay for that one directly.

Both runs below read the same forty RTP packets on the same four sockets. The only difference is whether the six control-plane packets are in the file.

Without the control plane, everything orphans:

Orphaned Streams:
SSRC         Source                   Destination              Pkts     Duration
--------------------------------------------------------------------------------
0x0a0a0a0a   192.0.2.60:40001         192.0.2.40:38156         10       0s
0x0b0b0b0b   192.0.2.60:40002         192.0.2.40:38664         10       0s
0x0a0a0a0a   192.0.2.40:38664         192.0.2.60:40002         10       0s
0x0b0b0b0b   192.0.2.40:38156         192.0.2.60:40001         10       0s

With it, the same media resolves to one call — and gains a codec, because the control plane carries the SDP too:

RTP Streams:
SSRC         PT   Codec    Clock  Source                Destination           Pkts
----------------------------------------------------------------------------------
0x0a0a0a0a   0    PCMU     8000   192.0.2.60:40001      192.0.2.40:38156      10
0x0b0b0b0b   0    PCMU     8000   192.0.2.60:40002      192.0.2.40:38664      10
0x0a0a0a0a   0    PCMU     8000   192.0.2.40:38664      192.0.2.60:40002      10
0x0b0b0b0b   0    PCMU     8000   192.0.2.40:38156      192.0.2.60:40001      10

Calls named by a media relay (no SIP for them in this capture):
Call-ID                                                      Streams
---------------------------------------------------------------------
km-670bd208@sipnab                                           4

Reproduce both from the committed fixtures:

Without the control plane, every stream orphans:

sipnab -N -I tests/fixtures/rtpengine-media-only.pcap --report

With it, the same media resolves to one call:

sipnab -N -I tests/fixtures/rtpengine-ng-hep.pcap --report

Four streams and not two: the relay holds its own socket per leg, plus the far party's on the other side of each. All four belong to one call.

Read down the first column and stop at the first row you can satisfy.

Every row here covers control messages that cross the wire while sipnab runs. None of them reaches a call the relay set up before the capture opened. For that, add --rtpengine-control, which asks the relay and runs alongside whichever row you picked.

rtpengine takes exactly one Homer destination. --homer is a single value, not a repeatable option, so "send to the real collector and to sipnab" does not exist. Anything that points rtpengine at sipnab takes it away from the collector it feeds. That is why sipnab reads the traffic off the wire by default: a diagnostic tool has no business in a production data path, and if sipnab stops, nothing else notices.

A proxy that load-balances across a pool of relays puts each call on exactly one of them. Run one sipnab per host: the proxy's sees the SIP, each relay's sees the media its own host relays.

                        sipnab@proxy
                        (SIP; no media)
                              |
        +---------------------+---------------------+
        |                     |                     |
  sipnab@relay-a        sipnab@relay-b        sipnab@relay-c
  (media it relays)     (media it relays)     (media it relays)

Each relay host runs sipnab against its own local relay:

# On relay-a, relay-b, relay-c — identical, and each asks only its own relay.
# The API binds to the host's own address, not loopback: the whole point of the
# procedure below is that ANOTHER machine queries this one. A non-loopback bind
# is refused without authentication, which is why --api-key is not optional
# here.
sudo sipnab -d eth0 --rtpengine-control 127.0.0.1:22222 \
  --api 0.0.0.0:8080 --api-key "$SIPNAB_API_KEY"

The queries below need the address, the key and a Call-ID. Set them once, on whichever machine you are running curl from:

$H interpolates $KEY, so set the key first — an $H built before $KEY exists carries an Authorization header with no token in it, and every query below then returns 401.

KEY="$SIPNAB_API_KEY"

H="-H 'Authorization: Bearer $KEY'"

Then the call you are chasing:

CALL_ID="[email protected]"

--rtpengine-control takes one address, and in this topology that is correct rather than a limitation: a relay can only answer for calls it is carrying, so a sipnab asking a relay on another host would get RelayDoesNotHoldIt for everything.

The naive procedure is a broadcast: ask all ten relays, discard nine misses. You do not have to. The proxy's own SDP names the relay, because the c= address the proxy negotiated IS the rtpengine it steered the call to.

Step 1 — ask the proxy which relay has it.

curl -fsS "http://proxy:8080/v1/dialogs/$CALL_ID" $H \
  | jq -r '.sdp_timeline[0] | "\(.media_addr):\(.media_port)"'
# 192.0.2.40:38664

Step 2 — ask that one relay, and no others.

curl -fsS "http://192.0.2.40:8080/v1/dialogs/$CALL_ID" $H | jq '.streams'

A relay that never carried the call answers 404, not an empty success. That distinction is load-bearing: "I do not have this call" and "this call had no media" are different findings, and a relay that answered 200 with an empty list would report the second when it meant the first.

Every node stamps its answers with its own capture_identity, which carries the host name:

curl -fsS "http://192.0.2.40:8080/v1/stats" $H | jq '.capture_identity'
# { "node": "relay-a", "instance": "1f4a…", "dialog_generation": 412, … }

Collecting replies from several hosts, that field is what keeps them attributable to the host that gave them. Without it, two JSON documents from two relays are indistinguishable once they are side by side in a terminal.

sipnab does not aggregate across nodes. There is no cluster mode, no central index, and no node that answers for another. Each instance answers for what it captured, and the correlation above is something you (or an agent holding several endpoints) perform. That is a deliberate limit, not a missing feature: an aggregator is infrastructure, and positioning rules it out.

tests/multi_node_relay_fanout_test.rs pins the three properties this procedure rests on — the proxy holds signaling and no media, exactly one relay claims a call, and the proxy's SDP names which one.

Nothing to configure. Capture on the relay host and sipnab decodes the control plane it sees:

sudo sipnab -d eth0

Confirm rtpengine is mirroring the control plane and not only RTCP stats:

grep -E '^homer' /etc/rtpengine/rtpengine.conf

You want homer-enable-ng = true in that output. Without it rtpengine sends RTCP statistics only, which carry no Call-ID and no SDP.

Add these to the [rtpengine] section of /etc/rtpengine/rtpengine.conf:

homer = 192.0.2.60:9060
homer-protocol = udp
homer-id = 2001
homer-enable-ng = true

Then restart it:

sudo systemctl restart rtpengine

homer-id becomes the per-node key when you correlate several nodes, so give each relay a distinct one.

Place one call through the relay, then check that sipnab saw control traffic:

sudo sipnab -N -d eth0 --report

A working setup names the call under Calls named by a media relay. If it does not, work down this list:

--hep-parse unwraps a HEP datagram and hands the payload inside it to the parser. That is what you want when a collector feeds you SIP over HEP, and until 0.5.128 it quietly switched this feature off.

sipnab spots mirrored ng two ways: by the metadata a HEP listener recorded before stripping the wrapper, or by parsing the wrapper off a sniffed datagram that still has one. Unwrapping used to discard the wrapper without recording what it said, so neither route had anything left to match on. The control plane reached the capture and went straight in the bin, no counter moved, and every relay stream stayed orphaned.

Nothing reported it, and that is the part worth remembering: the build carried the feature, the filter carried the ports, and the datagrams sat plainly on the wire. Every setting was correct and the answer was still empty.

From 0.5.128 the unwrap carries the wrapper's capture protocol and correlation id forward, so the two features coexist and the flag is safe to leave on. On an older build, turn it off for a relay capture.

rtpengine CONNECTS its Homer socket, so a destination that answers with ICMP port-unreachable makes it give up and log this:

ERR: [core] Connection error from Homer at 192.0.2.1:9060: Connection refused

After that it sends nothing, which looks exactly like the feature not working. Pointing --homer at an address nobody listens on is therefore not a way to "just put it on the wire" — something has to absorb it.

In the deployment this targets, that something is your real Homer collector, which is already there. If you are testing without one, run any UDP sink at the address. The harness ships one for exactly this reason — see harness/hep-sink.

Everything above reads the control plane off the wire, so it names the calls whose offer crossed that wire while sipnab was running. Incident response rarely starts that way. You attach to a relay because something is wrong now, and the calls you care about went up minutes or hours ago — their offer happened in the past, no capture can recover it, and their media lands in the orphan list.

--rtpengine-control closes that gap by asking the relay. Give it the address rtpengine's ng control socket listens on — its listen-ng value. Same host first:

sudo sipnab -N -d eth0 --rtpengine-control 127.0.0.1:22222 --report

A relay on its own host answers the same question over the network:

sudo sipnab -N -d eth0 --rtpengine-control 192.0.2.40:22222 --report

-N earns its place in both. sipnab writes these summaries as log lines on stderr, and the TUI silences logs to keep the alternate screen intact.

sipnab asks at two moments and no others: once at startup, before the capture opens, and again when a stream turns up that nothing else explains. There is no interval flag because there is no timer. A tool that talks to a production relay on a schedule is a service, and this is a diagnostic.

The startup question happens before the first packet, so its answer comes first. Against rtpengine 12.5.1:

rtpengine at 127.0.0.1:22222: 2 call(s) enumerated, complete; queried 2 of them, 8 relay port(s) now attributable

Read it in three parts. Enumerated, complete means the relay listed everything it held rather than capping the answer. Queried 2 of them is one query per call, which is the only command that says which relay-side port belongs to which call. 8 relay ports now attributable is the index sipnab keeps for the rest of the run: media on any of those eight sockets takes its Call-ID from memory and costs the relay nothing further.

Streams the relay accounts for stop orphaning. A call whose own signaling never appears in the capture — the whole reason for asking — lands under Calls named by a media relay, the same heading the mirrored control plane fills.

Every stream says who named its dialog, on every door:

GET /v1/streams and GET /v1/streams/{id} carry it as dialog_assertion, the MCP rtp_stats tool carries the same key with the same two spellings, and the TUI's stream detail prints (via media-relay) beside the Call-ID.

The distinction changes what the address means, which is why it is not folded away. A relay's answer is not an endpoint: it names the box the media passed through, not where either party sits. An operator tracing a one-way-audio fault to 192.0.2.40:38664 needs to know whether that is the far end or the box in the middle, and those lead to opposite next steps.

It is also only as trustworthy as the path it arrived on, and this page used to say it was "authoritative about the port". That was true of rtpengine and false of the assertion: rtpengine cannot be wrong about which socket it opened, but sipnab is not always hearing from rtpengine. Read the next section before you act on a media-relay address.

An absent key means nobody recorded who asserted the binding. It does not mean signaled — that is a claim, and keeping the two apart is the entire reason the field exists.

Two paths deliver it, and they are worth different amounts.

Nothing authenticates a sniffed assertion. It carries a Call-ID copied verbatim out of the HEP correlation-id chunk and a media address copied out of the SDP, so a forged datagram can name a call of the sender's choosing and bind it to a socket of the sender's choosing.

sipnab therefore believes a sniffed mirror only on UDP port 9060, the HEP port. That narrows the input without authenticating anybody, and this page says so plainly so nobody reads it as more. It stops every datagram on the segment being a candidate, and it stops nothing sent to 9060. sipnab logs one refusal, naming the port it saw.

--hep-allow does not reach this path. It guards the --hep-listen socket, and there is no socket here.

If any of that matters for your estate, take the delivered path and authenticate it:

sudo sipnab -N -d eth0 --hep-listen 0.0.0.0:9060 \
    --hep-auth-file /etc/sipnab/hep.key --hep-auth-mode hmac

and point rtpengine's homer = at sipnab. It costs you the Homer collector — rtpengine takes exactly one destination — which is the trade the table at the top of this page is about.

If your collector sits on a port other than 9060, sipnab does not read the sniffed copy at all. That is deliberate. The alternative is believing every port.

dialog_assertion is a different question from dialog_origin, which says which capture source delivered the assertion. A relay can name a stream over a HEP mirror or answer sipnab directly, and a party's SDP can arrive off the NIC. The two keys answer one half of that each.

The second summary arrives when the capture ends:

rtpengine at 127.0.0.1:22222: 2 unexplained stream(s) offered, 0 attributed, 4 control transaction(s) spent of a ceiling of 66

The capture path handed two streams to the asking thread, neither of them gained a Call-ID, and the run spent four control transactions of the 66 it may spend. A count of zero with no reason line beside it is the honest answer rather than a failure: the relay answered and holds neither port, so that media belongs to something else on the host.

Reading a file, sipnab refuses to ask, and says why:

sipnab -N -I capture.pcap --rtpengine-control 127.0.0.1:22222

--rtpengine-control 127.0.0.1:22222 asks a live relay which calls are up
right now, but this run is reading a capture FILE — offline analysis never
transmits. The calls in a file ended in the past; the relay's answer would
describe whatever is up TODAY, which is other people's traffic. Passive
decoding of any relay control plane already in the capture still runs;
capture live with -d <device> to ask.

The analysis still runs and still exits 0. What the run loses is the ask, and it says so before the capture opens rather than looking like a run that asked and learned nothing.

The two questions sipnab can put to a relay are list and query, and that is a property of the type reaching the relay rather than a convention. ng also carries offer, answer, delete and start recording, each of which changes a production relay. None of them has a representation on this path, so no call site reaches one by accident.

Three bounds hold the asking down, and none of them grows with how much traffic the capture carries:

• sipnab asks about each relay-side socket at most once for the whole run, so a stream that stays unexplained does not re-ask on every packet.
• A per-run ceiling caps control transactions at 66 — one list plus a query per call at rtpengine's own list limit of 32, twice over. Enough for a full startup snapshot and one comparable refresh, and no more.
• The queue handing sockets from the capture path to the asking thread stops at 4096 entries — two per stream, so 2048 unexplained streams — and counts what it turned away rather than letting a slow relay grow it without end.

When a bound bites, sipnab counts it and says so. The asking runs on its own thread, so the packet path offers a socket and moves on rather than waiting out a round trip to a relay that may be down.

"Nothing came back" has five meanings, and sipnab keeps them apart rather than collapsing them into one shrug:

• The relay did not answer, or refused the question. sipnab knows nothing about the port.
• The relay named calls sipnab could not read. The port may belong to one of those.
• The relay capped its own enumeration. The list came back partial, so the port may belong to a call the relay never named. sipnab does not accept that quietly: a capped first answer earns one wider ask, for up to 1024 Call-IDs, and the summary says whether that wider ask landed or failed.
• The run spent its transaction ceiling. sipnab never asked about the port at all.
• The relay does not hold the port. This one, and only this one, says something about the relay: the stream is not its media.

The first four are gaps in what sipnab learned. Reporting any of them as the fifth would turn a run that never reached the relay into a run that asked and heard the stream belongs to nobody.

Attribution answers whose media a stream is. A relay also keeps counters about itself — the packets it relayed, the sessions it holds, its own loss and jitter — and --relay-stats asks the relay named by --rtpengine-control for them and prints them:

sudo sipnab -N -d eth0 --rtpengine-control 127.0.0.1:22222 --relay-stats

Every figure names the relay as its source. The header marks it relay_reported: a claim the relay makes about itself, not a measurement sipnab took from the wire. Reading it as "sipnab saw this" is the one mistake this whole view exists to stop.

To watch a counter move rather than reading one snapshot, poll on a timer:

sudo sipnab -N -d eth0 --rtpengine-control 127.0.0.1:22222 --relay-stats-interval 5

Each reading marks itself polled and names the interval, so it reads as a timer's output rather than a one-shot answer.

The tier is the whole point here, not a footnote. One call carries three different "loss" figures, and each one describes a different socket at a different point on the path:

• The relay's own count (relay_reported) — what the relay counted on its own sockets since its last restart. --relay-compare puts it beside sipnab's own count for one call, labels each, and prints a one-word verdict with a note.
• What sipnab saw on the wire (sipnab_measured) — gaps in the RTP sequence numbers that reached the capture point, so it reflects only what the capture caught.
• What the far end claims (endpoint_reported) — the remote endpoint's own assertion over RTCP, which nobody can check.

sipnab never subtracts one tier from another. The relay's count minus sipnab's is not "packets sipnab missed", because the two count different sockets over different windows, so their difference describes nothing. Name the tier when you quote the number — "the relay reports 40 lost, our capture measured 3 gaps" — and the disagreement stops being a contradiction and becomes three facts about three points on the path.

The cookbook works each question end to end: the relay-statistics recipes cover whether a relay drops packets (recipe 61), whether it holds sessions nobody released (recipe 62), whether its view of one call matches yours (recipe 63), and this three-tier loss comparison (recipe 64).

--relay-stats needs a live source. Asking a relay transmits, and a capture file's addresses belong to third parties and describe the past, so on -I <file> sipnab refuses to ask and says why rather than reaching an address the file named:

sipnab -N -I capture.pcap --relay-stats --rtpengine-control 127.0.0.1:22222

--relay-stats will not ask a relay on a run that reads a file: asking
transmits, and a file's addresses are historical and belong to third
parties. Ask from a live capture (-d <device>).

The rest of the analysis still runs and still exits 0. What the run loses is the ask.

Everything above is a person at a terminal. query_relay gives an AI agent the same reach, and decode_ng lets it read a control message it finds in the capture.

query_relay transmits, so it stays off until you say otherwise. It needs three things, and refuses with a message naming whichever one is missing:

# Off by default: this server has the relay address but no opt-in, so
# query_relay refuses and every other tool works as usual.
sudo sipnab --mcp -N -d eth0 --rtpengine-control 127.0.0.1:22222

Add the opt-in to enable it:

# Now an agent can ask the relay directly. The destination is THIS address --
# there is no tool argument that can change it.
sudo sipnab --mcp -N -d eth0 --rtpengine-control 127.0.0.1:22222 \
  --mcp-allow-relay-query

The third requirement is a live source, and it is structural rather than a policy check. A run reading a capture file cannot obtain a transmit permit, so --mcp-allow-relay-query on an offline run leaves the tool refusing:

# Refuses: -I is a file, so no transmit permit exists to send with.
sipnab --mcp -N -I capture.pcap --rtpengine-control 127.0.0.1:22222 \
  --mcp-allow-relay-query

That matters more than it looks. An analyst opening a capture from another organization must not be able to make sipnab talk to the addresses inside it -- a host that served as a relay during the capture may be somebody's laptop now.

decode_ng needs none of this. It reads a control message already in the capture and reports which path delivered it, so an agent can tell "the relay told us" from "something claiming to be the relay told us". Those are the same bytes and very different claims.

Nothing, and a measurement says so rather than an assumption.

rtpengine forwards either in userspace or, with table = N and the xt_RTPENGINE kernel module, inside netfilter. If kernel-forwarded media were invisible to a capture, attributing it would be pointless — so this run checked it against rtpengine 12.5.1, with the module confirmed active and accounting the packets itself:

/proc/rtpengine/0/list   local 192.0.2.40:34232  250 packets   [kernel-forwarded]
                         output → 192.0.2.60:40001             250 packets
capture on the relay     ingress 500/500         egress 500/500

Both directions stay fully visible in both modes. The reason is structural: a capture's receive tap runs before netfilter, and the module re-injects through the normal transmit path, which passes the transmit tap. You do not have to know or care which mode a relay runs in.

Stated here rather than discovered later.

• It does not tie the two halves of a B2BUA call together. A B2BUA gives one call two Call-IDs. The control plane names each leg's media and ties neither to the other.
• It does not read DTLS-SRTP media. rtpengine terminates DTLS and emits no key log, so on a WebRTC-facing relay the payload stays unreadable.
• It does not attribute recording or forking streams. Those commands create media that belongs to the call without being one of its two legs, and counting one as an ordinary leg would turn a two-party call into a three-stream one. sipnab counts them and says so instead: --report prints Media-creating relay commands seen with the count, beside the relay-named calls, and their media joins the orphaned streams where the capture holds it.
• It does not name a call that started before the capture did, from the wire alone. A control message that already happened never reaches the wire, so no capture can recover it. Give --rtpengine-control and sipnab asks the relay instead, which closes the gap on a live run. Without the flag, or on an -I <file> run where sipnab refuses to ask, their media stays in the orphan list.

None of this section describes what sipnab does today. What this page documents above is one hop naming its own media, proven against a recorded capture and against a live relay. This section states what that foundation exists FOR, so the next phase aims at a goal rather than at whatever comes next.

The goal is a call that crosses an SBC, a proxy, an rtpengine and a PBX, assembled end to end, so that an operator can ask a question in their own words and an agent answers with evidence from every hop the call touched.

Naming media on the relay is the hop that was missing, because it is the only hop that carries no SIP of its own. Once each hop can name a call, an agent can filter to a customer — by caller, callee, trunk, address, or time window — and pull the whole call rather than one machine's view of it.

These are the questions that arrive, and what answering each one takes.

The hard part is not measuring quality, it is locating it. A relay reporting 2% loss tells you nothing on its own — the question is whether the loss was already there when the media arrived, or appeared on the way out.

That needs the same call measured on both sides of each hop, which needs each hop to agree on the call's name. On the relay that agreement is what this page adds. The analysis then reads as a chain: clean into the SBC, clean into the relay, lossy out of the relay, and now the fault has an address rather than a symptom.

Watch for the answer being "neither hop": identical loss at every hop points at the access network, and a codec that changes between hops points at transcoding rather than at the network.

Same filter, different evidence. Here the media is usually absent rather than degraded, so the question moves to signaling — and to the gap between them.

The failure modes separate cleanly once you can see every hop at once:

• A final response from an upstream hop, which names the rejecter. The response class matters: a 4xx is the far end declining, a 5xx is a server failing, a 6xx is a global refusal, and they lead to different owners.
• No response at all, which is a timeout, and the hop that stopped answering is the one to look at.
• A call answered with no media, which is a negotiation failure rather than a routing one — the signaling succeeded and the SDP did not.
• A relay with no ports left, which looks like a random failure from the proxy's side and is obvious from the relay's.

The last two are exactly the cases a signaling-only view gets wrong, because from the proxy the call looks answered.

A drop has a clock attached, and the clock identifies the cause. A tear-down at a round number is a timer — a session timer, a NAT binding, a relay's own media timeout. A tear-down when media stopped is the media stopping. Reading the relay's teardown against the proxy's BYE says which came first, which is the whole question.

One-way audio is a direction problem, and a relay is where directions become visible: it holds both parties' sockets, so a leg with media arriving and nothing leaving is plainly asymmetric. The value of naming the call is being able to say WHICH call is asymmetric on a box carrying hundreds.

The comparison case. Two calls of the same shape down two trunks, assembled the same way, differing in one hop's behavior. Hard to do by hand across machines and straightforward once every hop names its calls the same way.

• Encapsulations — HEP and the other wrappers sipnab reads
• Capture SIP over TLS — when encryption hides the signaling
• MCP tools — the agent-facing surface over these results