Rtpengine Relays - NormB/sipnab GitHub Wiki

You installed sipnab on a media relay and ran a capture. You got a screen full of orphaned RTP streams. Every packet is there, and none of it says which call it belongs to. This page fixes that.

The short version.

• A standalone relay carries no SIP. So a capture there has media and nothing to name it with.
• rtpengine knows the name. Its ng control protocol carries the Call-ID and the ports it allocated.
• rtpengine can already mirror that control plane to a Homer collector.
• sipnab reads that mirror and ties the media to the call.

A call that was already up when the capture opened left no control message for the mirror to carry. So --rtpengine-control asks the relay for that one directly.

Both runs below read the same forty RTP packets on the same four sockets. The only difference is whether the six control-plane packets are in the file.

Without the control plane, everything orphans:

Orphaned Streams:
SSRC         Source                   Destination              Pkts     Duration
--------------------------------------------------------------------------------
0x0a0a0a0a   192.0.2.60:40001         192.0.2.40:38156         10       0s
0x0b0b0b0b   192.0.2.60:40002         192.0.2.40:38664         10       0s
0x0a0a0a0a   192.0.2.40:38664         192.0.2.60:40002         10       0s
0x0b0b0b0b   192.0.2.40:38156         192.0.2.60:40001         10       0s

With it, the same media resolves to one call. It also gains a codec, because the control plane carries the SDP too:

RTP Streams:
SSRC         PT   Codec    Clock  Source                Destination           Pkts
----------------------------------------------------------------------------------
0x0a0a0a0a   0    PCMU     8000   192.0.2.60:40001      192.0.2.40:38156      10
0x0b0b0b0b   0    PCMU     8000   192.0.2.60:40002      192.0.2.40:38664      10
0x0a0a0a0a   0    PCMU     8000   192.0.2.40:38664      192.0.2.60:40002      10
0x0b0b0b0b   0    PCMU     8000   192.0.2.40:38156      192.0.2.60:40001      10

Calls named by a media relay (no SIP for them in this capture):
Call-ID                                                      Streams
---------------------------------------------------------------------
km-670bd208@sipnab                                           4

Reproduce both from the committed fixtures:

Without the control plane, every stream orphans:

sipnab -N -I tests/fixtures/rtpengine-media-only.pcap --report

With it, the same media resolves to one call:

sipnab -N -I tests/fixtures/rtpengine-ng-hep.pcap --report

There are four streams and not two.

• The relay holds its own socket per leg.
• It also holds the far party's on the other side of each.
• All four belong to one call.

Read down the first column and stop at the first row you can satisfy.

Every row here covers control messages that cross the wire while sipnab runs. None of them reaches a call the relay set up before the capture opened. For that, add --rtpengine-control.

• It asks the relay.
• It runs alongside whichever row you picked.

rtpengine takes exactly one Homer destination.

• --homer is a single value, not a repeatable option. So "send to the real collector and to sipnab" does not exist.
• Anything that points rtpengine at sipnab takes it away from the collector it feeds.
• That is why sipnab reads the traffic off the wire by default:
  • A diagnostic tool has no business in a production data path.
  • If sipnab stops, nothing else notices.

A proxy that load-balances across a pool of relays puts each call on exactly one of them. Run one sipnab per host:

• The proxy's sipnab sees the SIP.
• Each relay's sipnab sees the media its own host relays.

                        sipnab@proxy
                        (SIP; no media)
                              |
        +---------------------+---------------------+
        |                     |                     |
  sipnab@relay-a        sipnab@relay-b        sipnab@relay-c
  (media it relays)     (media it relays)     (media it relays)

Each relay host runs sipnab against its own local relay:

# On relay-a, relay-b, relay-c — identical, and each asks only its own relay.
# The API binds to the host's own address, not loopback: the whole point of the
# procedure below is that ANOTHER machine queries this one. A non-loopback bind
# is refused without authentication, which is why --api-key is not optional
# here.
sudo sipnab -d eth0 --rtpengine-control 127.0.0.1:22222 \
  --api 0.0.0.0:8080 --api-key "$SIPNAB_API_KEY"

The queries below need the address, the key and a Call-ID. Set them once, on whichever machine you are running curl from.

$H interpolates $KEY. So set the key first.

• An $H built before $KEY exists carries an Authorization header with no token in it.
• Every query below then returns 401.

KEY="$SIPNAB_API_KEY"

H="-H 'Authorization: Bearer $KEY'"

Then the call you are chasing:

CALL_ID="[email protected]"

--rtpengine-control takes one address. In this topology that is correct rather than a limitation.

• A relay can only answer for calls it is carrying.
• So a sipnab asking a relay on another host would get RelayDoesNotHoldIt for everything.

The naive procedure is a broadcast: ask all ten relays, discard nine misses. You do not have to. The proxy's own SDP names the relay. The c= address the proxy negotiated IS the rtpengine it steered the call to.

First, ask the proxy which relay has it.

curl -fsS "http://proxy:8080/v1/dialogs/$CALL_ID" $H \
  | jq -r '.sdp_timeline[0] | "\(.media_addr):\(.media_port)"'
# 192.0.2.40:38664

Then ask that one relay, and no others.

curl -fsS "http://192.0.2.40:8080/v1/dialogs/$CALL_ID" $H | jq '.streams'

A relay that never carried the call answers 404, not an empty success. That distinction is load-bearing.

• "I do not have this call" and "this call had no media" are different findings.
• A relay that answered 200 with an empty list would report the second when it meant the first.

Every node stamps its answers with its own capture_identity. It carries the host name:

curl -fsS "http://192.0.2.40:8080/v1/stats" $H | jq '.capture_identity'
# { "node": "relay-a", "instance": "1f4a…", "dialog_generation": 412, … }

When you collect replies from several hosts, that field keeps them attributable to the host that gave them. Without it, two JSON documents from two relays are indistinguishable once they are side by side in a terminal.

sipnab does not aggregate across nodes. There is:

• no cluster mode,
• no central index,
• no node that answers for another.

Each instance answers for what it captured. The correlation above is something you (or an agent holding several endpoints) perform.

That is a deliberate limit, not a missing feature. An aggregator is infrastructure, and positioning rules it out.

tests/multi_node_relay_fanout_test.rs pins the three properties this procedure rests on:

• The proxy holds signaling and no media.
• Exactly one relay claims a call.
• The proxy's SDP names which one.

Nothing to configure. Capture on the relay host, and sipnab decodes the control plane it sees:

sudo sipnab -d eth0

Confirm rtpengine is mirroring the control plane and not only RTCP stats:

grep -E '^homer' /etc/rtpengine/rtpengine.conf

You want homer-enable-ng = true in that output. Without it, rtpengine sends RTCP statistics only. Those carry no Call-ID and no SDP.

Add these to the [rtpengine] section of /etc/rtpengine/rtpengine.conf:

homer = 192.0.2.60:9060
homer-protocol = udp
homer-id = 2001
homer-enable-ng = true

Then restart it:

sudo systemctl restart ngcp-rtpengine-daemon

That is the unit name when you build rtpengine's packages from its own source tree, as Add rtpengine to your voice stack does. Debian's and Ubuntu's own rtpengine-daemon package names the unit rtpengine instead.

homer-id becomes the per-node key when you correlate several nodes. So give each relay a distinct one.

Place one call through the relay, then check that sipnab saw control traffic:

sudo sipnab -N -d eth0 --report

A working setup names the call under Calls named by a media relay. If it does not, work down this list:

--hep-parse unwraps a HEP datagram and hands the payload inside it to the parser.

• That is what you want when a collector feeds you SIP over HEP.
• Until 0.5.128, it quietly switched this feature off.

sipnab spots mirrored ng two ways:

• by the metadata a HEP listener recorded before stripping the wrapper,
• by parsing the wrapper off a sniffed datagram that still has one.

Unwrapping used to discard the wrapper without recording what it said. So neither route had anything left to match on.

• The control plane reached the capture and went straight in the bin.
• No counter moved.
• Every relay stream stayed orphaned.

Nothing reported it, and that is the part worth remembering:

• The build carried the feature.
• The filter carried the ports.
• The datagrams sat plainly on the wire.

Every setting was correct, and the answer was still empty.

From 0.5.128, the unwrap carries the wrapper's capture protocol and correlation id forward. So the two features coexist, and the flag is safe to leave on. On an older build, turn it off for a relay capture.

rtpengine CONNECTS its Homer socket. So a destination that answers with ICMP port-unreachable makes it drop the socket and log this:

ERR: [core] Connection error from Homer at 192.0.2.1:9060: Connection refused

rtpengine loses the copy it was sending. That looks exactly like the feature not working.

• rtpengine 26.3 (the build Add rtpengine to your voice stack makes) reconnects at the next call.
• While nothing listens, it logs Write error to Homer at 192.0.2.1:9060: Connection refused for each call it tries to mirror.

Pointing --homer at an address nobody listens on is therefore not a way to "just put it on the wire". Something has to absorb it.

• In the deployment this targets, that something is your real Homer collector, which is already there.
• If you are testing without one, run any UDP sink at the address. The harness ships one for exactly this reason. See harness/hep-sink.

Everything above reads the control plane off the wire. So it names the calls whose offer crossed that wire while sipnab was running.

Incident response rarely starts that way.

• You attach to a relay because something is wrong now.
• The calls you care about went up minutes or hours ago.
• Their offer happened in the past. No capture can recover it.
• Their media lands in the orphan list.

--rtpengine-control closes that gap by asking the relay. Give it the address rtpengine's ng control socket listens on, its listen-ng value. Same host first:

sudo sipnab -N -d eth0 --rtpengine-control 127.0.0.1:22222 --report

A relay on its own host answers the same question over the network:

sudo sipnab -N -d eth0 --rtpengine-control 192.0.2.40:22222 --report

-N earns its place in both.

• sipnab writes these summaries as log lines on stderr.
• The TUI silences logs to keep the alternate screen intact.

sipnab asks at two moments and no others:

• once at startup, before the capture opens,
• again when a stream turns up that nothing else explains.

There is no interval flag because there is no timer. A tool that talks to a production relay on a schedule is a service, and this is a diagnostic.

The startup question happens before the first packet. So its answer comes first. Against rtpengine 12.5.1:

rtpengine at 127.0.0.1:22222: 2 call(s) enumerated, complete; queried 2 of them, 8 relay port(s) now attributable

Read it in three parts:

• Enumerated, complete means the relay listed everything it held, rather than capping the answer.
• Queried 2 of them is one query per call. That is the only command that says which relay-side port belongs to which call.
• 8 relay ports now attributable is the index sipnab keeps for the rest of the run.
  • Media on any of those eight sockets takes its Call-ID from memory.
  • It costs the relay nothing further.

Streams the relay accounts for stop orphaning. Some calls' own signaling never appears in the capture. That is the whole reason for asking. Such a call lands under Calls named by a media relay, the same heading the mirrored control plane fills.

Every stream says who named its dialog, on every door:

• GET /v1/streams and GET /v1/streams/{id} carry it as dialog_assertion.
• The MCP rtp_stats tool carries the same key with the same two spellings.
• The TUI's stream detail prints (via media-relay) beside the Call-ID.

The distinction changes what the address means. That is why it is not folded away.

• A relay's answer is not an endpoint. It names the box the media passed through, not where either party sits.
• An operator may trace a one-way-audio fault to 192.0.2.40:38664. That operator needs to know whether that is the far end or the box in the middle. Those lead to opposite next steps.

It is also only as trustworthy as the path it arrived on. This page used to say it was "authoritative about the port".

• That was true of rtpengine and false of the assertion.
• rtpengine cannot be wrong about which socket it opened.
• But sipnab is not always hearing from rtpengine.

Read the next section before you act on a media-relay address.

An absent key means nobody recorded who asserted the binding. It does not mean signaled. That is a claim, and keeping the two apart is the entire reason the field exists.

Two paths deliver it, and they are worth different amounts.

Nothing authenticates a sniffed assertion. It carries:

• a Call-ID copied verbatim out of the HEP correlation-id chunk,
• a media address copied out of the SDP.

So a forged datagram can name a call of the sender's choosing. It can bind it to a socket of the sender's choosing.

sipnab therefore believes a sniffed mirror only on UDP port 9060, the HEP port.

• That narrows the input without authenticating anybody. This page says so plainly, so nobody reads it as more.
• It stops every datagram on the segment being a candidate.
• It stops nothing sent to 9060.
• sipnab logs one refusal, naming the port it saw.
• The rule is the same with --hep-parse on.
  • The flag unwraps the datagram before sipnab reads it.
  • sipnab still believes the mirror inside only if the datagram went to port 9060.

--hep-allow does not reach this path. It guards the --hep-listen socket, and there is no socket here.

If any of that matters for your estate, take the delivered path and authenticate it:

sudo sipnab -N -d eth0 --hep-listen 0.0.0.0:9060 \
    --hep-auth-file /etc/sipnab/hep.key --hep-auth-mode hmac

Then point rtpengine's homer = at sipnab.

• It costs you the Homer collector. rtpengine takes exactly one destination.
• That is the trade the table at the top of this page is about.

If your collector sits on a port other than 9060, sipnab does not read the sniffed copy at all. That is deliberate. The alternative is believing every port.

dialog_assertion is a different question from dialog_origin. That second key says which capture source delivered the assertion.

• A relay can name a stream over a HEP mirror, or answer sipnab directly.
• A party's SDP can arrive off the NIC.

The two keys answer one half of that each.

The second summary arrives when the capture ends:

rtpengine at 127.0.0.1:22222: 2 unexplained stream(s) offered, 0 attributed, 4 control transaction(s) spent of a ceiling of 66

• The capture path handed two streams to the asking thread.
• Neither of them gained a Call-ID.
• The run spent four control transactions of the 66 it may spend.

A count of zero with no reason line beside it is the honest answer, rather than a failure. The relay answered and holds neither port. So that media belongs to something else on the host.

Reading a file, sipnab refuses to ask, and says why:

sipnab -N -I capture.pcap --rtpengine-control 127.0.0.1:22222

--rtpengine-control 127.0.0.1:22222 asks a live relay which calls are up
right now, but this run is reading a capture FILE — offline analysis never
transmits. The calls in a file ended in the past; the relay's answer would
describe whatever is up TODAY, which is other people's traffic. Passive
decoding of any relay control plane already in the capture still runs;
capture live with -d <device> to ask.

The analysis still runs and still exits 0. What the run loses is the ask. It says so before the capture opens, rather than looking like a run that asked and learned nothing.

The two questions sipnab can put to a relay are list and query.

• That is a property of the type reaching the relay, rather than a convention.
• ng also carries offer, answer, delete and start recording. Each of them changes a production relay.
• None of them has a representation on this path. So no call site reaches one by accident.

Three bounds hold the asking down. None of them grows with how much traffic the capture carries:

• sipnab asks about each relay-side socket at most once for the whole run. So a stream that stays unexplained does not re-ask on every packet.
• A per-run ceiling caps control transactions at 66.
  • That is one list plus a query per call at rtpengine's own list limit of 32, twice over.
  • It is enough for a full startup snapshot and one comparable refresh, and no more.
• The queue handing sockets from the capture path to the asking thread stops at 4096 entries.
  • That is two per stream, so 2048 unexplained streams.
  • It counts what it turned away, rather than letting a slow relay grow it without end.

When a bound bites, sipnab counts it and says so. The asking runs on its own thread. So the packet path offers a socket and moves on, rather than waiting out a round trip to a relay that may be down.

"Nothing came back" has five meanings. sipnab keeps them apart rather than collapsing them into one shrug:

• The relay did not answer, or refused the question. sipnab knows nothing about the port.
• The relay named calls sipnab could not read. The port may belong to one of those.
• The relay capped its own enumeration. The list came back partial. So the port may belong to a call the relay never named. sipnab does not accept that quietly:
  • A capped first answer earns one wider ask, for up to 1024 Call-IDs.
  • The summary says whether that wider ask landed or failed.
• The run spent its transaction ceiling. sipnab never asked about the port at all.
• The relay does not hold the port. This one, and only this one, says something about the relay: the stream is not its media.

The first four are gaps in what sipnab learned. Reporting any of them as the fifth would turn a run that never reached the relay into a run that asked and heard the stream belongs to nobody.

Attribution answers whose media a stream is. A relay also keeps counters about itself:

• the packets it relayed,
• the sessions it holds,
• its own loss and jitter.

--relay-stats asks the relay named by --rtpengine-control for them, and prints them:

sudo sipnab -N -d eth0 --rtpengine-control 127.0.0.1:22222 --relay-stats

Every figure names the relay as its source.

• The header marks it relay_reported.
• That is a claim the relay makes about itself, not a measurement sipnab took from the wire.
• Reading it as "sipnab saw this" is the one mistake this whole view exists to stop.

To watch a counter move rather than reading one snapshot, poll on a timer:

sudo sipnab -N -d eth0 --rtpengine-control 127.0.0.1:22222 --relay-stats-interval 5

Each reading marks itself polled and names the interval. So it reads as a timer's output rather than a one-shot answer.

The tier is the whole point here, not a footnote. One call carries three different "loss" figures. Each one describes a different socket at a different point on the path:

• The relay's own count (relay_reported).
  • This is what the relay counted on its own sockets since its last restart.
  • --relay-compare puts it beside sipnab's own count for one call. It labels each, and prints a one-word verdict with a note.
• What sipnab saw on the wire (sipnab_measured).
  • These are gaps in the RTP sequence numbers that reached the capture point.
  • So it reflects only what the capture caught.
• What the far end claims (endpoint_reported).
  • This is the remote endpoint's own assertion over RTCP.
  • Nobody can check it.

sipnab never subtracts one tier from another.

• The relay's count minus sipnab's is not "packets sipnab missed".
• The two count different sockets over different windows. So their difference describes nothing.
• Name the tier when you quote the number: "the relay reports 40 lost, our capture measured 3 gaps".
• Then the disagreement stops being a contradiction. It becomes three facts about three points on the path.

The cookbook works each question end to end. The relay-statistics recipes cover:

• whether a relay drops packets (recipe "Ask a relay whether it is dropping packets"),
• whether it holds sessions nobody released (recipe "Ask a relay what it is still holding"),
• whether its view of one call matches yours (recipe "Compare a relay's per-call count with your capture"),
• this three-tier loss comparison (recipe "Read a relay's loss beside the loss you measured").

--relay-stats needs a live source.

• Asking a relay transmits.
• A capture file's addresses belong to third parties and describe the past.
• So on -I <file>, sipnab refuses to ask and says why. It does not reach an address the file named:

sipnab -N -I capture.pcap --relay-stats --rtpengine-control 127.0.0.1:22222

--relay-stats will not ask a relay on a run that reads a file: asking
transmits, and a file's addresses are historical and belong to third
parties. Ask from a live capture (-d <device>).

The rest of the analysis still runs and still exits 0. What the run loses is the ask.

Everything above is a person at a terminal.

• query_relay gives an AI agent the same reach.
• decode_ng lets it read a control message it finds in the capture.

query_relay transmits. So it stays off until you say otherwise. It needs three things. It refuses with a message naming whichever one is missing:

# Off by default: this server has the relay address but no opt-in, so
# query_relay refuses and every other tool works as usual.
sudo sipnab --mcp -N -d eth0 --rtpengine-control 127.0.0.1:22222

Add the opt-in to enable it:

# Now an agent can ask the relay directly. The destination is THIS address --
# there is no tool argument that can change it.
sudo sipnab --mcp -N -d eth0 --rtpengine-control 127.0.0.1:22222 \
  --mcp-allow-relay-query

The third requirement is a live source. It is structural rather than a policy check. A run reading a capture file cannot obtain a transmit permit. So --mcp-allow-relay-query on an offline run leaves the tool refusing:

# Refuses: -I is a file, so no transmit permit exists to send with.
sipnab --mcp -N -I capture.pcap --rtpengine-control 127.0.0.1:22222 \
  --mcp-allow-relay-query

That matters more than it looks.

• An analyst may open a capture from another organization. That analyst must not be able to make sipnab talk to the addresses inside it.
• A host that served as a relay during the capture may be somebody's laptop now.

decode_ng needs none of this.

• It reads a control message already in the capture.
• It reports which path delivered it.
• So an agent can tell "the relay told us" from "something claiming to be the relay told us". Those are the same bytes and very different claims.

Nothing, and a measurement says so rather than an assumption.

rtpengine forwards in one of two places:

• in userspace,
• or, with table = N and the xt_RTPENGINE kernel module, inside netfilter.

If kernel-forwarded media were invisible to a capture, attributing it would be pointless. So this run checked it against rtpengine 12.5.1. The run confirmed that the module was active and accounting the packets itself:

/proc/rtpengine/0/list   local 192.0.2.40:34232  250 packets   [kernel-forwarded]
                         output → 192.0.2.60:40001             250 packets
capture on the relay     ingress 500/500         egress 500/500

Both directions stay fully visible in both modes. The reason is structural:

• A capture's receive tap runs before netfilter.
• The module re-injects through the normal transmit path. That path passes the transmit tap.

You do not have to know or care which mode a relay runs in.

Stated here rather than discovered later.

• It does not tie the two halves of a B2BUA call together. A B2BUA gives one call two Call-IDs. The control plane names each leg's media and ties neither to the other.
• It does not read DTLS-SRTP media. rtpengine terminates DTLS and emits no key log. So on a WebRTC-facing relay the payload stays unreadable.
• It does not attribute recording or forking streams.
  • Those commands create media that belongs to the call without being one of its two legs.
  • Counting one as an ordinary leg would turn a two-party call into a three-stream one.
  • sipnab counts them and says so instead. --report prints Media-creating relay commands seen with the count, beside the relay-named calls.
  • Their media joins the orphaned streams where the capture holds it.
• It does not name a call that started before the capture did, from the wire alone.
  • A control message that already happened never reaches the wire. So no capture can recover it.
  • Give --rtpengine-control, and sipnab asks the relay instead. That closes the gap on a live run.
  • Their media stays in the orphan list in two cases: without the flag, or on an -I <file> run where sipnab refuses to ask.

None of this section describes what sipnab does today.

• What this page documents above is one hop naming its own media.
• The proof ran against a recorded capture and against a live relay.
• This section states what that foundation exists FOR. So the next phase aims at a goal rather than at whatever comes next.

The goal is a call that crosses an SBC, a proxy, an rtpengine and a PBX, assembled end to end.

• Then an operator can ask a question in their own words.
• An agent answers with evidence from every hop the call touched.

Naming media on the relay is the hop that was missing. It is the only hop that carries no SIP of its own. Once each hop can name a call, an agent can:

• filter to a customer, by caller, callee, trunk, address, or time window,
• pull the whole call rather than one machine's view of it.

These are the questions that arrive, and what answering each one takes.

The hard part is not measuring quality. It is locating it.

• A relay reporting 2% loss tells you nothing on its own.
• The question is whether the loss was already there when the media arrived, or appeared on the way out.

That needs the same call measured on both sides of each hop. That needs each hop to agree on the call's name. On the relay, that agreement is what this page adds.

The analysis then reads as a chain:

• clean into the SBC,
• clean into the relay,
• lossy out of the relay.

Now the fault has an address rather than a symptom.

Watch for the answer being "neither hop":

• Identical loss at every hop points at the access network.
• A codec that changes between hops points at transcoding rather than at the network.

Same filter, different evidence. Here the media is usually absent rather than degraded. So the question moves to signaling, and to the gap between them.

The failure modes separate cleanly once you can see every hop at once:

• A final response from an upstream hop, which names the rejecter. The response class matters, and the classes lead to different owners:
  • A 4xx is the far end declining.
  • A 5xx is a server failing.
  • A 6xx is a global refusal.
• No response at all, which is a timeout. The hop that stopped answering is the one to look at.
• A call answered with no media, which is a negotiation failure rather than a routing one. The signaling succeeded and the SDP did not.
• A relay with no ports left. This looks like a random failure from the proxy's side. It is obvious from the relay's.

The last two are exactly the cases a signaling-only view gets wrong. From the proxy, the call looks answered.

A drop has a clock attached, and the clock identifies the cause.

• A tear-down at a round number is a timer: a session timer, a NAT binding, a relay's own media timeout.
• A tear-down when media stopped is the media stopping.
• Reading the relay's teardown against the proxy's BYE says which came first. That is the whole question.

One-way audio is a direction problem. A relay is where directions become visible.

• It holds both parties' sockets.
• So a leg with media arriving and nothing leaving is plainly asymmetric.

The value of naming the call is being able to say WHICH call is asymmetric on a box carrying hundreds.

The comparison case.

• Two calls of the same shape go down two trunks.
• Their assembly is the same.
• They differ in one hop's behavior.

This is hard to do by hand across machines. It is straightforward once every hop names its calls the same way.

• Encapsulations — HEP and the other wrappers sipnab reads
• Capture SIP over TLS — when encryption hides the signaling
• MCP tools — the agent-facing surface over these results