Troubleshooting - NormB/sipnab GitHub Wiki

Under pressure? Each scenario below has four parts:

• Problem
• Command
• What to look for
• Next steps

Copy-paste and go.

Match the complaint, not what you think is wrong. The section works back to the cause from there.

Whatever the symptom, three things decide whether the answer is in the capture at all:

• The capture covers the right time window.
• It comes from a point that sees both directions.
• --portrange covers the ports in use.

Start here deals with the third.

Don't know which problem you have yet? The --problems alias surfaces every troubled call in a single sweep. That means failed calls plus:

• one-way audio
• high loss/jitter
• NAT mismatches
• retransmit storms
• slow setup
• media asymmetries

# Every problematic call
sipnab -N -I capture.pcap --problems --json

--problems is shorthand for the full Filter-DSL expression state == 'Failed' OR one_way == true OR rtp.loss > 5.0 OR rtp.jitter > 50.0 OR nat_mismatch == true OR retransmits > 3 OR pdd > 11.0 OR .... So it is a superset of "failed calls".

Once you know the symptom, jump to the matching section below for the precise filter.

First, check you are reading the whole capture.

• --portrange defaults to 5060-5061.
• sipnab skips any SIP message whose source and destination ports both fall outside it.
• Skipped messages reach no count, no dialog and no output.
• So a call carried on 5070 or 5080 is simply not in any answer below. Those ports are ordinary on carrier trunks and SBCs.

sipnab says so on stderr and again at the end of the run:

NOT ANALYZED: 1 further SIP message(s) were seen on ports outside --portrange
and are in none of the totals above. Busiest: 8090 (1).

You may see that line, or the call you are chasing may be missing. Then add --portrange 1-65535 to every command on this page and start again.

• On a live capture the range becomes the kernel BPF filter.
• So the traffic never reaches sipnab, and nothing reports it missing.
• Set the range before you capture.

Second, check sipnab could read the capture at all.

• Some frames are on a link type, EtherType or IP protocol sipnab has no decoder for. Such a frame produces nothing.
• It counts as a packet (it arrived).
• It contributes to no message, dialog or stream.

sipnab reports those separately, with the numbers that name them:

NOT DECODED: 3 of 3 frame(s) (100.0%) produced nothing and are in none of
the counts above. Reasons: unsupported link type 147 (3). NOTHING IN THIS
CAPTURE WAS READ — every frame failed to decode, so the totals above
describe no traffic whatsoever and a zero among them is not evidence of
absence.

This line shows the difference between two cases:

• there is no SIP in this capture
• I could not read one single frame of this

The totals alone render both cases identically. When the share is high:

• Treat every zero on this page as unknown, not as a finding.
• Fix the decode first.

The number in each reason is what you act on:

Reason	What it means	What to do
unsupported link type N	The pcap's DLT has no decoder here. Link types lists the ones sipnab reads, and 147 to 162 are the private-use DLT_USER range.	Convert it: editcap -T ether in.pcap out.pcap. If the link type is one sipnab should read, open an issue naming the number.
not IP (EtherType 0xNNNN)	The frame decoded and carried no IP. 0x0806 is ARP, 0x8847 MPLS, 0x88CC LLDP.	ARP and LLDP are ordinary background -- expect a few on any Ethernet capture. A large MPLS or PPPoE share means the mirror is giving you the encapsulated form.
ESP not NULL-encrypted (IP protocol 50)	IPsec ESP whose payload failed the checks that prove NULL encryption, so it carries real encryption, or a protection sipnab cannot read.	sipnab takes no ESP keys. Take the capture inside the tunnel, or run the lab with NULL encryption, which sipnab reads. See Tunnels above the link layer.
no transport (IP protocol N)	IP decoded, and its payload is no transport sipnab handles. 2 is IGMP, 89 OSPF, 103 PIM.	Routing and multicast control traffic is ordinary background. A large share of another number means a tunnel sipnab does not strip.
truncated frame	The frame is shorter than a header it declares.	Raise --snaplen on the capture, or re-take it.
decode error	The decoder rejected the bytes outright.	Usually a corrupt or mis-declared file; try editcap or tshark -r on it.

Are calls rejected with 403 Forbidden, 404 Not Found, 486 Busy Here or 488 Not Acceptable Here? Or do they time out with 408 Request Timeout? Find every call that never established. Then triage by response code.

List every failed call, one line per response message. Each line carries:

• the Call-ID
• the response code
• the reason text

sipnab -N -I capture.pcap --filter "state == 'Failed'" --json \
  | jq -c 'select(.is_request == false) | {call_id, status_code, reason}'

You should see one line per response message of each failed call (minimal example):

{"call_id":"abc123@host","status_code":100,"reason":"Trying"}
{"call_id":"abc123@host","status_code":486,"reason":"Busy Here"}
{"call_id":"def456@host","status_code":408,"reason":"Request Timeout"}

One of those Call-IDs may be worth escalating. Then write the detailed Markdown report for that single call and attach it to a ticket.

• Substitute the Call-ID you picked.
• The redirect overwrites report.md in the current directory.

--no-cli-print is not optional here.

• Without it, sipnab dumps every message in the capture ahead of the report.
• The file you attach then opens with hundreds of lines of raw SIP.

sipnab -N -I capture.pcap --call-report "abc123@host" --markdown --no-cli-print > report.md

What to look for: sipnab includes response code intelligence. The status field tells you why:

Next steps:

• The response code may be 408, or you may see high retransmits. Then the problem is network-level.
  • Check connectivity and firewall rules before touching SIP config.
  • Nothing came back turns that "network-level" guess into evidence.

A 408 and a call that simply stops both look like "the far end did not answer". Usually that is an inference from silence. Sometimes it is not.

• A router or the host itself may have sent an ICMP error quoting your request.
• Then the network stated the cause, and sipnab reads it.

Any run that saw such an error ends with a capture-wide tally on stderr. So you do not have to go looking for it. A run that saw none prints nothing:

ICMP: 4 error(s) quoting a SIP request, naming 2 unreachable endpoint(s).
Busiest: 192.0.2.10:5060 (2, port unreachable), 192.0.2.11:5080 (2, host unreachable).

When you see that line, pull the affected calls out as one object each. Then read the finding:

sipnab -N -I capture.pcap --json-dialogs --no-cli-print --quiet \
  | jq -c 'select(.signaling_diagnosis.icmp_unreachable) |
           {call_id, method,
            code: (.final_status_code // .signaling_diagnosis.final_failure.code),
            icmp: .signaling_diagnosis.icmp_unreachable}'

Then read one of them in full. The report:

• lays the ICMP finding beside the SIP timeline
• names the messages it came from

sipnab -N -I capture.pcap --call-report 'abc123@host' --no-cli-print

Signaling Issues:
  - Final failure: 408 Request Timeout
    evidence: #2 408 Request Timeout
  - ICMP port unreachable: 192.0.2.10:5060 unreachable (2 times), reported by 198.51.100.1
    evidence: #0 OPTIONS, #1 OPTIONS

What to look for:

Next steps:

• port unreachable means the host is up and nothing is listening on that port.
  • Check that the SIP service is running and bound where you think.
  • Check that the port in the Contact/Via matches what it actually listens on.
  • This is not a network fault.
• host unreachable or network unreachable is a routing problem short of the destination.
  • The reporter names the hop that gave up.
  • Nothing reached the host, so the capture says nothing about its ports.
• administratively prohibited (v4 codes 9, 10, 13; v6 type 1 code 1) is a firewall or router ACL, not a dead host.
  • The peer may be perfectly healthy and answering everyone else.
  • The fix is on the filtering device.
  • That is a different team from the one you call about an unreachable host.
  • Count the two kinds separately before you dispatch anyone. One capture can hold both.
• No ICMP at all does not clear the network.
  • Most firewalls drop ICMP errors outright.
  • The summary can only report what the capture holds.

A call may show both retransmissions and icmp_unreachable. Read them together rather than as two problems.

• The retransmission count is how hard the sender tried.
• The ICMP error is why nothing came back.

sipnab annotates the retransmission finding with icmp_cause.

• It stops offering its own guess at the reason.
• It keeps the count. "11 OPTIONS over 300 s" and "3 INVITEs over 3 s" are the same cause. They are a very different operational picture.

There is no icmp field in the Filter DSL. So select these calls with jq on --json-dialogs as above, not with --filter.

The call sets up fine and both sides talk. Then it dies partway through. This often happens after a suspiciously round number of minutes.

Enumerate the calls that established and then ended early. These calls completed, but were far shorter than a real conversation:

sipnab -N -I capture.pcap --filter "duration < 120.0 AND state == 'Completed'" --json \
  | jq -r '.call_id' | sort -u

Then print the timeline for one of those calls. It shows who sent the BYE, and exactly when. Substitute a Call-ID from the list above for abc123@host:

sipnab -N -I capture.pcap --call-report 'abc123@host' --no-cli-print

The call report shows the full SIP message timeline. It also shows per-stream RTP stats, including first/last packet timestamps. Match the signature:

• BYE at a round interval after answer: RFC 4028 session-timer expiry.
  • The interval is exactly 15 min, 30 min or 1 h. For example: 200 OK at 14:00:02, BYE at 14:30:02.
  • One side never sent (or never received) the session refresh re-INVITE/UPDATE.
  • It tore the call down when Session-Expires ran out.
• RTP last packet well before the BYE: a NAT/firewall idle timeout silently dropped the media path.
  • The stream's last_seen is minutes earlier than the BYE.
  • The endpoint's RTP-timeout watchdog eventually hung up.
• BYE from the carrier side, accompanied by SIP retransmits: trunk-side reset, or an upstream element recycling the session.

Next steps:

• Session timer:
  • Align Min-SE / Session-Expires between PBX and trunk.
  • Confirm the negotiated refresher actually sends the refresh before expiry.
• NAT/firewall:
  • Enable RTP keepalives (or comfort noise) on the endpoints.
  • Raise the firewall's UDP session timeout above the keepalive interval.
• Disable SIP ALG on intermediate NAT devices. It corrupts dialogs in ways that surface as mid-call drops.

An INVITE comes back 488 Not Acceptable Here. The callee (or an SBC in the path) found no common codec between the SDP offer and what it supports.

Find every 488 rejection in the capture:

sipnab -N -I capture.pcap --filter "state == 'Failed'" --json \
  | jq -c 'select(.status_code == 488) | {call_id, status_code, reason}'

You should see:

{"call_id":"abc123@host","status_code":488,"reason":"Not Acceptable Here"}

Then compare the SDP offer against the answer for one of those calls. Substitute its Call-ID for abc123@host:

sipnab -N -I capture.pcap --call-report 'abc123@host' --no-cli-print

The call report's SDP timeline lists each offer/answer with its codec set.

• A 488 means there is no overlap.
• For example, the offer carries only G729, while the callee's profile allows only PCMU, PCMA.
• The reject may come from an SBC. Then the offer may never have reached the far end at all.

Next steps: Do one of these, so the offer and answer share at least one codec:

• Enable transcoding on the SBC/media server.
• Add the missing codec to the endpoint/trunk profile.

One direction of RTP has zero packets. The caller can hear the callee (or vice versa) but not both.

The --one-way alias finds every call flagged for one-way audio. It dumps the matching SIP messages as NDJSON:

sipnab -N -I capture.pcap --one-way --json

You should see one NDJSON record per SIP message of each flagged call. This example shows an abridged record. Real records carry the full field set:

{"is_request":true,"method":"INVITE","call_id":"[email protected]","from":"...","to":"...", "...":"..."}

For the same set of calls rendered as the full diagnostic report rather than raw records:

sipnab -N -I capture.pcap --one-way --report

--one-way is shorthand for a Filter-DSL condition. To combine it with others, write that condition out:

sipnab -N -I capture.pcap --filter "one_way == true" --json

Get the diagnosis detail (one_way_audio, nat_mismatch, hints) per call with --call-report <call-id>.

Before reasoning about NAT, look at the end of the run.

• Something may have sent an ICMP error about the media itself.
• Then the network has already stated the cause, and there is nothing to infer:

ICMP: 27 error(s) quoting non-SIP traffic, 27 of them media, across 6 flow(s). Attributed to a stream or SDP endpoint: 27; matched nothing this capture holds: 0.
  ICMP port unreachable: RTP (payload type 0) from 192.0.2.5:42180 to 198.51.100.9:21750 could not be delivered (11 times), reported by 198.51.100.9. This is one of the media streams in this capture (1 call(s) affected). Audio sent that way is discarded before it arrives, which is heard as one-way or missing audio. The host answered, so it is reachable -- nothing was listening on that port. Check the service and the address it binds, not the network.

sipnab ties each media ICMP error to a call by trying these matches in order, and each line names the one that succeeded:

• the quoted datagram's own 5-tuple against a tracked stream
• the SSRC inside the quote
• either socket against a tracked stream endpoint
• an SDP-advertised media address, or the RTCP port one above it (RFC 3550 Section 11)

The first two identify the stream itself. The last two identify an address the stream used, or one the SDP offered.

A quote that matches nothing is still counted and printed, under matched nothing this capture holds. The endpoint it names is real. Widen the capture to include that endpoint's traffic, then read the run again.

What to look for:

• ICMP errors against the media ports (above).
  • This is a stated cause, not an inference.
  • Read these before anything else in this section.
• nat_mismatch == true alongside one_way.
  • RTP reached the capture point from an address no SDP in the dialog advertised.
  • That is what a NAT rewriting the media source looks like.
  • This is the most common cause.
• Codec asymmetry in the SDP offer/answer. One side offers a codec the other doesn't support.
• RTP ports in the SDP that never receive traffic. A firewall is blocking the return path.

Next steps:

• The run may have printed a media ICMP line. Then act on that first.
  • It names the socket that rejected the audio and the device that reported it.
  • Nothing below can improve on a router's own statement.
• Check NAT: sipnab -N -I capture.pcap --filter "one_way == true AND nat_mismatch == true" --json
• If NAT is the cause: enable fix_nated_contact / fix_nated_register on the proxy, or deploy a TURN server.
• If NAT is clean:
  • Verify symmetric RTP.
  • Check for SIP ALG on intermediate firewalls (disable it).
  • Confirm both endpoints negotiate a common codec.

The hardest silent call to find is the one where nothing is wrong with the packets.

• A gateway may send full-rate frames of digital silence.
• It puts every packet on the wire, in sequence, at the right rate, in both directions.
• Loss is zero and jitter is zero.
• The MOS is 4.36.
• No other check on this page fires.

Counting comfort-noise frames does not find it either. Those frames say "I am sending silence", and this gateway is not saying anything.

So sipnab measures the audio itself. It needs the samples, which means --retain-audio. Set CALL_ID to the call that went quiet, then run:

CALL_ID='[email protected]'

sipnab -N -I capture.pcap --retain-audio --call-report "$CALL_ID" --json --no-cli-print

For a clean call, here the first call in the repository sample tests/pcap-samples/sip-rtp-g711.pcap, the report's diagnosis.amplitude object reads:

{
  "streams_measured": 1,
  "dead_air": false,
  "clipping": false,
  "streams": [
    {
      "ssrc": "0x343da99b",
      "src": "10.0.2.15:27942",
      "dst": "10.0.2.20:6000",
      "codec": "PCMU",
      "report": {
        "sample_rate": 8000,
        "duration_ms": 8500,
        "full_scale": 32124,
        "floor_dbfs": -60.0,
        "window_ms": 100,
        "min_dead_air_ms": 1000,
        "clip_threshold": 31802,
        "min_clip_run": 3,
        "dead_air": [],
        "dead_air_ms": 0,
        "clipping": [],
        "clipped_samples": 0
      }
    }
  ]
}

The diagnosis.amplitude object appears only on a run that kept the samples. Absent means not measured, never "clean". The same command without --retain-audio prints null.

There are two measurements. Each has the threshold that produced it beside the number:

Both are amplitude measurements, not quality scores. They report what the samples did, and they do not change the MOS.

Reading the result.

• Dead air with a start time near the answer points at the media path.
  • Check NAT traversal issues and One-way audio first.
  • A call that never carried audio is a different fault from one that stopped.
• Dead air starting mid-call, with packets still flowing both ways, is the gateway or the far-end mixer.
• Clipping is a gain stage.
  • Something upstream is driving the encoder into its ceiling.
  • No network change fixes it.

sipnab_diagnosis_total{kind="dead_air"} and {kind="clipping"} count calls where the measurement fired. Neither ticks on a run that kept no audio.

MOS below 3.0 means quality degradation users notice. Below 2.5, calls are unusable.

Find the calls in a capture whose MOS fell below the noticeable-degradation line:

sipnab -N -I capture.pcap --filter "rtp.mos < 3.0" --json

You should see the SIP messages of every matching call as NDJSON (abridged):

{"is_request":true,"method":"INVITE","call_id":"bad-audio@pbx1", "...":"..."}

The per-message records identify which calls are bad. Pull the per-stream numbers (jitter_ms, loss_pct, quality intervals) with --call-report <call-id>.

To watch a live interface instead:

• Widen the same filter to jitter spikes and let it run.
• Each matching call prints as it happens.
• Live capture needs raw-socket access, hence sudo:

sudo sipnab -N -d eth0 --filter "rtp.mos < 3.0 OR rtp.jitter > 50" --json

What to look for:

Next steps:

• Jitter is high but loss is low: the problem is buffering or path instability. Check for Wi-Fi hops, VPN tunnels, or missing QoS marking.
• Loss is high:
  • First rule out sipnab itself.
  • Then run a path MTR/traceroute to find where packets are dropping.

Before you chase high loss on the network, check whether sipnab dropped the packets.

• A capture may lose packets to a full kernel ring buffer.
• It then reports the missing RTP as network loss.
• The numbers look identical, and the fix is on the wrong machine.

sipnab warns on the first drop. It prints a summary at the end of a live capture.

• You may see that warning.
• Or loss may look implausibly high across every call at once.
• In either case, the figure is measuring your capture rather than the call.

Tuning capture on a busy server covers:

• how to read the two drop counters
• what each one means
• what to change

This applies to live capture only.

• An offline -I read of an existing pcap loses nothing.
• The pcap itself may have lost detail in capture, by whatever wrote it.

In the TUI, open the Stream Detail view:

• Navigate to a call's flow view, press F6 to show its RTP streams, and press Enter on a stream's row.
• Or press r to jump to the streams list, then Enter on a stream.

This view shows:

• MOS and jitter sparklines.
  • These are visual trend graphs across the stream's lifetime.
  • They make it easy to spot the exact moment quality degraded.
• Quality intervals.
  • This is a per-interval breakdown of MOS, jitter, and loss.
  • You can correlate degradation with specific time windows.
• Burst/gap analysis (RFC 3611).
  • It distinguishes between bursty loss (congestion events) and gap loss (steady-state impairment).
  • Bursty loss points to queue overflow.
  • Gap loss points to a consistently lossy link.
• Silence detection.
  • It identifies periods where no RTP was flowing.
  • These can indicate hold events, codec DTX, or network black holes.

This same data is available in the browser analyzer at sipnab.com/analyze/ under the Streams tab.

PDD over 3 seconds is perceptible to users. Over 5 seconds and they'll hang up.

Find the calls whose post-dial delay crossed the 3-second perceptibility line:

sipnab -N -I capture.pcap --filter "pdd > 3.0" --json

The --slow-setup alias asks a looser question than that filter.

• It expands to pdd > 11.0, sipnab's own post-dial-delay threshold.
• That threshold comes from ITU-T E.721's target for an international connection.
• A capture does not say what kind of call it holds. So the shipped figure is the most generous one.

A network that knows its traffic is local or toll wants a tighter one.

• Pass --pdd-threshold 6.0 (or set [diagnosis] post_dial_delay_secs).
• Both --slow-setup and --problems move with it.

Pair the alias with --report for a summary instead of per-message records:

sipnab -N -I capture.pcap --slow-setup --report

What to look for:

• High retransmits alongside high PDD. The caller keeps retransmitting the INVITE, for one of two reasons:
  • The first one never landed.
  • The remote side is slow to respond.
• DNS resolution delays. These are common when the proxy does NAPTR/SRV lookups for every call.
• Deep proxy chains adding latency at each hop.

Next steps: Compare pdd with retransmits.

• If retransmits > 0, the delay is network loss or an unresponsive next hop.
• If retransmits == 0 but PDD is still high, the downstream server is slow to route. Check its logs, database lookups, or LCR table performance.

The Contact or Via header advertises a private IP that doesn't match the actual packet source.

Find the calls where the address in the SIP headers disagrees with the address the packet came from:

sipnab -N -I capture.pcap --filter "nat_mismatch == true" --json

The --nat-issues alias is the same selection without writing the filter out:

sipnab -N -I capture.pcap --nat-issues

What to look for: nat_mismatch == true means RTP for the call reached the capture point from an address that no SDP in the dialog advertised.

• The far end therefore sends its media to the address the SDP named.
• Nothing answers on that address.
• So the return path fails, even though signaling completed.

sipnab compares addresses only, not ports. NAT and RTP proxies rewrite the port on healthy calls too.

Next steps:

• Proxy-side: Enable fix_nated_contact and fix_nated_register (OpenSIPS/Kamailio). They rewrite Contact headers with the observed source address.
• Endpoint-side: Configure STUN/TURN on the phone or softclient, so it discovers its public address.
  • Check that it worked. See NAT discovery went unanswered below.
  • A phone configured with a STUN server it cannot reach behaves exactly like a phone with no STUN at all.
• Network-side: Disable SIP ALG on every NAT device in the path. SIP ALGs almost always make things worse.

private_media_address == true means the dialog advertised a private address in its c= line.

• The address is RFC 1918, RFC 4193 or link-local.
• The peer it went to is not itself private.

sipnab raises this as a warning, not a fault. There are two situations, and only one of them fails:

• Something downstream rewrites the SDP: an SBC, an ALG, or a media proxy.
  • The address the far end finally sees is routable.
  • This is correct, common, and there is nothing to do.
• Nothing rewrites it.
  • The far end sends media to an address the internet cannot route.
  • The call still signals cleanly and answers 200.
  • The audio is one-way.

Without more to go on, sipnab cannot tell those apart from one capture. So it reports the evidence rather than a verdict.

Carrier-grade NAT space (RFC 6598, 100.64.0.0/10) is NOT flagged. It is routable inside the carrier that assigned it, so a mobile call that offers it can work.

Unless STUN settles it. The capture may also hold the client's STUN or TURN exchange. Then:

• The two situations stop being indistinguishable.
• The stun_sdp_mismatch object on the diagnosis carries the proof:

The first two raise private_media_address on their own, with no observed stream needed. So a call whose media never arrived at all still gets the finding.

The correlation is by the client's IP address. The STUN exchange and the dialog share nothing.

• The evidence may sit well outside the call: merged captures, or a probe from an earlier registration.
• Then the finding says so.
• A persistent NAT-discovery failure is an inference about the client, not an observation of this call.

sipnab reads STUN and TURN. It reports transactions that went out and never came back:

STUN/TURN: 192.0.2.10:5060 sent Binding to 198.51.100.1:3478 2 time(s) and got
no reply. An endpoint that cannot learn its reflexive address falls back to
advertising its PRIVATE address in SDP ...

How this leads to one-way audio. It is the first link in the one-way-audio chain above.

• The phone asks the network what address the outside world sees it as.
• It gets nothing.
• It falls back to the only address it knows: its private one.
• It then writes that address into its SDP.
• Everything after that looks healthy.

Read what is MISSING, not just that something is. A reply that never comes is a different fault from a reply that says no:

A retransmission counts as one unanswered question with N attempts, not N questions. A phone that retries five times has asked once, and nothing answered it five times.

A capture holding nothing else is still worth reading.

• Two unanswered Binding Requests and no SIP at all is not an empty capture.
• It is the cause of a one-way-audio complaint.
• sipnab says so, rather than reporting "no SIP traffic found".

sipnab_nat_unanswered_requests carries the same signal for a dashboard. See Prometheus Metrics.

sipnab --stun prints the whole exchange as a table. It shows one row per transaction, what came back, and how long it took. --json-stun is the same thing as NDJSON.

A TURN allocation has a lifetime. The client must Refresh it before that lifetime runs out.

• The client may not do so, or its Refresh may never reach the server.
• Then the relay tears the allocation down.
• The media stops mid-call.

This is the only fault sipnab reports that has no other symptom anywhere.

• No SIP message says the audio stopped.
• The signaling shows a healthy call that went quiet.
• Both endpoints are still happily sending into a relay that no longer exists.

sipnab flags it wherever the run reports anything:

• on stderr as TURN: N allocation(s) were still carrying traffic after the lifetime they were last granted had run out
• as LAPSED in the --stun allocations table
• as lapsed: true in --json-stun
• as a turn_allocation_lapsed finding in --analyze
• as sipnab_nat_lapsed_turn_allocations for a dashboard, /v1/stats and the MCP capture_status tool

Two things it does not claim:

• It says no Refresh was seen, never that none was sent. A capture that started late or lost a packet cannot tell those apart.
• A Refresh carrying LIFETIME 0 is a deliberate release. It is never reported, because the client asked for the teardown.

Find the audio that stopped with the relay. sipnab unwraps TURN ChannelData, so the RTP inside it appears in the stream list as ordinary media with phone-to-relay addresses. These surfaces name which streams the relay carried:

• The --stun allocations section says which channel carried which SSRC.
• The lapsed-allocation lines say media on it:.
• The --analyze finding counts relayed_streams.
• --json-stun puts a channels array on the allocation.
• A relayed call's own diagnosis carries media_relay, naming the relay its audio crossed.

On a dashboard, sipnab_nat_lapsed_turn_allocation_streams is the scale beside the allocation count. A relay torn down with nothing on it cost nobody a call.

ICE gives one agent the controlling role and the other controlled. The controlling one picks the candidate pair. When both claim the same role:

• The agent that notices answers 487 Role Conflict (RFC 8445 section 7.3.1.1).
• One side switches role.
• It repeats every check it had already sent.

Often ICE fixes this itself, and the call costs a round trip. Where it does not:

• Nothing is ever nominated.
• The call has no media path.
• The signaling looks perfectly healthy.

The usual source is one of these:

• two endpoints configured with the same role
• a B2BUA relaying one side's role attribute straight to the other

sipnab reports it:

• on stderr as ICE: N candidate pair(s) show a role conflict
• as a ROLE CONFLICT line in the --stun ICE section
• as an ice_role_conflict finding in --analyze
• in the ice record of --json-stun
• as sipnab_nat_ice_role_conflicts for a dashboard, /v1/stats and the MCP capture_status tool

Each report says whether the conflict resolved. Resolved means the agents nominated a pair between them anyway, and the conflict cost one round trip. UNRESOLVED means no pair was ever nominated: that conflict is a candidate cause of media that never started, and is the one to act on.

ICE:   192.0.2.11:50006 <-> 203.0.113.11:16002: 2 487 response(s), UNRESOLVED -- no pair between them was ever nominated, so this is a candidate cause of media that never started

The same section answers the question ICE otherwise leaves unanswered: which candidate pair won.

• nominated 192.0.2.10:50004 -> 203.0.113.9:16000 is the ICE analogue of the mapped address.
• It names the path the media actually took.

Nothing may have answered a single check. Then the ICE section says so outright: ICE never completed, so the call has no media path.

• Those individual transactions also appear in the unanswered list below it.
• That is where sipnab reports the silence itself: one silence, stated once.

Scanners probe for open registrations and try credential stuffing. Detect them early and feed the IPs to fail2ban.

Detect scanners on a live interface. Append fail2ban-compatible lines to a log file fail2ban can watch.

• --kill-scanner additionally sends the kill response back to the scanner.
• So run it only where you mean to:

sudo sipnab -N -d eth0 --kill-scanner --fail2ban >> /var/log/sipnab/scanners.log

After the fact, match the known scanner User-Agents in a capture. This only reads the file, and sends nothing:

sipnab -N -I capture.pcap --filter "ua =~ 'friendly-scanner|sipcli|sipvicious'"

What to look for:

• known scanner fingerprints (friendly-scanner, sipvicious, sipcli)
• high REGISTER rates from a single source
• sequential extension enumeration (INVITE to 100, 101, 102, and upward)

Next steps:

• Count who the detectors would ban before fail2ban sees any of it.

  • On a carrier trunk the enumeration signature and a busy hunt group look alike.
  • The addresses at the top of this list are routinely your own SBCs:

  sipnab -N -I capture.pcap --kill-scanner --fail2ban \
    | grep -oE 'src=[^ ]+' | sort | uniq -c | sort -rn

  Put every address you recognize into the jail's ignoreip. Raise maxretry until the list holds only what you meant. A jail that bans your carrier takes the phone system down more thoroughly than the scanner would have.

  sipnab does that arithmetic for you and hands back the rule itself. --recommend-block groups the detections by source. It prints one block per accused address, with:

  • the evidence
  • the counter-evidence
  • a fail2ban, nftables or iptables rule

  sipnab -N -I capture.pcap --kill-scanner --recommend-block fail2ban

  sipnab recommends and does not apply.

  • It reaches no firewall, holds no credential and runs nothing.
  • The output is text you read first.
  • Read the COUNTER-EVIDENCE: line of every block before you run any of it.
  • An address that also completed a registration or a call is a working peer. For those:
    • The address-specific dialects arrive commented out.
    • The fail2ban dialect arrives with an ignoreip line already filled in.

• Then point fail2ban at the log file.

  • --fail2ban chooses the format and detects nothing on its own.
  • --kill-scanner produces scanner_detected lines. Add --kill-ua to it to match another User-Agent; sipnab refuses --kill-ua alone.
  • --reg-flood produces reg_flood lines.
  • Without one of those, the log stays empty while looking like an all-clear.
  • sipnab warns on stderr when you ask for the format with no scanner detector armed.

• For broader detection, combine flags: sudo sipnab -N -d eth0 --kill-scanner --fraud-detect --reg-flood --alert syslog

• Use --digest-leak to check if any endpoints are leaking credentials in cleartext.

--reg-flood counts a failure only when all of these hold:

• A registrar's 401 or 407 answers a REGISTER that already carried credentials.
• The answer is on the same transaction.
• It arrives before that transaction ends.

Three settings decide what it reports:

• --reg-flood-threshold ([security] reg_flood_threshold, default 50).
  • How many such failures from one source make a flood.
• --reg-flood-window ([security] reg_flood_window_secs, default 1 second).
  • How much capture time one count spans.
  • Raise it to catch a slow guessing run, such as one refused REGISTER every few seconds.
• --reg-flood-transaction-timeout ([security] reg_flood_transaction_timeout_ms, default 32000 ms).
  • How long a REGISTER stays open to its challenge.
  • sipnab treats a challenge that arrives later as a stray and never counts it.

These are not the --alert reg-flood:50/10s:5m rule. That rule sits on top.

• It counts the flood findings this detector files, per source.
• It decides when to raise an alert, and how long to stay quiet afterward.
• The settings above decide whether a finding exists at all.

Set the transaction timeout to your Timer F.

• RFC 3261 section 17.1.2.2 ends a non-INVITE transaction, REGISTER included, at Timer F, 64 times T1.
• Section 17.1.1.1 sets T1 to 500 ms by default.
• It recommends a larger value on links known to have a longer round trip.
• That is where the default 32 seconds comes from.

Pick the value this way:

• Your phones or SBC run T1 above 500 ms: set 64 x T1.
  • T1 at 1 s gives 64000.
  • T1 at 2 s gives 128000.
• You capture on the registrar side of a stateful proxy, and its final-response timer is longer than that: set the proxy's timer instead. The proxy keeps relaying the registrar's answer until that timer fires.
  • OpenSIPS: the tm module fr_timeout is in seconds (default 30), so multiply by 1000. Its T1_timer is in milliseconds (default 500).
  • Kamailio: the tm module fr_timer is already in milliseconds (default 30000).
    • Its retr_timer1 is T1 in milliseconds (default 500).
    • max_noninv_lifetime (default 32000) caps a non-INVITE transaction at Timer F.
• Neither applies: leave the default. Both proxies ship a 30-second final-response timer, inside the default 32 seconds.

fr_inv_timeout (OpenSIPS) and fr_inv_timer (Kamailio) govern INVITE transactions only. So they never apply to a REGISTER.

When the capture cannot show a failure. The detector counts the registrar's answers. A capture without them gives it nothing to count. At the end of the run --reg-flood then warns one of these:

• reg_flood cannot establish credential failures: the capture holds no answer to any REGISTER.
• reg_flood cannot establish the outcome of N of M credentialed REGISTER(s): some went unanswered inside the transaction timeout.

The MCP security_findings tool and GET /v1/security/findings carry the same statement in observation_gaps. It names no source and bans nothing.

Fix the capture first:

• Capture both directions of the registrar's traffic.
• Check that no capture filter keeps requests only.
• The answers may be present but slow. Then raise --reg-flood-transaction-timeout as described above.

Symptom of a timeout set too short:

• A registrar on a slow or congested link answers late.
• sipnab drops its 401s as strays.
• So a credential-stuffing run against it never reaches the threshold.

Symptom of one set too long: a stray challenge on a reused branch can settle a REGISTER that the client gave up on long ago.

The accepted range is 1000-600000 ms.

Is REGISTER rejected with 401 Unauthorized, 403 Forbidden, or 423 Interval Too Brief? Phones not registering means no inbound calls, and potentially no outbound.

sipnab -N -I capture.pcap --filter "method == 'REGISTER' AND state == 'Failed'" --json

For one line per registration rather than one per message, read the registration_failure finding.

• It answers this question directly.
• It also catches the failure that is not a rejection at all.
  • A registrar may grant less time than the phone asked for.
  • Then the phone re-registers far more often than it planned:

sipnab -N -I capture.pcap --json-dialogs --no-cli-print --quiet \
  | jq -c 'select(.signaling_diagnosis.registration_failure) |
           {call_id, from} + .signaling_diagnosis.registration_failure'

kind is rejected or shortened_expiry. On the second, compare requested_expiry_sec against granted_expiry_sec.

final_status_code on a REGISTER dialog is the registrar's final answer to the REGISTER. A challenge that was then answered does not count.

• So it shows that the registrar refused a registration.
• It does not show why it granted a short expiry.
• registration_failure names both.

What to look for:

Next steps:

• This sequence is healthy: a REGISTER gets 401, then a second REGISTER with credentials, then 200.
• Repeated 401s with no successful registration: the password or auth username is wrong.
• retransmits > 3 on REGISTERs: the registrar may be unreachable.

Three or more challenges with no 200 is exactly what signaling_diagnosis.auth_loop reports. Its kind separates the two causes:

• credential_failure: the phone answers and is re-challenged. The password is wrong.
• silent_drop: the phone never sends Authorization at all. It has none configured, or does not understand the challenge.

These are tasks rather than symptoms, so the cookbook carries them:

• A Markdown or JSON report of one call, for a ticket: recipe "Generate a call report (text / Markdown / JSON)". A failure count per response code: recipe "Find every failed call, grouped by response code".
• The call's audio as a WAV, to hear what the caller heard: recipe "Export RTP audio as WAV".
  • It lists the codecs sipnab decodes.
  • A call with two streams exports as one stereo file, with the first stream on the left channel.
• A capture read in the browser with nothing installed and nothing uploaded: recipe "Analyze a pcap without installing anything".

sipnab -d eth0 runs, exits cleanly, and reports No SIP traffic found. But you know the link carries calls, and tcpdump -i eth0 shows them.

Check what sipnab asked the kernel for.

• Run with SIPNAB_LOG=info.
• Read the Auto-generated BPF filter: line.
• Take that exact expression to tcpdump, against the same interface or against a capture from it:

tcpdump -r capture.pcap -nn 'portrange 5060-5061' | wc -l

Zero there means the filter never matched. The kernel drops those frames before sipnab sees them, so no counter, metric, NOT DECODED line or report shows them missing.

The usual cause is encapsulation.

• A port filter matches the outer headers only.
• So SIP inside a VLAN tag, QinQ, PPPoE or MPLS never matches one.

Confirm by comparing. Against the repository's 32-frame PPPoE sample, tests/pcap-samples/DTMFsipinfo.pcap, the plain filter above counts 0. The same filter behind the PPPoE keyword counts all 32:

tcpdump -r tests/pcap-samples/DTMFsipinfo.pcap -nn 'pppoes and portrange 5060-5061' | wc -l

If you passed your own filter (a positional expression or --bpf-file), that is the cause. sipnab uses your expression exactly as typed and never edits it.

Drop it and let sipnab generate one.

• The generated filter carries an arm for VLAN, QinQ, PPPoE, VLAN-over-PPPoE and one or two MPLS labels.
• That arm fires on Ethernet and on both Linux cooked headers alike.
• Omitting -d on Linux therefore costs no encapsulation coverage.

On raw IP from a tun device and on the two loopback headers, sipnab leaves the arm out. None of those link types carries a tag.

Two cases the generated filter still does not cover:

• SIP inside a UDP tunnel (GTP-U, VXLAN, GENEVE).
  • Off by default, and sipnab warns about it at startup.
  • BPF cannot reach the inner port. So the only way to cover it is to take the whole port.
  • Add --capture-tunnels, and size the buffer for the extra volume first (Tuning capture).
• SIP on a port outside --portrange. Widen it (see Start here).

The generated filter also admits RTP on any port.

• It does so untagged only, over IPv4 and IPv6 without extension headers.
• Media inside a VLAN tag or a tunnel needs a filter of your own that names it.

The generated filter also misses SIP inside an encapsulation and in an IPv4 header with options. Untagged IPv4 with options matches.

To confirm which of these you are looking at, run the generated filter and a plain portrange over the same capture with tcpdump -r, and compare the counts.

-I session.tgz reads every capture in the archive, the way -I dir/ reads a directory. Read the lines sipnab prints before the analysis starts. They say what came out of the archive and what did not:

Skipping 'session.tgz/lab/cell-trace.pcap': empty (0 bytes)
'session.tgz' unpacked (gzip > tar): 12 capture(s) to read, 1 member(s) not read, 1 directory entr(ies)
-I resolved to 12 capture file(s), 1 archive member(s) not read (each named above with its reason)

• not a capture (starts with ...): the member is not pcap or pcapng. The four bytes identify what it is. A text file starts with printable bytes.
• ZIP data, which sipnab does not unwrap (or 7-Zip, zstd, xz, bzip2, LZ4).
  • Unpack that member with the tool that made it.
  • Then point -I at what comes out.
• has link type N, which sipnab does not decode.
  • The member is a capture of something other than IP, such as an LTE or NR MAC log from a test handset.
  • Its frames appear under NOT DECODED.
  • Nothing else in the run changes.
• could not be read past this point or ends before its archive says it should: the archive stops early.
  • sipnab keeps what arrived.
  • The run ends INCOMPLETE RUN with exit status 1, because members past that point are in no report.
• decompressing reached the N-byte ceiling: the archive inflates past --max-gunzip-bytes. For an archive you trust, raise it.

The CLI reference covers the rest:

• how nesting works
• how frame pointers name a member
• where the unpacked copies live while the run reads them

A -L listener that shows no calls has one of two problems. The silence warning in its log already tells you which. The listener warns when it admits nothing for 30 seconds (--hep-silence-warn):

• no packets for 30s means nothing arrived at all. The sender is not reaching the port. Check:
  • the address and port it targets
  • the firewall
  • whether the SIP server is mirroring at all
• no packets admitted for 30s ... every one was refused means packets arrive and sipnab turns every one away.
  • The line names the reason most of them met, and the peer that sent them.
  • The network is fine, and the sender's configuration is not.

Some proxies may get through while one does not. Then no warning fires, because the listener as a whole is busy. Ask the roster instead:

• --hep-senders at the end of a headless run
• GET /v1/hep/senders
• the MCP hep_senders tool
• s from the TUI's capture-health panel (recipe "Find who feeds the collector and who it refuses")

A proxy that stopped shows as silent. A proxy that sends and gets refused has no sender row. Its address appears under refused_sources with the reason:

The roster names senders by the capture id each claims and the address it sends from. Give each sender its own id (--hep-id on a sipnab agent). Then two agents on one host stay apart.

The sender may itself be a sipnab (--hep-send). Then ask it too.

• Over TCP or TLS, a first connect that fails stops the sender's run at startup with exit status 2:

  Failed to create HEP sender: Failed to connect HEP sender to '127.0.0.1:19301' over TCP: Connection refused (os error 111)
  

  Start the collector, or correct the address, then start the sender again.

• Its runtime_stats or GET /v1/runtime carries hep_export.

• A headless run ends with a HEP export to ... line.

  • connect failures are reconnects to a collector that went down or became unreachable.
  • tls_handshake failures are a collector whose certificate the sender does not accept.

Over UDP the sender cannot tell.

• A collector that is down produces no failure.
• So the collector's roster is the only witness.

The plaintext legs may show while the proxy's TLS or WebSocket legs do not. Then look at the NOT DECODED line.

• OpenSIPS and Kamailio tracers name TLS and WebSocket legs over HEP with IP protocol numbers 22 and 50. sipnab 0.5.188 and later decode both. See How --hep-listen reads the transport.
• The collector may report no transport (IP protocol 22) or ESP not NULL-encrypted (IP protocol 50). Then it is older than 0.5.188: upgrade it.
• Any other number is one no tracer convention covers. Check the sender's configuration.

The calls are on port 5061, and sipnab reports nothing. Or it reports fewer messages than the phone did.

• Decryption has several independent ways to produce that one symptom.
• One log line separates most of them.

Command:

SIPNAB_LOG=info sipnab -N -I capture.pcapng --portrange 1-65535 --report

What to look for. One line decides where to go next:

TLS decryption active: 6 secret(s) from embedded DSB in capture.pcapng

Its presence means keys reached sipnab and were usable. Its absence means they did not, whatever else the run printed.

Widen the port range. This is the most common ending.

• Decryption and port filtering are separate steps.
• The second still applies afterwards.

sipnab says so rather than staying silent:

SIP outside --portrange 5060-5061 is being skipped: 1 message(s) so far,
in no count, no dialog, and no output. Busiest port(s): 15061 (1).

Re-run with --portrange 1-65535. A lab or a carrier trunk on a non-standard TLS port produces exactly this.

Three causes, in the order they are cheapest to rule out.

The binary has no tls feature. It is a build-time feature, not a runtime flag. A build without it ignores the whole subject:

sipnab --version

tls must appear in the feature list it prints. If it is absent, no flag helps. Rebuild with --features tls, or install a build that has it.

The keylog covers different sessions.

• Keys only exist for handshakes the producer was running for.
• A keylog started after the INVITE cannot decrypt that call.
• Neither can one from a different host.
• Compare the line count against the number of TLS sessions in the capture before you blame the file.

The producer is a pipe and nothing is reading it yet. See A FIFO keylog, and capture appears to hang.

--keylog accepts a named pipe, so the keys never touch a disk (recipe 7f in the cookbook). sipnab opens the pipe without waiting for a writer and reads it as a stream. Check that it recognized the pipe:

SIPNAB_LOG=info sipnab -N -I capture.pcap --keylog kl.fifo --report 2>&1 | grep FIFO

2026-10-16T11:52:15.421081Z  INFO sipnab::app::bootstrap: TLS decryption: kl.fifo is a FIFO, reading it as a live stream

That line means the capture does not wait for the producer. If it is present and no keys arrive, the producer is not writing to that pipe: check the path you gave the producer.

If the line is absent and the run stalls until something writes to the pipe, or loads no keys from a pipe that has them, the binary is older than the FIFO support. Install the current release, then run the command again.

The producer rotated or truncated the file. sipnab detects both and reloads the file from the beginning, with one warning that names the file and ends in one of these:

was truncated by its producer — reloading it from the beginning
was replaced by its producer — reloading it from the beginning

Look for that warning first. If there is no warning and no new keys, the producer is not writing to the path you passed: check its configuration.

couldn't add a ingress filter to interface 1: netlink receive: no such file
or directory

That is the kernel, not eCapture. -m pcap attaches a TC classifier to the interface as well as its uprobes. So it needs CONFIG_NET_CLS_BPF:

zcat /proc/config.gz | grep NET_CLS_BPF

# CONFIG_NET_CLS_BPF is not set means this mode cannot run on that kernel.

• Use -m keylog instead (cookbook 7e).
• It needs only the uprobes, and runs there.

A missing /sys/kernel/btf/vmlinux is not a reason to expect failure. eCapture falls back to its own non-CO-RE bytecode. It logs which file it loaded.

--export-vcon may fail in one of three ways. Each has one usual cause.

• It refuses to run.
• It cannot find the call.
• It writes a container with no audio in it.

--export-vcon needs the 'vcon' Cargo feature, which this build does not carry. Rebuild with --features vcon (or --features full); `sipnab --version` lists the features a binary was built with

The run stops with exit status 2. The exporter is behind a non-default Cargo feature, so a default build does not carry it. Check what you have:

sipnab --version        # the feature list is on the version line

If vcon is absent, build one that has it:

cargo build --release --features vcon

Call-ID 'x@y' not found in tracked dialogs, so there is no dialog to export. --report lists the Call-IDs this run holds.

The run exits with status 1. It holds no dialog under that Call-ID. --report lists the ones it does hold.

• The usual cause is a Call-ID copied with surrounding quotes, or a truncated one.
• The second is a capture that never saw the dialog open.

Quote the Call-ID in the shell. Most contain @. Many contain characters your shell would otherwise interpret.

The recording dialog object carries no body. Read the media_note in the container. It states what the run kept, and is never a finding that the call was silent:

grep -o '"media_note": "[^"]*"' call.vcon.json

"media_note": "No audio payload retained: sipnab measured 425 RTP packet(s) of PCMU on 1 decodable stream, but kept none of their payload, so there is nothing to decode. This is a statement about what this run kept, not a finding that the call was silent. Most often audio payload retention was off — start the server with --retain-audio to hold payload for export. With retention already on, the other causes are a --snaplen that truncated the payload away before sipnab saw it, or packets that carried no payload at all."

Export again with --retain-audio:

sipnab -N -I capture.pcap --retain-audio --export-vcon 'abc123@host' --vcon-out call.vcon.json --no-cli-print

Then check that the recording now has a body:

jq -c '.dialog[] | {type, has_body: has("body"), mediatype}' call.vcon.json

{"type":"recording","has_body":true,"mediatype":"audio/x-wav"}

Audio also has an inline size budget, --vcon-max-inline-media (default 5 MiB). sipnab refuses a recording over it rather than truncating it, and the caveat names the size it turned down. Raise the budget only if the system that receives the container accepts larger bodies.

Problem: one of these happens:

• A root run with -O, --export-vcon-dir, --vcon-out or --redact-map stops at startup with exit code 1.
• A long run stops when --split rotates.

It stops with a line such as:

Failed to create output file '/var/tmp/e2e/rtp03.pcap': Permission denied (os error 13). sipnab dropped privileges to user 'nobody' before opening output; make the directory writable by that user (e.g. install -d -o nobody <dir>), pass --user <name> for a user that can write there, or --no-priv-drop.

Why: run as root, sipnab opens the capture device and then drops privileges.

• It drops to nobody, unless --user or [privilege] user names another account.
• It creates the -O file after the drop, and each --split file later still.
• So the drop user creates the file, not root.
• sipnab checks right after the drop that the drop user can create each of those outputs.
• So the refusal comes before the first packet, not when the file is first needed.

A directory made with a plain sudo mkdir is root:root mode 0755. nobody may read it but not write it.

Command: pick one.

Give the directory to the drop user:

sudo install -d -o nobody /var/tmp/e2e

Or run as an account that can already write there (sipcap is an example name):

sudo sipnab -N -d any -O /var/tmp/e2e/rtp03.pcap --user sipcap udp

Or keep root for the whole run:

sudo sipnab -N -d any -O /var/tmp/e2e/rtp03.pcap --no-priv-drop udp

What to look for: the text after the file name is the operating system's reason.

• Permission denied is the privilege drop above.
• No such file or directory means the directory does not exist. sipnab does not create it, so make it first.
• The privilege sentence appears only when sipnab did drop privileges.
  • A run without it may still say Permission denied.
  • That means your own account may not write there.

When sipnab hits a bug it cannot recover from, it stops. It prints two lines to the terminal:

sipnab panicked at src/main.rs:152:9:
crash report written to /home/you/.local/state/sipnab/sipnab-crash-20260923-191230-2507858-0.log

That file is what we need. You do not have to reproduce the crash or install anything extra.

• Find the report. It is the path on the crash report written to line.
  • By default reports land in ~/.local/state/sipnab/.
  • Each name has the form sipnab-crash-<date>-<time>-<pid>-<n>.log.
  • A report_dir in the [crash] config section moves them.
  • If you closed the terminal, list the newest one:

ls -t ~/.local/state/sipnab/sipnab-crash-*.log | head -1

• Check what it contains before you send it. The report is plain text. It holds:

  • the panic message
  • the source line
  • the thread name
  • the version
  • the build ID of the executable
  • its load address
  • one raw address per stack frame, up to 256 frames

  It contains no capture data and no packets. But:

  • The panic message can quote a value sipnab was handling.
  • The path names your home directory.
  • Read it and redact what you need to.

• Send it. Open an issue.

  • Attach the .log file.
  • Add the output of sipnab --version and the command you ran.

A core dump is not needed, and it is not safe to share.

• A core holds the process's memory.
• That memory includes the SIP messages and media it had captured.
• Only turn on core = true if we ask for a core. Then keep it on your side.

The release strips every sipnab binary it publishes.

• So the backtrace in the report cannot name a function.
• Most of its lines read <unknown>.
• The Image: and Raw frames sections are what make it readable anyway:

Image:
  Path:      /usr/bin/sipnab
  Build ID:  edbd855707e15a8b69f6e0fb3f0ac2bf925c23b1
  Load base: 0xaaaac4090000
  Target:    aarch64-unknown-linux-gnu

Raw frames (runtime return address, then image+address of the call in
that image's file. Resolve these against the symbol file named by
the ID above):
    0  0xaaaac4a904d4  sipnab+0xa004d3
    1  0xaaaac4a8ffe8  sipnab+0x9fffe7

Each release publishes a symbol file for every binary it ships, built by the same compile:

• sipnab-<version>-<target>.debug for Linux
• sipnab-<version>-<target>.dSYM.zip for macOS
• The -noaudio binaries inside the -noaudio .deb and .rpm packages have their own, with -noaudio in the name.

The Build ID (Linux) or UUID (macOS) line pairs the report with exactly one of them. So a rebuild cannot stand in for it.

To resolve a report yourself:

• Download the symbol file named by the report's version and Target: line from the release page.
• Name it once in a shell variable, here for an aarch64 glibc report. Then the commands below paste as they are:

DEBUG=$(ls sipnab-*-aarch64-unknown-linux-gnu.debug)

Check that it is the right one:

readelf -n "$DEBUG" | grep 'Build ID'

The ID must equal the report's Build ID line.

Then pass each sipnab+0x… address to addr2line. The two below are the first frames of the example report above:

addr2line -f -C -i -e "$DEBUG" 0xa004d3 0x9fffe7

Or to llvm-symbolizer, if you have LLVM:

llvm-symbolizer --obj="$DEBUG" 0xa004d3 0x9fffe7

Each address prints the function and the src/…rs:line it came from.

• The numbers after + are addresses inside the file.
• So they work however the system placed the binary in memory.
• They point at the call instruction, one byte before the return address in the first column.
• So each frame names the line that made the call.

On macOS:

• Unzip the .dSYM.zip.
• Compare dwarfdump --uuid sipnab.dSYM with the report's UUID line.
• Pass the numbers after sipnab+ to atos.
  • On macOS they are addresses in the binary's own layout, which starts at 0x100000000.
  • So they look larger than the Linux ones.
• Put them in ADDRS, separated by spaces:

atos -o sipnab.dSYM/Contents/Resources/DWARF/sipnab $ADDRS

For gdb or a core file on Linux, put the .debug file in the same directory as the sipnab binary.

• The binary's .gnu_debuglink names that file.
• gdb loads it from there without further setup.

Build custom queries with the Filter DSL. It offers:

• 42 fields plus any header by name
• named filters
• regex support
• boolean logic

See the CLI Reference for every flag and more recipes.

The capture itself may be the problem. For example:

• drops on a busy link
• a full kernel ring buffer
• loss that appears on every call at once

Then see Tuning capture on a busy server.