Troubleshooting - NormB/sipnab GitHub Wiki

Under pressure? Each scenario below has four parts:

  • Problem
  • Command
  • What to look for
  • Next steps

Copy-paste and go.

Find your symptom

Match the complaint, not what you think is wrong. The section works back to the cause from there.

What the caller says Section
"It just says the number is unavailable" / call never connects Failed calls
"It rings and rings, then nothing" / 408, or no response at all Nothing came back
"It hangs up after 30 seconds" / after exactly 15 or 30 minutes Dropped calls
"Calls to that one carrier fail immediately" / 488 488 Not Acceptable Here
"They can hear me but I can't hear them" One-way audio
"Dead air" / "the line went quiet" but the call stayed up Dead air and clipping
"It sounds choppy / robotic / underwater" Poor call quality
Every call in the capture looks lossy at once Poor call quality, then Tuning capture -- the loss may be sipnab's, not the network's
"There's a long pause before it rings" Slow call setup
"Audio dies as soon as they answer" / works internally, not externally NAT traversal issues
"The phones keep dropping off" / no inbound calls arrive Registration failures
"Something is hammering the PBX" / probes across unknown extensions SIP scanner detection
"No SIP traffic found" on a link you know carries calls A live capture that sees nothing
A HEP collector (-L) shows no calls, or one proxy's calls are missing A HEP collector that receives nothing
The two legs of a call through OpenSIPS topology_hiding("C") do not join Troubleshooting the OpenSIPS mapping, or Decode the hidden Call-ID
SIP is over TLS and the calls never appear Encrypted SIP that does not decrypt
Somebody handed you a .tgz or .tar of captures, and the run looks short An archive of captures
--export-vcon refuses, cannot find the call, or writes no audio A vCon export that fails
Failed to create output file for a -O path, or for a --split file mid-run An output file sipnab cannot create
A terminal interface still runs, and uses CPU, after its window or tmux session has gone A terminal interface that outlives its terminal
sipnab --vcon-fetch -, or a run given a pipe with -I, keeps running after SIGTERM A run that ignores SIGTERM while it waits for input
sipnab itself stopped with "sipnab panicked at" Send us a crash report
sipnab refused to start, or stopped with an error message Error messages: look up the message by its text
The systemd service does not start, or stops and stays down A service that does not start
Permission denied on 'eth0' on a live capture Live capture says Permission denied
A run stops with Permission denied on a key, token, certificate or revocation file A key file sipnab cannot read
A client of the REST API, MCP or metrics gets 401, 403, 404 or 503 A client the listener turns away
A client or HEP sender fails the TLS handshake A TLS handshake that fails
A setting in sipnab.toml has no effect A setting that has no effect
The disk fills, or memory grows, on a long run A long run that fills the disk or the memory
Nothing yet -- you have a capture and a complaint Start here

Whatever the symptom, three things decide whether the answer is in the capture at all:

  • The capture covers the right time window.
  • It comes from a point that sees both directions.
  • --portrange covers the ports in use.

Start here deals with the third.

Start here: one pass over everything

Don't know which problem you have yet? The --problems alias surfaces every troubled call in a single sweep. That means failed calls plus:

  • one-way audio
  • high loss/jitter
  • NAT mismatches
  • retransmit storms
  • slow setup
  • media asymmetries
# Every problematic call
sipnab -N -I capture.pcap --problems --json

--problems is shorthand for the full Filter-DSL expression state == 'Failed' OR one_way == true OR rtp.loss > 5.0 OR rtp.jitter > 50.0 OR nat_mismatch == true OR retransmits > 3 OR pdd > 11.0 OR codec_asymmetry == true OR ptime_asymmetry == true OR payload_asymmetry == true OR duration_asymmetry == true OR late_media == true, with the default thresholds. So it is a superset of "failed calls".

Once you know the symptom, jump to the matching section below for the precise filter.

First, check you are reading the whole capture.

  • --portrange defaults to 5060-5061.
  • sipnab skips any SIP message whose source and destination ports both fall outside it.
  • Skipped messages reach no count, no dialog and no output.
  • So a call carried on 5070 or 5080 is simply not in any answer below. Those ports are ordinary on carrier trunks and SBCs.

sipnab says so on stderr and again at the end of the run:

NOT ANALYZED: 1 further SIP message(s) were seen on ports outside --portrange
and are in none of the totals above. Busiest: 8090 (1).

You may see that line, or the call you are chasing may be missing. Then add --portrange 1-65535 to every command on this page and start again.

  • On a live capture the range becomes the kernel BPF filter.
  • So the traffic never reaches sipnab, and nothing reports it missing.
  • Set the range before you capture.

Second, check sipnab could read the capture at all.

  • Some frames are on a link type, EtherType or IP protocol sipnab has no decoder for. Such a frame produces nothing.
  • It counts as a packet (it arrived).
  • It contributes to no message, dialog or stream.

sipnab reports those separately, with the numbers that name them:

NOT DECODED: 3 of 3 frame(s) (100.0%) produced nothing and are in none of
the counts above. Reasons: unsupported link type 147 (3). NOTHING IN THIS
CAPTURE WAS READ โ€” every frame failed to decode, so the totals above
describe no traffic whatsoever and a zero among them is not evidence of
absence.

This line shows the difference between two cases:

  • there is no SIP in this capture
  • I could not read one single frame of this

The totals alone render both cases identically. When the share is high:

  • Treat every zero on this page as unknown, not as a finding.
  • Fix the decode first.

The number in each reason is what you act on:

Reason What it means What to do
unsupported link type N The pcap's DLT has no decoder here. Link types lists the ones sipnab reads, and 147 to 162 are the private-use DLT_USER range. Convert it: editcap -T ether in.pcap out.pcap. If the link type is one sipnab should read, open an issue naming the number.
not IP (EtherType 0xNNNN) The frame decoded and carried no IP. 0x0806 is ARP, 0x8847 MPLS, 0x88CC LLDP. ARP and LLDP are ordinary background -- expect a few on any Ethernet capture. A large MPLS or PPPoE share means the mirror is giving you the encapsulated form.
ESP not NULL-encrypted (IP protocol 50) IPsec ESP whose payload failed the checks that prove NULL encryption, so it carries real encryption, or a protection sipnab cannot read. sipnab takes no ESP keys. Take the capture inside the tunnel, or run the lab with NULL encryption, which sipnab reads. See Tunnels above the link layer.
no transport (IP protocol N) IP decoded, and its payload is no transport sipnab handles. 2 is IGMP, 89 OSPF, 103 PIM. Routing and multicast control traffic is ordinary background. A large share of another number means a tunnel sipnab does not strip.
truncated frame The frame is shorter than a header it declares. Raise --snaplen on the capture, or re-take it.
decode error The decoder rejected the bytes outright. Usually a corrupt or mis-declared file; try editcap or tshark -r on it.

Failed calls

Are calls rejected with 403 Forbidden, 404 Not Found, 486 Busy Here or 488 Not Acceptable Here? Or do they time out with 408 Request Timeout? Find every call that never established. Then triage by response code.

List every failed call, one line per response message. Each line carries:

  • the Call-ID
  • the response code
  • the reason text
sipnab -N -I capture.pcap --filter "state == 'Failed'" --json \
  | jq -c 'select(.is_request == false) | {call_id, status_code, reason}'

You should see one line per response message of each failed call (minimal example):

{"call_id":"abc123@host","status_code":100,"reason":"Trying"}
{"call_id":"abc123@host","status_code":486,"reason":"Busy Here"}
{"call_id":"def456@host","status_code":408,"reason":"Request Timeout"}

One of those Call-IDs may be worth escalating. Then write the detailed Markdown report for that single call and attach it to a ticket.

  • Substitute the Call-ID you picked.
  • The redirect overwrites report.md in the current directory.

--no-cli-print is not optional here.

  • Without it, sipnab dumps every message in the capture ahead of the report.
  • The file you attach then opens with hundreds of lines of raw SIP.
sipnab -N -I capture.pcap --call-report "abc123@host" --markdown --no-cli-print > report.md

What to look for: sipnab includes response code intelligence. The status field tells you why:

Code Meaning Typical fix
401/407 Authentication required Check credentials, realm mismatch, nonce expiry
403 Forbidden ACL/IP allowlist, registration required, call barring
404 Not found Bad dial plan, missing route, number not provisioned
408 Request timeout Endpoint unreachable, DNS failure, firewall
486 Busy here Endpoint occupied, no call waiting
488 Not acceptable here Codec mismatch, SDP incompatibility -- full recipe below
503 Service unavailable Upstream overload, trunk down, proxy crash

Next steps:

  • The response code may be 408, or you may see high retransmits. Then the problem is network-level.
    • Check connectivity and firewall rules before touching SIP config.
    • Nothing came back turns that "network-level" guess into evidence.

Nothing came back (408, or silence): Ask the network

A 408 and a call that simply stops both look like "the far end did not answer". Usually that is an inference from silence. Sometimes it is not.

  • A router or the host itself may have sent an ICMP error quoting your request.
  • Then the network stated the cause, and sipnab reads it.

Any run that saw such an error ends with a capture-wide tally on stderr. So you do not have to go looking for it. A run that saw none prints nothing:

ICMP: 4 error(s) quoting a SIP request, naming 2 unreachable endpoint(s).
Busiest: 192.0.2.10:5060 (2, port unreachable), 192.0.2.11:5080 (2, host unreachable).

When you see that line, pull the affected calls out as one object each. Then read the finding:

sipnab -N -I capture.pcap --json-dialogs --no-cli-print --quiet \
  | jq -c 'select(.signaling_diagnosis.icmp_unreachable) |
           {call_id, method,
            code: (.final_status_code // .signaling_diagnosis.final_failure.code),
            icmp: .signaling_diagnosis.icmp_unreachable}'

Then read one of them in full. The report:

  • lays the ICMP finding beside the SIP timeline
  • names the messages it came from
sipnab -N -I capture.pcap --call-report 'abc123@host' --no-cli-print
Signaling Issues:
  - Final failure: 408 Request Timeout
    evidence: #2 408 Request Timeout
  - ICMP port unreachable: 192.0.2.10:5060 unreachable (2 times), reported by 198.51.100.1
    evidence: #0 OPTIONS, #1 OPTIONS

What to look for:

Field What it tells you
description The network's own words -- port unreachable, host unreachable, administratively prohibited. Each points somewhere different.
unreachable_endpoint The socket that did not answer. This is the machine to go and look at.
reported_by The device that noticed and said so -- usually a working router in the path. Do not send an engineer here.
errors Exact count of such errors for this call. Two or three says the peer was consistently unreachable, not momentarily busy.

Next steps:

  • port unreachable means the host is up and nothing is listening on that port.
    • Check that the SIP service is running and bound where you think.
    • Check that the port in the Contact/Via matches what it actually listens on.
    • This is not a network fault.
  • host unreachable or network unreachable is a routing problem short of the destination.
    • The reporter names the hop that gave up.
    • Nothing reached the host, so the capture says nothing about its ports.
  • administratively prohibited (v4 codes 9, 10, 13; v6 type 1 code 1) is a firewall or router ACL, not a dead host.
    • The peer may be perfectly healthy and answering everyone else.
    • The fix is on the filtering device.
    • That is a different team from the one you call about an unreachable host.
    • Count the two kinds separately before you dispatch anyone. One capture can hold both.
  • No ICMP at all does not clear the network.
    • Most firewalls drop ICMP errors outright.
    • The summary can only report what the capture holds.

A call may show both retransmissions and icmp_unreachable. Read them together rather than as two problems.

  • The retransmission count is how hard the sender tried.
  • The ICMP error is why nothing came back.

sipnab annotates the retransmission finding with icmp_cause.

  • It stops offering its own guess at the reason.
  • It keeps the count. "11 OPTIONS over 300 s" and "3 INVITEs over 3 s" are the same cause. They are a very different operational picture.

There is no icmp field in the Filter DSL. So select these calls with jq on --json-dialogs as above, not with --filter.


Dropped calls (call answers, then disconnects mid-conversation)

The call sets up fine and both sides talk. Then it dies partway through. This often happens after a suspiciously round number of minutes.

Enumerate the calls that established and then ended early. These calls completed, but were far shorter than a real conversation:

sipnab -N -I capture.pcap --filter "duration < 120.0 AND state == 'Completed'" --json \
  | jq -r '.call_id' | sort -u

Then print the timeline for one of those calls. It shows who sent the BYE, and exactly when. Substitute a Call-ID from the list above for abc123@host:

sipnab -N -I capture.pcap --call-report 'abc123@host' --no-cli-print

The call report shows the full SIP message timeline. It also shows per-stream RTP stats, including first/last packet timestamps. Match the signature:

  • BYE at a round interval after answer: RFC 4028 session-timer expiry.
    • The interval is exactly 15 min, 30 min or 1 h. For example: 200 OK at 14:00:02, BYE at 14:30:02.
    • One side never sent (or never received) the session refresh re-INVITE/UPDATE.
    • It tore the call down when Session-Expires ran out.
  • RTP last packet well before the BYE: a NAT/firewall idle timeout silently dropped the media path.
    • The stream's last_seen is minutes earlier than the BYE.
    • The endpoint's RTP-timeout watchdog eventually hung up.
  • BYE from the carrier side, accompanied by SIP retransmits: trunk-side reset, or an upstream element recycling the session.

Next steps:

  • Session timer:
    • Align Min-SE / Session-Expires between PBX and trunk.
    • Confirm the negotiated refresher actually sends the refresh before expiry.
  • NAT/firewall:
    • Enable RTP keepalives (or comfort noise) on the endpoints.
    • Raise the firewall's UDP session timeout above the keepalive interval.
  • Disable SIP ALG on intermediate NAT devices. It corrupts dialogs in ways that surface as mid-call drops.

488 Not Acceptable Here (codec mismatch)

An INVITE comes back 488 Not Acceptable Here. The callee (or an SBC in the path) found no common codec between the SDP offer and what it supports.

Find every 488 rejection in the capture:

sipnab -N -I capture.pcap --filter "state == 'Failed'" --json \
  | jq -c 'select(.status_code == 488) | {call_id, status_code, reason}'

You should see:

{"call_id":"abc123@host","status_code":488,"reason":"Not Acceptable Here"}

Then compare the SDP offer against the answer for one of those calls. Substitute its Call-ID for abc123@host:

sipnab -N -I capture.pcap --call-report 'abc123@host' --no-cli-print

The call report's SDP timeline lists each offer/answer with its codec set.

  • A 488 means there is no overlap.
  • For example, the offer carries only G729, while the callee's profile allows only PCMU, PCMA.
  • The reject may come from an SBC. Then the offer may never have reached the far end at all.

Next steps: Do one of these, so the offer and answer share at least one codec:

  • Enable transcoding on the SBC/media server.
  • Add the missing codec to the endpoint/trunk profile.

One-way audio

One direction of RTP has zero packets. The caller can hear the callee (or vice versa) but not both.

The --one-way alias finds every call flagged for one-way audio. It dumps the matching SIP messages as NDJSON:

sipnab -N -I capture.pcap --one-way --json

You should see one NDJSON record per SIP message of each flagged call. Each record carries the message's full field set. To read only the fields that name each message and its call, select them with jq. Against tests/pcap-samples/sip-rtp-g711.pcap:

sipnab -N -I tests/pcap-samples/sip-rtp-g711.pcap --one-way --json | jq -c '{is_request, method, status_code, call_id, from, to}'
{"is_request":true,"method":"BYE","status_code":null,"call_id":"[email protected]","from":"test <sip:[email protected]:5060>;tag=QvN92t713vSZK","to":"\"PCMU/8000\" <sip:[email protected]:5060>;tag=1"}
{"is_request":false,"method":null,"status_code":200,"call_id":"[email protected]","from":"test <sip:[email protected]:5060>;tag=QvN92t713vSZK","to":"\"PCMU/8000\" <sip:[email protected]:5060>;tag=1"}

For the same set of calls rendered as the full diagnostic report rather than raw records:

sipnab -N -I capture.pcap --one-way --report

--one-way is shorthand for a Filter-DSL condition. To combine it with others, write that condition out:

sipnab -N -I capture.pcap --filter "one_way == true" --json

Get the diagnosis detail (one_way_audio, nat_mismatch, hints) per call with --call-report <call-id>.

First: check whether the network already answered

Before reasoning about NAT, look at the end of the run.

  • Something may have sent an ICMP error about the media itself.
  • Then the network has already stated the cause, and there is nothing to infer:
ICMP: 27 error(s) quoting non-SIP traffic, 27 of them media, across 6 flow(s). Attributed to a stream or SDP endpoint: 27; matched nothing this capture holds: 0.
  ICMP port unreachable: RTP (payload type 0) from 192.0.2.5:42180 to 198.51.100.9:21750 could not be delivered (11 times), reported by 198.51.100.9. This is one of the media streams in this capture (1 call(s) affected). Audio sent that way is discarded before it arrives, which is heard as one-way or missing audio. The host answered, so it is reachable -- nothing was listening on that port. Check the service and the address it binds, not the network.

sipnab ties each media ICMP error to a call by trying these matches in order, and each line names the one that succeeded:

  • the quoted datagram's own 5-tuple against a tracked stream
  • the SSRC inside the quote
  • either socket against a tracked stream endpoint
  • an SDP-advertised media address, or the RTCP port one above it (RFC 3550 Section 11)

The first two identify the stream itself. The last two identify an address the stream used, or one the SDP offered.

A quote that matches nothing is still counted and printed, under matched nothing this capture holds. The endpoint it names is real. Widen the capture to include that endpoint's traffic, then read the run again.

What to look for:

  • ICMP errors against the media ports (above).
    • This is a stated cause, not an inference.
    • Read these before anything else in this section.
  • nat_mismatch == true alongside one_way.
    • RTP reached the capture point from an address no SDP in the dialog advertised.
    • That is what a NAT rewriting the media source looks like.
    • This is the most common cause.
  • Codec asymmetry in the SDP offer/answer. One side offers a codec the other doesn't support.
  • RTP ports in the SDP that never receive traffic. A firewall is blocking the return path.

Next steps:

  • The run may have printed a media ICMP line. Then act on that first.
    • It names the socket that rejected the audio and the device that reported it.
    • Nothing below can improve on a router's own statement.
  • Check NAT: sipnab -N -I capture.pcap --filter "one_way == true AND nat_mismatch == true" --json
  • If NAT is the cause: enable fix_nated_contact / fix_nated_register on the proxy, or deploy a TURN server.
  • If NAT is clean:
    • Verify symmetric RTP.
    • Check for SIP ALG on intermediate firewalls (disable it).
    • Confirm both endpoints negotiate a common codec.

Dead air and clipping

The hardest silent call to find is the one where nothing is wrong with the packets.

  • A gateway may send full-rate frames of digital silence.
  • It puts every packet on the wire, in sequence, at the right rate, in both directions.
  • Loss is zero and jitter is zero.
  • The MOS is 4.36.
  • No other check on this page fires.

Counting comfort-noise frames does not find it either. Those frames say "I am sending silence", and this gateway is not saying anything.

So sipnab measures the audio itself. It needs the samples, which means --retain-audio. Set CALL_ID to the call that went quiet, then run:

sipnab -N -I capture.pcap --retain-audio --call-report "$CALL_ID" --json --no-cli-print

For a clean call, here the first call in the repository sample tests/pcap-samples/sip-rtp-g711.pcap, the report's diagnosis.amplitude object reads:

{
  "streams_measured": 1,
  "dead_air": false,
  "clipping": false,
  "streams": [
    {
      "ssrc": "0x343da99b",
      "src": "10.0.2.15:27942",
      "dst": "10.0.2.20:6000",
      "codec": "PCMU",
      "report": {
        "sample_rate": 8000,
        "duration_ms": 8500,
        "full_scale": 32124,
        "floor_dbfs": -60.0,
        "window_ms": 100,
        "min_dead_air_ms": 1000,
        "clip_threshold": 31802,
        "min_clip_run": 3,
        "dead_air": [],
        "dead_air_ms": 0,
        "clipping": [],
        "clipped_samples": 0
      }
    }
  ]
}

The diagnosis.amplitude object appears only on a run that kept the samples. Absent means not measured, never "clean". The same command without --retain-audio prints null.

There are two measurements. Each has the threshold that produced it beside the number:

Finding What it measures Threshold
dead_air Windows whose RMS sits below a floor relative to the codec's own full scale, for at least 1,000 milliseconds in a row -60 dBFS, measured over 100 ms windows
clipping Runs of at least 3 consecutive samples at 99% of the codec's full scale or beyond The codec's ceiling, not the 16-bit container's: 32,124 for G.711 mu-law, 32,256 for A-law

Both are amplitude measurements, not quality scores. They report what the samples did, and they do not change the MOS.

Reading the result.

  • Dead air with a start time near the answer points at the media path.
  • Dead air starting mid-call, with packets still flowing both ways, is the gateway or the far-end mixer.
  • Clipping is a gain stage.
    • Something upstream is driving the encoder into its ceiling.
    • No network change fixes it.

sipnab_diagnosis_total{kind="dead_air"} and {kind="clipping"} count calls where the measurement fired. Neither ticks on a run that kept no audio.

Poor call quality

MOS below 3.0 means quality degradation users notice. Below 2.5, calls are unusable.

Find the calls in a capture whose MOS fell below the noticeable-degradation line:

sipnab -N -I capture.pcap --filter "rtp.mos < 3.0" --json

You should see the SIP messages of every matching call as NDJSON, one record per message. To list only which messages and calls matched, select those fields with jq. Against tests/pcap-samples/Asterisk_ZFONE_XLITE.pcap:

sipnab -N -I tests/pcap-samples/Asterisk_ZFONE_XLITE.pcap --filter "rtp.mos < 3.0" --json | jq -c '{is_request, method, status_code, call_id}'
{"is_request":true,"method":"INVITE","status_code":null,"call_id":"ZDYzOWVlNjEwM2NjZTBjNzliNmM1ZTNiOGZjNWFhN2E."}
{"is_request":false,"method":null,"status_code":200,"call_id":"ZDYzOWVlNjEwM2NjZTBjNzliNmM1ZTNiOGZjNWFhN2E."}
{"is_request":true,"method":"ACK","status_code":null,"call_id":"ZDYzOWVlNjEwM2NjZTBjNzliNmM1ZTNiOGZjNWFhN2E."}
{"is_request":true,"method":"BYE","status_code":null,"call_id":"ZDYzOWVlNjEwM2NjZTBjNzliNmM1ZTNiOGZjNWFhN2E."}
{"is_request":false,"method":null,"status_code":200,"call_id":"ZDYzOWVlNjEwM2NjZTBjNzliNmM1ZTNiOGZjNWFhN2E."}

The per-message records identify which calls are bad. Pull the per-stream numbers (jitter_ms, loss_pct, quality intervals) with --call-report <call-id>.

To watch a live interface instead:

  • Widen the same filter to jitter spikes and let it run.
  • Each matching call prints as it happens.
  • Live capture needs raw-socket access, hence sudo:
sudo sipnab -N -d eth0 --filter "rtp.mos < 3.0 OR rtp.jitter > 50" --json

What to look for:

Metric Threshold Likely cause
rtp.mos < 3.0 Quality degradation Aggregated impairment -- check jitter and loss
rtp.jitter > 30ms Congestion or buffering Network saturation, Wi-Fi, VPN overhead
rtp.loss > 2% Packet drops Overloaded links, QoS misconfiguration, carrier issue

Next steps:

  • Jitter is high but loss is low: the problem is buffering or path instability. Check for Wi-Fi hops, VPN tunnels, or missing QoS marking.
  • Loss is high:
    • First rule out sipnab itself.
    • Then run a path MTR/traceroute to find where packets are dropping.
  • The run logs a warning that starts IMPOSSIBLE RATE:: sipnab read the same traffic twice, such as a capture and its copy in one -I directory. Measure call quality quotes the warning in full. Every count and rate in that run is too high, the loss and MOS included. Read one copy.
  • To see when in the call the quality changed, and what the far end reported over RTCP, see Measure call quality.

Before you chase high loss on the network, check whether sipnab dropped the packets.

  • A capture may lose packets to a full kernel ring buffer.
  • It then reports the missing RTP as network loss.
  • The numbers look identical, and the fix is on the wrong machine.

sipnab warns on the first drop. It prints a summary at the end of a live capture.

  • You may see that warning.
  • Or loss may look implausibly high across every call at once.
  • In either case, the figure is measuring your capture rather than the call.

Tuning capture on a busy server covers:

  • how to read the two drop counters
  • what each one means
  • what to change

This applies to live capture only.

  • An offline -I read of an existing pcap loses nothing.
  • The pcap itself may have lost detail in capture, by whatever wrote it.

Deep-dive with stream detail

In the TUI, open the Stream Detail view:

  • Navigate to a call's flow view, press F6 to show its RTP streams, and press Enter on a stream's row.
  • Or press r to jump to the streams list, then Enter on a stream.

This view shows:

  • MOS and jitter sparklines.
    • These are visual trend graphs across the stream's lifetime.
    • They make it easy to spot the exact moment quality degraded.
  • Quality intervals.
    • This is a per-interval breakdown of MOS, jitter, and loss.
    • You can correlate degradation with specific time windows.
  • Burst/gap analysis (RFC 3611).
    • It distinguishes between bursty loss (congestion events) and gap loss (steady-state impairment).
    • Bursty loss points to queue overflow.
    • Gap loss points to a consistently lossy link.
  • Silence detection.
    • It identifies periods where no RTP was flowing.
    • These can indicate hold events, codec DTX, or network black holes.

This same data is available in the browser analyzer at sipnab.com/analyze/ under the Streams tab.


Slow call setup (post-dial delay)

PDD over 3 seconds is perceptible to users. Over 5 seconds and they'll hang up.

Find the calls whose post-dial delay crossed the 3-second perceptibility line:

sipnab -N -I capture.pcap --filter "pdd > 3.0" --json

The --slow-setup alias asks a looser question than that filter.

  • It expands to pdd > 11.0, sipnab's own post-dial-delay threshold.
  • That threshold comes from ITU-T E.721's target for an international connection.
  • A capture does not say what kind of call it holds. So the shipped figure is the most generous one.

A network that knows its traffic is local or toll wants a tighter one.

  • Pass --pdd-threshold 6.0 (or set [diagnosis] post_dial_delay_secs).
  • Both --slow-setup and --problems move with it.

Pair the alias with --report for a summary instead of per-message records:

sipnab -N -I capture.pcap --slow-setup --report

What to look for:

  • High retransmits alongside high PDD. The caller keeps retransmitting the INVITE, for one of two reasons:
    • The first one never landed.
    • The remote side is slow to respond.
  • DNS resolution delays. These are common when the proxy does NAPTR/SRV lookups for every call.
  • Deep proxy chains adding latency at each hop.

Next steps: Compare pdd with retransmits.

  • If retransmits > 0, the delay is network loss or an unresponsive next hop.
  • If retransmits == 0 but PDD is still high, the downstream server is slow to route. Check its logs, database lookups, or LCR table performance.

NAT traversal issues

The Contact or Via header advertises a private IP that doesn't match the actual packet source.

Find the calls where the address in the SIP headers disagrees with the address the packet came from:

sipnab -N -I capture.pcap --filter "nat_mismatch == true" --json

The --nat-issues alias is the same selection without writing the filter out:

sipnab -N -I capture.pcap --nat-issues

What to look for: nat_mismatch == true means RTP for the call reached the capture point from an address that no SDP in the dialog advertised.

  • The far end therefore sends its media to the address the SDP named.
  • Nothing answers on that address.
  • So the return path fails, even though signaling completed.

sipnab compares addresses only, not ports. NAT and RTP proxies rewrite the port on healthy calls too.

Next steps:

  • Proxy-side: Enable fix_nated_contact and fix_nated_register (OpenSIPS/Kamailio). They rewrite Contact headers with the observed source address.
  • Endpoint-side: Configure STUN/TURN on the phone or softclient, so it discovers its public address.
    • Check that it worked. See NAT discovery went unanswered below.
    • A phone configured with a STUN server it cannot reach behaves exactly like a phone with no STUN at all.
  • Network-side: Disable SIP ALG on every NAT device in the path. SIP ALGs almost always make things worse.

The SDP offered a private address

private_media_address == true means the dialog advertised a private address in its c= line.

  • The address is RFC 1918, RFC 4193 or link-local.
  • The peer it went to is not itself private.

sipnab raises this as a warning, not a fault. There are two situations, and only one of them fails:

  • Something downstream rewrites the SDP: an SBC, an ALG, or a media proxy.
    • The address the far end finally sees is routable.
    • This is correct, common, and there is nothing to do.
  • Nothing rewrites it.
    • The far end sends media to an address the internet cannot route.
    • The call still signals cleanly and answers 200.
    • The audio is one-way.

Without more to go on, sipnab cannot tell those apart from one capture. So it reports the evidence rather than a verdict.

Carrier-grade NAT space (RFC 6598, 100.64.0.0/10) is NOT flagged. It is routable inside the carrier that assigned it, so a mobile call that offers it can work.

Unless STUN settles it. The capture may also hold the client's STUN or TURN exchange. Then:

  • The two situations stop being indistinguishable.
  • The stun_sdp_mismatch object on the diagnosis carries the proof:
What STUN shows What it means
A mapped or relayed address in public space, and the SDP named the private one anyway Nothing rewrote it. The client knew its routable address and did not use it.
The probe drew no reply, and the STUN server is itself public The client tried to reach the internet and could not, so it fell back to the private address.
The probe drew no reply, but the STUN server is on the LAN This proves nothing โ€” a LAN-only exchange says nothing about internet reachability, and sipnab stays quiet.

The first two raise private_media_address on their own, with no observed stream needed. So a call whose media never arrived at all still gets the finding.

The correlation is by the client's IP address. The STUN exchange and the dialog share nothing.

  • The evidence may sit well outside the call: merged captures, or a probe from an earlier registration.
  • Then the finding says so.
  • A persistent NAT-discovery failure is an inference about the client, not an observation of this call.

NAT discovery went unanswered

sipnab reads STUN and TURN. It reports transactions that went out and never came back. Against tests/fixtures/stun_nat_probe.pcap, sipnab -N -I tests/fixtures/stun_nat_probe.pcap --no-cli-print logs these two lines:

2026-10-18T15:27:34.907106Z  WARN sipnab::app::batch: STUN/TURN: 192.0.2.10:5060 sent Binding to 198.51.100.20:3478 2 time(s) and got no reply (traversal-2.1.0 45). An endpoint that cannot learn its reflexive address falls back to advertising its PRIVATE address in SDP, which the far end cannot route back to โ€” the usual cause of a call that signals cleanly and carries audio one way. Note WHAT is missing: no reply at all, rather than a refusal. Silence points at something dropping the packets in the path rather than at the server refusing them, and on school, campus and corporate networks the usual culprit is a security appliance โ€” web filter, secure web gateway, firewall or IPS โ€” discarding UDP it does not recognize. Check whether such a device sits in this path and whether it permits UDP to the STUN/TURN port before suspecting the server.
2026-10-18T15:27:34.907131Z  WARN sipnab::app::batch: STUN/TURN: 1 of 2 transaction(s) went unanswered.

How this leads to one-way audio. It is the first link in the one-way-audio chain above.

  • The phone asks the network what address the outside world sees it as.
  • It gets nothing.
  • It falls back to the only address it knows: its private one.
  • It then writes that address into its SDP.
  • Everything after that looks healthy.

Read what is MISSING, not just that something is. A reply that never comes is a different fault from a reply that says no:

What you see What it means
No reply at all Something in the path is discarding the packets. On school, campus and corporate networks that is most often a security appliance -- web filter, secure web gateway, firewall or IPS -- dropping UDP it does not recognize. Check whether one sits in this path and whether it permits UDP to the STUN/TURN port before suspecting the server.
An error response The server was reachable and refused. sipnab counts this as answered, not as unanswered. Look at the code: 401/438 are authentication, not connectivity.
A reply arrives, media still one-way STUN worked. The fault is downstream -- check the SDP the far end actually received, and see The SDP offered a private address.

A retransmission counts as one unanswered question with N attempts, not N questions. A phone that retries five times has asked once, and nothing answered it five times.

A capture holding nothing else is still worth reading.

  • Two unanswered Binding Requests and no SIP at all is not an empty capture.
  • It is the cause of a one-way-audio complaint.
  • sipnab says so, rather than reporting "no SIP traffic found".

sipnab_nat_unanswered_requests carries the same signal for a dashboard. See Prometheus Metrics.

sipnab --stun prints the whole exchange as a table. It shows one row per transaction, what came back, and how long it took. --json-stun is the same thing as NDJSON.

A relayed call that went quiet halfway through

A TURN allocation has a lifetime. The client must Refresh it before that lifetime runs out.

  • The client may not do so, or its Refresh may never reach the server.
  • Then the relay tears the allocation down.
  • The media stops mid-call.

This is the only fault sipnab reports that has no other symptom anywhere.

  • No SIP message says the audio stopped.
  • The signaling shows a healthy call that went quiet.
  • Both endpoints are still happily sending into a relay that no longer exists.

sipnab flags it wherever the run reports anything:

  • on stderr as TURN: N allocation(s) were still carrying traffic after the lifetime they were last granted had run out
  • as LAPSED in the --stun allocations table
  • as lapsed: true in --json-stun
  • as a turn_allocation_lapsed finding in --analyze
  • as sipnab_nat_lapsed_turn_allocations for a dashboard, /v1/stats and the MCP capture_status tool

Two things it does not claim:

  • It says no Refresh was seen, never that none was sent. A capture that started late or lost a packet cannot tell those apart.
  • A Refresh carrying LIFETIME 0 is a deliberate release. It is never reported, because the client asked for the teardown.

Find the audio that stopped with the relay. sipnab unwraps TURN ChannelData, so the RTP inside it appears in the stream list as ordinary media with phone-to-relay addresses. These surfaces name which streams the relay carried:

  • The --stun allocations section says which channel carried which SSRC.
  • The lapsed-allocation lines say media on it:.
  • The --analyze finding counts relayed_streams.
  • --json-stun puts a channels array on the allocation.
  • A relayed call's own diagnosis carries media_relay, naming the relay its audio crossed.

On a dashboard, sipnab_nat_lapsed_turn_allocation_streams is the scale beside the allocation count. A relay torn down with nothing on it cost nobody a call.

Both ICE agents think they are in charge

ICE gives one agent the controlling role and the other controlled. The controlling one picks the candidate pair. When both claim the same role:

  • The agent that notices answers 487 Role Conflict (RFC 8445 section 7.3.1.1).
  • One side switches role.
  • It repeats every check it had already sent.

Often ICE fixes this itself, and the call costs a round trip. Where it does not:

  • Nothing is ever nominated.
  • The call has no media path.
  • The signaling looks perfectly healthy.

The usual source is one of these:

  • two endpoints configured with the same role
  • a B2BUA relaying one side's role attribute straight to the other

sipnab reports it:

  • on stderr as ICE: N candidate pair(s) show a role conflict
  • as a ROLE CONFLICT line in the --stun ICE section
  • as an ice_role_conflict finding in --analyze
  • in the ice record of --json-stun
  • as sipnab_nat_ice_role_conflicts for a dashboard, /v1/stats and the MCP capture_status tool

Each report says whether the conflict resolved. Resolved means the agents nominated a pair between them anyway, and the conflict cost one round trip. UNRESOLVED means no pair was ever nominated: that conflict is a candidate cause of media that never started, and is the one to act on.

ICE:   192.0.2.11:50006 <-> 203.0.113.11:16002: 2 487 response(s), UNRESOLVED -- no pair between them was ever nominated, so this is a candidate cause of media that never started

The same section answers the question ICE otherwise leaves unanswered: which candidate pair won.

  • nominated 192.0.2.10:50004 -> 203.0.113.9:16000 is the ICE analogue of the mapped address.
  • It names the path the media actually took.

Nothing may have answered a single check. Then the ICE section says so outright: ICE never completed, so the call has no media path.

  • Those individual transactions also appear in the unanswered list below it.
  • That is where sipnab reports the silence itself: one silence, stated once.

SIP scanner detection

Scanners probe for open registrations and try credential stuffing. Detect them early and feed the IPs to fail2ban.

Detect scanners on a live interface. Append fail2ban-compatible lines to a log file fail2ban can watch.

  • --kill-scanner additionally sends the kill response back to the scanner.
  • To detect and send nothing, use --kill-scanner-detect-only instead. See Detect scanners without answering them.
  • So run --kill-scanner only where you mean to:
sudo sipnab -N -d eth0 --kill-scanner --fail2ban >> /var/log/sipnab/scanners.log

After the fact, match the known scanner User-Agents in a capture. This only reads the file, and sends nothing:

sipnab -N -I capture.pcap --filter "ua =~ 'friendly-scanner|sipcli|sipvicious'"

What to look for:

  • known scanner fingerprints (friendly-scanner, sipvicious, sipcli)
  • high REGISTER rates from a single source
  • sequential extension enumeration (INVITE to 100, 101, 102, and upward)

Next steps:

  • Count who the detectors would ban before fail2ban sees any of it.

    • On a carrier trunk the enumeration signature and a busy hunt group look alike.
    • The addresses at the top of this list are routinely your own SBCs:
    sipnab -N -I capture.pcap --kill-scanner --fail2ban \
      | grep -oE 'src=[^ ]+' | sort | uniq -c | sort -rn

    Put every address you recognize into the jail's ignoreip. Raise maxretry until the list holds only what you meant. A jail that bans your carrier takes the phone system down more thoroughly than the scanner would have.

    sipnab does that arithmetic for you and hands back the rule itself. --recommend-block groups the detections by source. It prints one block per accused address, with:

    • the evidence
    • the counter-evidence
    • a fail2ban, nftables or iptables rule
    sipnab -N -I capture.pcap --kill-scanner --recommend-block fail2ban

    sipnab recommends and does not apply.

    • It reaches no firewall, holds no credential and runs nothing.
    • The output is text you read first.
    • Read the COUNTER-EVIDENCE: line of every block before you run any of it.
    • An address that also completed a registration or a call is a working peer. For those:
      • The address-specific dialects arrive commented out.
      • The fail2ban dialect arrives with an ignoreip line already filled in.
  • Then point fail2ban at the log file.

    • --fail2ban chooses the format and detects nothing on its own.
    • --kill-scanner produces scanner_detected lines. Add --kill-ua to it to match another User-Agent; sipnab refuses --kill-ua alone.
    • --reg-flood produces reg_flood lines.
    • Without one of those, the log stays empty while looking like an all-clear.
    • sipnab warns on stderr when you ask for the format with no scanner detector armed.
  • For broader detection, combine flags: sudo sipnab -N -d eth0 --kill-scanner --fraud-detect --reg-flood --alert syslog

  • Use --digest-leak to check if any endpoints are leaking credentials in cleartext.

The run stops answering scanners when stderr goes away

--kill-scanner sends its responses from a separate worker process, which writes its log lines to the run's stderr.

  • Before 0.5.263, the worker ended at its next log line when a write to that stderr failed. Two examples: a pipe whose reader had exited (2>&1 | head), and a full disk.
  • From then on the run detected scanners and answered none, and its DISABLED warning went to the same lost stderr.
  • Since 0.5.263, the worker drops a log line it cannot write, and keeps answering.
  • To keep the log lines themselves, send stderr to a file or to the journal, not to a pipe that can end before the run.

Registration-flood timers

--reg-flood counts a failure only when all of these hold:

  • A registrar's 401 or 407 answers a REGISTER that already carried credentials.
  • The answer is on the same transaction.
  • It arrives before that transaction ends.

Three settings decide what it reports:

  • --reg-flood-threshold ([security] reg_flood_threshold, default 50).
    • How many such failures from one source make a flood.
  • --reg-flood-window ([security] reg_flood_window_secs, default 1 second).
    • How much capture time one count spans.
    • Raise it to catch a slow guessing run, such as one refused REGISTER every few seconds.
  • --reg-flood-transaction-timeout ([security] reg_flood_transaction_timeout_ms, default 32000 ms).
    • How long a REGISTER stays open to its challenge.
    • sipnab treats a challenge that arrives later as a stray and never counts it.

These are not the --alert reg-flood:50/10s:5m rule. That rule sits on top.

  • It counts the flood findings this detector files, per source.
  • It decides when to raise an alert, and how long to stay quiet afterward.
  • The settings above decide whether a finding exists at all.

Set the transaction timeout to your Timer F.

  • RFC 3261 section 17.1.2.2 ends a non-INVITE transaction, REGISTER included, at Timer F, 64 times T1.
  • Section 17.1.1.1 sets T1 to 500 ms by default.
  • It recommends a larger value on links known to have a longer round trip.
  • That is where the default 32 seconds comes from.

Pick the value this way:

  • Your phones or SBC run T1 above 500 ms: set 64 x T1.
    • T1 at 1 s gives 64000.
    • T1 at 2 s gives 128000.
  • You capture on the registrar side of a stateful proxy, and its final-response timer is longer than that: set the proxy's timer instead. The proxy keeps relaying the registrar's answer until that timer fires.
    • OpenSIPS: the tm module fr_timeout is in seconds (default 30), so multiply by 1000. Its T1_timer is in milliseconds (default 500).
    • Kamailio: the tm module fr_timer is already in milliseconds (default 30000).
      • Its retr_timer1 is T1 in milliseconds (default 500).
      • max_noninv_lifetime (default 32000) caps a non-INVITE transaction at Timer F.
  • Neither applies: leave the default. Both proxies ship a 30-second final-response timer, inside the default 32 seconds.

fr_inv_timeout (OpenSIPS) and fr_inv_timer (Kamailio) govern INVITE transactions only. So they never apply to a REGISTER.

When the capture cannot show a failure. The detector counts the registrar's answers. A capture without them gives it nothing to count. At the end of the run --reg-flood then warns one of these:

  • reg_flood cannot establish credential failures: the capture holds no answer to any REGISTER.
  • reg_flood cannot establish the outcome of N of M credentialed REGISTER(s): some went unanswered inside the transaction timeout.

The MCP security_findings tool and GET /v1/security/findings carry the same statement in observation_gaps. It names no source and bans nothing.

Fix the capture first:

  • Capture both directions of the registrar's traffic.
  • Check that no capture filter keeps requests only.
  • The answers may be present but slow. Then raise --reg-flood-transaction-timeout as described above.

Symptom of a timeout set too short:

  • A registrar on a slow or congested link answers late.
  • sipnab drops its 401s as strays.
  • So a credential-stuffing run against it never reaches the threshold.

Symptom of one set too long: a stray challenge on a reused branch can settle a REGISTER that the client gave up on long ago.

The accepted range is 1000-600000 ms.


Registration failures

Is REGISTER rejected with 401 Unauthorized, 403 Forbidden, or 423 Interval Too Brief? Phones not registering means no inbound calls, and potentially no outbound.

sipnab -N -I capture.pcap --filter "method == 'REGISTER' AND state == 'Failed'" --json

For one line per registration rather than one per message, read the registration_failure finding.

  • It answers this question directly.
  • It also catches the failure that is not a rejection at all.
    • A registrar may grant less time than the phone asked for.
    • Then the phone re-registers far more often than it planned:
sipnab -N -I capture.pcap --json-dialogs --no-cli-print --quiet \
  | jq -c 'select(.signaling_diagnosis.registration_failure) |
           {call_id, from} + .signaling_diagnosis.registration_failure'

kind is rejected or shortened_expiry. On the second, compare requested_expiry_sec against granted_expiry_sec.

final_status_code on a REGISTER dialog is the registrar's final answer to the REGISTER. A challenge that was then answered does not count.

  • So it shows that the registrar refused a registration.
  • It does not show why it granted a short expiry.
  • registration_failure names both.

What to look for:

Code Meaning Typical fix
401/407 Auth challenge Normal first response -- check if the phone retries with credentials. If it doesn't, credentials are misconfigured.
403 Forbidden IP not in ACL, registration not allowed for this user, or domain mismatch
423 Interval too brief The registrar wants a longer expiry. Increase the registration interval on the phone.

Next steps:

  • This sequence is healthy: a REGISTER gets 401, then a second REGISTER with credentials, then 200.
  • Repeated 401s with no successful registration: the password or auth username is wrong.
  • retransmits > 3 on REGISTERs: the registrar may be unreachable.

Three or more challenges with no 200 is exactly what signaling_diagnosis.auth_loop reports. Its kind separates the two causes:

  • credential_failure: the phone answers and is re-challenged. The password is wrong.
  • silent_drop: the phone never sends Authorization at all. It has none configured, or does not understand the challenge.

Reports, audio and the browser analyzer

These are tasks rather than symptoms, so the cookbook carries them:


A live capture that sees nothing

sipnab -d eth0 runs, exits cleanly, and reports No SIP traffic found. But you know the link carries calls, and tcpdump -i eth0 shows them.

Check what sipnab asked the kernel for.

  • Run with SIPNAB_LOG=info.
  • Read the Auto-generated BPF filter: line.
  • Take that exact expression to tcpdump, against the same interface or against a capture from it:
tcpdump -r capture.pcap -nn 'portrange 5060-5061' | wc -l

Zero there means the filter never matched. The kernel drops those frames before sipnab sees them, so no counter, metric, NOT DECODED line or report shows them missing.

The usual cause is encapsulation.

  • A port filter matches the outer headers only.
  • So SIP inside a VLAN tag, QinQ, PPPoE or MPLS never matches one.

Confirm by comparing. Against the repository's 32-frame PPPoE sample, tests/pcap-samples/DTMFsipinfo.pcap, the plain filter above counts 0. The same filter behind the PPPoE keyword counts all 32:

tcpdump -r tests/pcap-samples/DTMFsipinfo.pcap -nn 'pppoes and portrange 5060-5061' | wc -l

If you passed your own filter (a positional expression or --bpf-file), that is the cause. sipnab uses your expression exactly as typed and never edits it.

Drop it and let sipnab generate one.

  • The generated filter carries an arm for VLAN, QinQ, PPPoE, VLAN-over-PPPoE and one or two MPLS labels.
  • That arm fires on Ethernet and on both Linux cooked headers alike.
  • Omitting -d on Linux therefore costs no encapsulation coverage.

On raw IP from a tun device and on the two loopback headers, sipnab leaves the arm out. None of those link types carries a tag.

Two cases the generated filter still does not cover:

  • SIP inside a UDP tunnel (GTP-U, VXLAN, GENEVE).
    • Off by default, and sipnab warns about it at startup.
    • BPF cannot reach the inner port. So the only way to cover it is to take the whole port.
    • Add --capture-tunnels, and size the buffer for the extra volume first (Tuning capture).
  • SIP on a port outside --portrange. Widen it (see Start here).

The generated filter also admits RTP on any port.

  • It does so untagged only, over IPv4 and IPv6 without extension headers.
  • Media inside a VLAN tag or a tunnel needs a filter of your own that names it.

The generated filter also misses SIP inside an encapsulation and in an IPv4 header with options. Untagged IPv4 with options matches.

To confirm which of these you are looking at, run the generated filter and a plain portrange over the same capture with tcpdump -r, and compare the counts.

An archive of captures

-I session.tgz reads every capture in the archive, the way -I dir/ reads a directory. Read the lines sipnab prints before the analysis starts. They say what came out of the archive and what did not:

Skipping 'session.tgz/lab/cell-trace.pcap': empty (0 bytes)
'session.tgz' unpacked (gzip > tar): 12 capture(s) to read, 1 member(s) not read, 1 directory entr(ies)
-I resolved to 12 capture file(s), 1 archive member(s) not read (each named above with its reason)
  • not a capture (starts with {head}, which is not pcap or pcapng): the member is not pcap or pcapng. {head} is its first four bytes in hex, which identify what it is. A text file starts with printable bytes: a member notes.txt holding call notes reads Skipping 'session.tgz/lab/notes.txt': not a capture (starts with 63 61 6c 6c, which is not pcap or pcapng).
  • xz data, which sipnab does not unwrap (or zstd, bzip2, LZ4).
    • Unpack that member with the tool that made it.
    • Then point -I at what comes out.
    • sipnab reads ZIP and 7z members itself, password-protected ones included. A build without the archive feature names them here instead (Read captures from tar, ZIP and 7z archives).
  • encrypted, and no password was supplied or encrypted, and no password supplied opens it: a locked ZIP or 7z member. Give sipnab the password (Read a password-protected ZIP or 7z).
  • has link type N, which sipnab does not decode.
    • The member is a capture of something other than IP, such as an LTE or NR MAC log from a test handset.
    • Its frames appear under NOT DECODED.
    • Nothing else in the run changes.
  • could not be read past this point or ends before its archive says it should: the archive stops early.
    • sipnab keeps what arrived.
    • The run ends INCOMPLETE RUN with exit status 1, because members past that point are in no report.
  • decompressing reached the N-byte ceiling: the archive inflates past --max-gunzip-bytes. For an archive you trust, raise it.

The CLI reference covers the rest:

  • how nesting works
  • how frame pointers name a member
  • where the unpacked copies live while the run reads them

A HEP collector that receives nothing

A -L listener that shows no calls has one of two problems. The silence warning in its log already tells you which. The listener warns when it admits nothing for 30 seconds (--hep-silence-warn):

  • HEP listener on 0.0.0.0:9060: no packets for 30s โ€” upstream sender may be down (UDP gives no error for a dead peer); capture is still listening means nothing arrived at all. The sender is not reaching the port. Check:
    • the address and port it targets
    • the firewall
    • whether the SIP server is mirroring at all
  • HEP listener on 0.0.0.0:9060: no packets admitted for 30s โ€” 30 arrived and every one was refused, mostly auth_mismatch (the auth key does not match this listener's secret), the last of those from 192.0.2.7; the sender is reaching this port and being turned away; capture is still listening means packets arrive and sipnab turns every one away.
    • The line names the reason most of them met, and the peer that sent them.
    • The network is fine, and the sender's configuration is not.

Some proxies may get through while one does not. Then no warning fires, because the listener as a whole is busy. Ask the roster instead:

A proxy that stopped shows as silent. A proxy that sends and gets refused has no sender row. Its address appears under refused_sources with the reason:

Reason What to fix
auth_mismatch The sender's shared secret differs from --hep-auth-file
auth_missing The sender sends no key, and this listener requires one
allowlist The sender's address is outside every --hep-allow range
hmac_timestamp_out_of_window The sender's clock is off by more than --hep-hmac-window; fix NTP there
hmac_bad_mac A wrong key under --hep-auth-mode hmac, or a packet altered on the way
hmac_unsupported_version The sending sipnab predates the current HMAC token; upgrade it
hmac_bad_format The packet carries no HMAC token this listener can read: most often a sender in --hep-auth-mode plain against a listener in hmac mode. Set the same mode on both
hmac_replay A token this listener has already accepted, inside the window: a packet sent twice, or replayed
rate_limit_global, rate_limit_per_peer Traffic above --hep-rate-limit or --hep-rate-limit-per-peer
peer_tracking_full More senders at once than [limits] max_tracked_peers
malformed The bytes are not HEP; check the sender speaks HEP v3, or v2 over UDP
event_sender_not_listed An OpenSIPS E_SIPNAB_TH_MAP event on a listener with no --hep-allow; list OpenSIPS's address there (Have OpenSIPS send the mapping)

The roster names senders by the capture id each claims and the address it sends from. Give each sender its own id (--hep-id on a sipnab agent). Then two agents on one host stay apart.

The sender may itself be a sipnab (--hep-send). Then ask it too.

  • Over TCP or TLS, a first connect that fails stops the sender's run at startup with exit status 2:

    Failed to create HEP sender: Failed to connect HEP sender to '127.0.0.1:19301' over TCP: Connection refused (os error 111)
    

    Start the collector, or correct the address, then start the sender again.

  • Its runtime_stats or GET /v1/runtime carries hep_export.

  • A headless run ends with one HEP export to line. A UDP run of sipnab -N -I tests/pcap-samples/sip-rtp-g711.pcap --hep-send 127.0.0.1:19999 --no-cli-print ends with:

    HEP export to 127.0.0.1:19999 over udp: 10 packet(s) sent, 10 queued, 0 replayed after an outage, none dropped, 0 still queued, none failed, 0 reconnect(s); sent means handed to the kernel: UDP reports no delivery, so a collector that is down produces no failure here unless its host refuses with ICMP
    
    • none dropped, or the number lost and each reason: queue_full, stopped or unsent_at_exit. These are the only packets lost.
    • N failed attempt(s), with each kind and its count, such as 9 failed attempt(s) (connect 8, write 1). A failed attempt keeps its packet queued, so a failure is not a loss.
      • connect and write failures are a collector that went down or became unreachable. The sender redials with backoff and replays the queue.
      • tls_handshake failures are a collector whose certificate the sender does not accept.
    • replayed after an outage counts what the sender held and delivered once the collector came back. When the collector goes away shows a restart.

Over UDP the sender cannot tell.

  • A collector that is down produces no failure.
  • So the collector's roster is the only witness.

The plaintext legs may show while the proxy's TLS or WebSocket legs do not. Then look at the NOT DECODED line.

  • OpenSIPS and Kamailio tracers name TLS and WebSocket legs over HEP with IP protocol numbers 22 and 50. sipnab 0.5.188 and later decode both. See How --hep-listen reads the transport.
  • The collector may report no transport (IP protocol 22) or ESP not NULL-encrypted (IP protocol 50). Then it is older than 0.5.188: upgrade it.
  • Any other number is one no tracer convention covers. Check the sender's configuration.

Encrypted SIP that does not decrypt

The calls are on port 5061, and sipnab reports nothing. Or it reports fewer messages than the phone did.

  • Decryption has several independent ways to produce that one symptom.
  • One log line separates most of them.

Command:

SIPNAB_LOG=info sipnab -N -I capture.pcapng --portrange 1-65535 --report

What to look for. One line decides where to go next:

TLS decryption active: 6 secret(s) from embedded DSB in capture.pcapng

Its presence means keys reached sipnab and were usable. Its absence means they did not, whatever else the run printed.

Keys arrived and there are still no calls

Widen the port range. This is the most common ending.

  • Decryption and port filtering are separate steps.
  • The second still applies afterwards.

sipnab says so rather than staying silent:

SIP outside --portrange 5060-5061 is being skipped: 1 message(s) so far,
in no count, no dialog, and no output. Busiest port(s): 15061 (1).

Re-run with --portrange 1-65535. A lab or a carrier trunk on a non-standard TLS port produces exactly this.

No TLS decryption active line at all

Three causes, in the order they are cheapest to rule out.

The binary has no tls feature. It is a build-time feature, not a runtime flag. A build without it ignores the whole subject:

sipnab --version

tls must appear in the feature list it prints. If it is absent, no flag helps. Rebuild with --features tls, or install a build that has it.

The keylog covers different sessions.

  • Keys only exist for handshakes the producer was running for.
  • A keylog started after the INVITE cannot decrypt that call.
  • Neither can one from a different host.
  • Compare the line count against the number of TLS sessions in the capture before you blame the file.

The producer is a pipe and nothing is reading it yet. See A FIFO keylog, and capture appears to hang.

A FIFO keylog, and capture appears to hang

--keylog accepts a named pipe, so the keys never touch a disk (recipe 7f in the cookbook). sipnab opens the pipe without waiting for a writer and reads it as a stream. Check that it recognized the pipe:

SIPNAB_LOG=info sipnab -N -I capture.pcap --keylog kl.fifo --report 2>&1 | grep FIFO
2026-10-16T11:52:15.421081Z  INFO sipnab::app::bootstrap: TLS decryption: kl.fifo is a FIFO, reading it as a live stream

That line means the capture does not wait for the producer. If it is present and no keys arrive, the producer is not writing to that pipe: check the path you gave the producer.

If the line is absent and the run stalls until something writes to the pipe, or loads no keys from a pipe that has them, the binary is older than the FIFO support. Install the current release, then run the command again.

Keys loaded at first, then stopped

The producer rotated or truncated the file. sipnab detects both and reloads the file from the beginning, with one warning that names the file and ends in one of these:

was truncated by its producer โ€” reloading it from the beginning
was replaced by its producer โ€” reloading it from the beginning

Look for that warning first. If there is no warning and no new keys, the producer is not writing to the path you passed: check its configuration.

eCapture refuses to start in -m pcap mode

couldn't add a ingress filter to interface 1: netlink receive: no such file
or directory

That is the kernel, not eCapture. -m pcap attaches a TC classifier to the interface as well as its uprobes. So it needs CONFIG_NET_CLS_BPF:

zcat /proc/config.gz | grep NET_CLS_BPF

# CONFIG_NET_CLS_BPF is not set means this mode cannot run on that kernel.

  • Use -m keylog instead (cookbook 7e).
  • It needs only the uprobes, and runs there.

A missing /sys/kernel/btf/vmlinux is not a reason to expect failure. eCapture falls back to its own non-CO-RE bytecode. It logs which file it loaded.

A vCon export that fails

--export-vcon may fail in one of three ways. Each has one usual cause.

  • It refuses to run.
  • It cannot find the call.
  • It writes a container with no audio in it.

sipnab refuses --export-vcon

--export-vcon needs the 'vcon' Cargo feature, which this build does not carry. Rebuild with --features vcon (or --features full); `sipnab --version` lists the features a binary was built with

The run stops with exit status 2. The exporter is behind a non-default Cargo feature, so a default build does not carry it. Check what you have:

sipnab --version        # the feature list is on the version line

If vcon is absent, build one that has it:

cargo build --release --features vcon

--export-vcon says the Call-ID is not there

Call-ID 'x@y' not found in tracked dialogs, so there is no dialog to export. --report lists the Call-IDs this run holds.

The run exits with status 1. It holds no dialog under that Call-ID. --report lists the ones it does hold.

  • The usual cause is a Call-ID copied with surrounding quotes, or a truncated one.
  • The second is a capture that never saw the dialog open.

Quote the Call-ID in the shell. Most contain @. Many contain characters your shell would otherwise interpret.

The container came out with no audio in it

The recording dialog object carries no body. Read the media_note in the container. It states what the run kept, and is never a finding that the call was silent:

grep -o '"media_note": "[^"]*"' call.vcon.json
"media_note": "No audio payload retained: sipnab measured 425 RTP packet(s) of PCMU on 1 decodable stream, but kept none of their payload, so there is nothing to decode. This is a statement about what this run kept, not a finding that the call was silent. Most often audio payload retention was off โ€” start the server with --retain-audio to hold payload for export. With retention already on, the other causes are a --snaplen that truncated the payload away before sipnab saw it, or packets that carried no payload at all."

Export again with --retain-audio:

sipnab -N -I capture.pcap --retain-audio --export-vcon 'abc123@host' --vcon-out call.vcon.json --no-cli-print

Then check that the recording now has a body:

jq -c '.dialog[] | {type, has_body: has("body"), mediatype}' call.vcon.json
{"type":"recording","has_body":true,"mediatype":"audio/x-wav"}

Audio also has an inline size budget, --vcon-max-inline-media (default 5 MiB). sipnab refuses a recording over it rather than truncating it, and the caveat names the size it turned down. Raise the budget only if the system that receives the container accepts larger bodies.


An output file sipnab cannot create

Problem: one of these happens:

  • A root run with -O, --export-vcon-dir, --vcon-out or --redact-map stops at startup with exit code 1.
  • A long run stops when --split rotates.

It stops with a line such as:

Failed to create output file '/var/tmp/e2e/rtp03.pcap': Permission denied (os error 13). sipnab dropped privileges to user 'nobody' before opening output; make the directory writable by that user (e.g. install -d -o nobody <dir>), pass --user <name> for a user that can write there, or --no-priv-drop.

Why: run as root, sipnab opens the capture device and then drops privileges.

  • It drops to nobody, unless --user or [privilege] user names another account.
  • It creates the -O file after the drop, and each --split file later still.
  • So the drop user creates the file, not root.
  • sipnab checks right after the drop that the drop user can create each of those outputs.
  • So the refusal comes before the first packet, not when the file is first needed.

A directory made with a plain sudo mkdir is root:root mode 0755. nobody may read it but not write it.

Command: pick one.

Give the directory to the drop user:

sudo install -d -o nobody /var/tmp/e2e

Or run as an account that can already write there (sipcap is an example name):

sudo sipnab -N -d any -O /var/tmp/e2e/rtp03.pcap --user sipcap udp

Or keep root for the whole run:

sudo sipnab -N -d any -O /var/tmp/e2e/rtp03.pcap --no-priv-drop udp

What to look for: the text after the file name is the operating system's reason.

  • Permission denied is the privilege drop above.
  • No such file or directory means the directory does not exist. sipnab does not create it, so make it first.
  • The privilege sentence appears only when sipnab did drop privileges.
    • A run without it may still say Permission denied.
    • That means your own account may not write there.

A service that does not start

systemctl status sipnab names the exit status, and journalctl -u sipnab -n 20 the reason. sipnab prints that reason at every log level.

  • Exit status 2: a flag, or a pair of flags, sipnab refuses. The usual one on a service is a listener bound to a routable address with no credential:

    REST API refuses to start: --api 0.0.0.0:18080 is non-loopback but no --api-key / SIPNAB_API_KEY or --api-signing-key / SIPNAB_API_SIGNING_KEY was supplied. Bind 127.0.0.1, or configure authentication.
    
  • Exit status 1: a runtime failure, or a value in sipnab.toml it refuses. A HEP listener with no allowlist or secret also exits 1.

  • Exit status 101: sipnab panicked. See Send us a crash report.

  • Exit status 129: something sent SIGHUP, or the terminal of a terminal interface run went away. sipnab stopped as it does for SIGTERM, and the packaged unit's Restart=on-failure starts it again. 0.5.233 and earlier did not handle SIGHUP: the service went inactive with Result=success and stayed down until started again.

When the service does not start lists each refusal message with its fix.

Live capture says Permission denied

Permission denied on 'eth0'. Grant capture capabilities once (Linux), then re-run without sudo:

The process lacks CAP_NET_RAW and CAP_NET_ADMIN. Which fix applies depends on how sipnab runs. sipnab reads NoNewPrivs and CapBnd from /proc/self/status (NoNewPrivs as it read at startup, before sipnab sets the flag on itself) and looks for a container marker (/.dockerenv, /run/.containerenv, or a container environment variable), and the lines under the message name the cause it found:

How it runs Cause Fix
From a shell, as your user The binary has no file capabilities, or an upgrade replaced it and cleared them sudo sipnab --setup-caps
Under systemd with NoNewPrivileges=true NoNewPrivileges stops file capabilities taking effect, so setcap cannot help AmbientCapabilities=CAP_NET_RAW CAP_NET_ADMIN in the unit (Write a unit of your own)
In the container image, newer than 0.5.245 The container's bounding set lacks CAP_NET_RAW (--cap-drop ALL, or a policy that drops it), or a shell starts sipnab under --security-opt no-new-privileges docker run --cap-add NET_RAW --cap-add NET_ADMIN, with sipnab as the entrypoint (Capture live traffic in the container)
In the container image, 0.5.245 or earlier, or in another container as a non-root user The sipnab binary has no file capabilities, so its user gets no capabilities, even with --cap-add A newer tag, or docker run --user root; sipnab drops to nobody after it opens the capture
In a container or unit whose bounding set lacks CAP_NET_RAW Nothing inside the process, root included, can gain a capability missing from its bounding set docker run --cap-add=NET_RAW, or CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN in the unit

Each cause has its own first line after Permission denied on '<dev>'.:

sipnab found The message continues with
NoNewPrivs: 1 at startup This process runs with NoNewPrivs set (systemd NoNewPrivileges=yes, or a container started with no-new-privileges), then the AmbientCapabilities= lines
CAP_NET_RAW missing from CapBnd CAP_NET_RAW is not in this process's capability bounding set, then docker run --cap-add=NET_RAW and CapabilityBoundingSet=
A container marker, and neither of the above This process runs in a container (/.dockerenv) as a user that holds no capture capability, then docker run --user root --cap-add=NET_RAW --cap-add=NET_ADMIN <image>
None of these Grant capture capabilities once (Linux), then re-run without sudo:, then sipnab --setup-caps, sudo and setcap

To check which cause applies, read the same two lines sipnab reads:

grep -E '^(NoNewPrivs|CapBnd)' /proc/self/status

Run it in the environment sipnab runs in, for example in the container with docker exec. NoNewPrivs: 1 is the first cause. In CapBnd, bit 13 is CAP_NET_RAW and bit 12 is CAP_NET_ADMIN. capsh decodes the mask into names.

A key file sipnab cannot read

metrics auth: --metrics-auth-file '/etc/sipnab/metrics.auth': Permission denied (os error 13)

A run started as root reads its key, token, credential and certificate files before it drops to nobody (or the --user account), so a file only root can read works there. This message means the user sipnab started as cannot read the file. Under the packaged unit that user is sipnab: make the file root:sipnab mode 640, as Make a certificate shows. Every listener names a TLS certificate or key the same way, by its flag, path and reason:

--mcp-tls-key: /etc/sipnab/mcp.key: Permission denied (os error 13)

In 0.5.244 and earlier the MCP message was the path alone, such as /etc/sipnab/mcp.key.

A root run reads the [vcon_fetch] store's files (auth_file, decrypt_key, ca, extra_ca, signer_ca) before the drop as well: --api-allow-vcon-fetch and --mcp-allow-vcon-fetch read them when the servers start. 0.5.238 and earlier read them after the drop and stopped with exit status 2:

--api-allow-vcon-fetch: [vcon_fetch] auth_file '/etc/sipnab/fetch.auth': Permission denied (os error 13)

The credential's permission rule is the same as before, applied as the user sipnab runs as after the drop. sipnab refuses a credential that user owns when other users can read it, and accepts one that root or another account owns whatever its mode:

--api-allow-vcon-fetch: [vcon_fetch] auth_file '/etc/sipnab/fetch.auth' can be read or written by other users (mode 0640); chmod 600 it

A revocation list is different, because sipnab reads it again whenever it changes, as the user it dropped to:

--api-revoked-file '/etc/sipnab/revoked': Permission denied (os error 13). user 'nobody', which this run dropped privileges to, cannot read it, and a revocation list sipnab cannot read revokes nothing. Make it readable by that user (for example root:<its group> 0640)

The run stops with exit status 2. Make the file readable by the --user account, for example root:sipnab mode 640 with --user sipnab.

A client the listener turns away

Answer Listener Cause
401 REST API, MCP No bearer token, an expired or revoked one, one from the other surface's key, or one scoped too narrowly for the route. A metrics token gets 401 on /v1/stats
401 MCP /health No token: on MCP, /health needs one. The REST API's /health does not
401 metrics No HTTP Basic credential, on a routable bind
403 REST API, MCP The Host header names a host not on --api-allowed-host or --mcp-allowed-host
404 metrics /health The standalone metrics server has no /health. Check /metrics instead
503 REST API, MCP, metrics The client is over its rate limit or the in-flight cap. The limiter runs before authentication, so 503 says nothing about the credential

Use a token explains each REST and MCP status, and how to read the WWW-Authenticate challenge MCP adds to a 401.

A TLS handshake that fails

  • curl exits 60 with SSL certificate problem: unable to get local issuer certificate. The client does not trust the issuer of sipnab's certificate. Pass it with --cacert ca.pem.

  • curl exits 1, or a client reports a reset, against an http:// URL. The listener serves HTTPS only once it has a certificate. Use https://.

  • A HEP sender stops at startup with exit status 2:

    Failed to create HEP sender: Failed to establish a TLS session with '127.0.0.1:19063': invalid peer certificate: UnknownIssuer
    

    The sender does not trust the collector's issuer. Name it with --hep-tls-ca. The listener logs nothing for the failed handshake at the default level, so this line on the sender is the evidence.

  • A HEP sender in the container image stops with no system CA bundle found; name the collector's issuer with --hep-tls-ca. Images up to 0.5.245 have no CA bundle. Pull a newer tag, which carries Debian's bundle, or mount the issuer's certificate and name it with --hep-tls-ca.

API TLS, MCP TLS and Carry the feed over TCP and TLS set each listener up.

A setting that has no effect

Ask sipnab which file it read and what it made of it:

sipnab -D --config /etc/sipnab/sipnab.toml

Then check, in order:

  • A misspelled key. sipnab ignores an unknown key and warns: WARN sipnab::config: Unknown config key: capture.snaplenn. In the terminal interface that warning does not show unless you set SIPNAB_LOG: a TUI run logs errors only by default.
  • Another file. sipnab reads only the first file it finds. -D prints # Loaded from: and, when a later candidate also exists, # Also present and NOT read:. See File Locations.
  • A flag. A flag beats the same key in the file. A systemd unit or drop-in may pass one.
  • A running service. sipnab reads the file at startup only. Restart it.

A long run that fills the disk or the memory

A terminal interface that outlives its terminal

sipnab 0.5.234 has a defect: a terminal interface run whose terminal goes away keeps running. You closed the window, the SSH session dropped, or something killed the tmux server, and sipnab still shows in ps in state R, using CPU. It does not stop on SIGTERM.

  • Fix: upgrade to 0.5.235 or later. From 0.5.235, sipnab stops on its own within a second of losing its terminal, and exits 129.

  • On 0.5.234: end the process with SIGKILL. First check that the pid is that run. Replace 12345 with its pid:

    ps -o pid,stat,etime,args -p 12345

    Then end it:

    kill -KILL 12345

From 0.5.235, a terminal interface whose terminal is still there also stops on SIGTERM, SIGINT and SIGHUP. 0.5.234 ignored all three. 0.5.233 and earlier ignored SIGTERM and SIGINT, and SIGHUP ended the process at once, with no end-of-run summary.

0.5.235 to 0.5.260 can still keep running in one case: the terminal goes in the moment sipnab reads a key from it. The process shows in state R and uses a full CPU, on macOS and on Linux. In CI on macOS, 1 launch in 60 did this after tmux kill-session.

  • Fix: upgrade to the release after 0.5.260. sipnab reads its terminal itself, and a read that finds the terminal gone stops the run, with exit status 129.
  • On 0.5.235 to 0.5.260: end the process with SIGKILL, as above.

A run that ignores SIGTERM while it waits for input

sipnab 0.5.234 and earlier have a defect: sipnab --vcon-fetch - whose standard input stays open with nothing to read keeps waiting. SIGTERM, SIGINT and SIGHUP do not stop it. A run given a FIFO or a pipe with -I (-I /dev/stdin, a process substitution, a named pipe) can wait the same way, and such an input never worked.

  • Fix: upgrade to 0.5.235 or later. From 0.5.235, sipnab stops --vcon-fetch - within a second of the signal. It fetches nothing and exits 0, or 129 after SIGHUP. It refuses a FIFO, a pipe or a device given to -I at startup, with exit status 1. Save the capture to a file and give that file instead.

  • On 0.5.234 or earlier: end the input. Close the program that writes to sipnab's standard input, or end the --vcon-fetch - run with SIGKILL. First check that the pid is that run. Replace 12345 with its pid:

    ps -o pid,stat,etime,args -p 12345

    Then end it:

    kill -KILL 12345

Send us a crash report

When sipnab hits a bug it cannot recover from, it stops. It prints two lines to the terminal:

sipnab panicked at src/main.rs:152:9:
crash report written to /home/you/.local/state/sipnab/sipnab-crash-20260923-191230-2507858-0.log

That file is what we need. You do not have to reproduce the crash or install anything extra.

  • Find the report. It is the path on the crash report written to line.
    • By default reports land in ~/.local/state/sipnab/, and in /var/lib/sipnab/ under the packaged systemd unit.
    • Each name has the form sipnab-crash-<date>-<time>-<pid>-<n>.log.
    • A report_dir in the [crash] config section moves them.
    • If you closed the terminal, list the newest one:
ls -t ~/.local/state/sipnab/sipnab-crash-*.log | head -1
  • Check what it contains before you send it. The report is plain text. It holds:

    • the panic message
    • the source line
    • the thread name
    • the version
    • the build ID of the executable
    • its load address
    • one raw address per stack frame, up to 256 frames

    It contains no capture data and no packets. But:

    • The panic message can quote a value sipnab was handling.
    • The path names your home directory.
    • Read it and redact what you need to.
  • Send it. Open an issue.

    • Attach the .log file.
    • Add the output of sipnab --version and the command you ran.

Under a systemd unit, the default report directory may not be writable. The journal then holds the report's text and the line failed to write crash report: Read-only file system (os error 30). Copy the text from the journal, and see Keep crash reports.

A panic in the parser of one input does not stop the run. sipnab drops that frame, packet, record or archive member, logs one warning such as contained a panic at <location>: <message>, and goes on. It writes no crash report for it. Send that line, with sipnab --version and the input if you can share it.

A core dump is not needed, and it is not safe to share.

  • A core holds the process's memory.
  • That memory includes the SIP messages and media it had captured.
  • Only turn on core = true if we ask for a core. Then keep it on your side.

What the report records, and how we read it

The release strips every sipnab binary it publishes.

  • So the backtrace in the report cannot name a function.
  • Most of its lines read <unknown>.
  • The Image: and Raw frames sections are what make it readable anyway:
Image:
  Path:      /usr/bin/sipnab
  Build ID:  edbd855707e15a8b69f6e0fb3f0ac2bf925c23b1
  Load base: 0xaaaac4090000
  Target:    aarch64-unknown-linux-gnu

Raw frames (runtime return address, then image+address of the call in
that image's file. Resolve these against the symbol file named by
the ID above):
    0  0xaaaac4a904d4  sipnab+0xa004d3
    1  0xaaaac4a8ffe8  sipnab+0x9fffe7

Each release publishes a symbol file for every binary it ships, built by the same compile:

  • sipnab-<version>-<target>.debug for Linux
  • sipnab-<version>-<target>.dSYM.zip for macOS
  • The -noaudio binaries inside the -noaudio .deb and .rpm packages have their own, with -noaudio in the name.

The Build ID (Linux) or UUID (macOS) line pairs the report with exactly one of them. So a rebuild cannot stand in for it.

To resolve a report yourself:

  • Download the symbol file named by the report's version and Target: line from the release page.
  • Name it once in a shell variable, here for an aarch64 glibc report. Then the commands below paste as they are:
DEBUG=$(ls sipnab-*-aarch64-unknown-linux-gnu.debug)

Check that it is the right one:

readelf -n "$DEBUG" | grep 'Build ID'

The ID must equal the report's Build ID line.

Then pass the hexadecimal offset of each sipnab+0x address, such as 0xa004d3 from sipnab+0xa004d3, to addr2line. The two below are the first frames of the example report above:

addr2line -f -C -i -e "$DEBUG" 0xa004d3 0x9fffe7

Or to llvm-symbolizer, if you have LLVM:

llvm-symbolizer --obj="$DEBUG" 0xa004d3 0x9fffe7

Each address prints the function, and the source file and line it came from, in the form src/<file>.rs:<line>.

  • The numbers after + are addresses inside the file.
  • So they work however the system placed the binary in memory.
  • They point at the call instruction, one byte before the return address in the first column.
  • So each frame names the line that made the call.

On macOS:

  • Unzip the .dSYM.zip.
  • Compare dwarfdump --uuid sipnab.dSYM with the report's UUID line.
  • Pass the numbers after sipnab+ to atos.
    • On macOS they are addresses in the binary's own layout, which starts at 0x100000000.
    • So they look larger than the Linux ones.
  • Put them in ADDRS, separated by spaces:
atos -o sipnab.dSYM/Contents/Resources/DWARF/sipnab $ADDRS

For gdb or a core file on Linux, put the .debug file in the same directory as the sipnab binary.

  • The binary's .gnu_debuglink names that file.
  • gdb loads it from there without further setup.

Still stuck?

Build custom queries with the Filter DSL. It offers:

  • 42 fields plus any header by name
  • named filters
  • regex support
  • boolean logic

See the CLI Reference for every flag and more recipes.

The capture itself may be the problem. For example:

  • drops on a busy link
  • a full kernel ring buffer
  • loss that appears on every call at once

Then see Tuning capture on a busy server.

โš ๏ธ **GitHub.com Fallback** โš ๏ธ