Tuning Capture - NormB/sipnab GitHub Wiki

Use this page when a live capture drops packets.

• sipnab's defaults already suit a busy production link.
• So check first whether you are dropping anything at all.
• If you are, change the settings below in the order they appear. That is the order that helps most.

Start here, always: find out whether you are actually dropping packets. Every other decision on this page depends on that number, and sipnab reports it.

sipnab polls libpcap's kernel counters once a second and reports two numbers.

PACKETS ARE BEING DROPPED on 'eth0' (kernel buffer: 18432, interface/driver: 0).
The analysis for this run is INCOMPLETE — dialogs may be missing messages and
RTP loss figures will overstate what was on the wire.

You get that warning the moment the first drop happens. You also get a summary at the end of the capture:

Live capture on 'eth0' finished: 4821003 packets captured, but 18432 dropped by
the kernel buffer and 0 by the interface — THIS ANALYSIS IS INCOMPLETE

A clean run says so explicitly, so silence is never ambiguous:

Live capture on 'eth0' finished: 4821003 packets, no drops

That distinction is the single most useful thing on this page.

• Operators routinely respond to any drop by raising -B.
• That does nothing at all for interface drops.
• It wastes memory while the real problem goes unaddressed.

sipnab polls pcap_stats on each open capture handle — the socket this run opened. libpcap keeps those counters per handle.

• A handle is one capture socket.
• An interface is the NIC every process on the host shares.

The difference changes what a zero means and what a total covers.

• Zero drops means this run missed nothing.
  • It says nothing about the interface.
  • Another sniffer on the same NIC keeps its own counters.
  • Neither run can read the other's.
• One run can hold several handles.
  • --multi-device opens one per named interface.
  • --cores N on a live device opens N sockets on one interface (On a live device it means capture sockets, not workers).
  • sipnab adds each handle's increment to one process-wide pair. So the totals cover the whole run rather than any single socket.
  • The first-drop warning names the device that dropped. That tells you where to act.
• The packet count beside them is sipnab's own tally, not a libpcap counter.
  • It counts frames sipnab read.
  • The drop pair is what says how many more the wire carried.

sipnab reports both counters on four surfaces. It keeps them apart on every one of them. So whichever you are looking at can answer "which kind of drop is this?"

Two separate Prometheus counters rather than one sipnab_capture_dropped_total with a label is deliberate:

• A dashboard that sums them produces a number whose remedy is ambiguous.
• An alert on that sum pages someone to raise a buffer that cannot help.
• Alert on them separately, or the alert cannot say what to do.

The REST API and MCP report the same pair on the capture-quality object:

• kernel_dropped_packets and interface_dropped_packets.
• They sit beside the invalid_timestamps, undecodable_frames and snapped-frame counts. Those make up the rest of a capture's quality.
• See GET /v1/stats.

Why a drop is not just "missing packets".

• A dropped SIP message means a dialog reconstructs wrong.
  • A missing BYE leaves a call that never ends.
  • A missing 200 OK leaves one that never answered.
• A dropped RTP packet counts as network loss that never happened. So MOS and loss figures read worse than the call actually was.
• A lossy capture does not produce a smaller answer. It produces a wrong one.

Default: 64 MiB per capture device.

This is the ring libpcap fills and sipnab drains. It absorbs the difference between two times:

• when packets arrive (bursty, driven by your traffic);
• when sipnab gets scheduled to read them (jittery, driven by your kernel).

It is the setting that matters most.

Busy trunk — give it room for a bigger burst:

sudo sipnab -N -d eth0 -B 256

Small or embedded host — cap the memory instead:

sudo sipnab -N -d eth0 -B 8

-B alone may buy less than you think.

• libpcap divides the ring into fixed-size slots. Their size comes from --snaplen.
• So how many packets 64 MiB holds depends on:
  • the snapshot length;
  • whether NIC offloads are on — anywhere from ~1,000 to ~41,000;
  • whether you named an interface at all.
• Read Snapshot length and Which interfaces: any versus named before concluding that a bigger -B did not help.
• The three settings multiply.

Rules of thumb:

• Raise it when kernel buffer drops are non-zero and CPU is not pinned.
  • Dropping with idle CPU means bursts.
  • Bursts are exactly what a ring absorbs.
• Do not raise it when the drops persist and a core is at 100%.
  • A bigger ring buys a longer burst, not more throughput.
  • You are not keeping up on average, and no buffer size fixes that.
  • Go to Capture less: BPF filters.
• Lower it on --multi-device runs.
  • The cost is per device.
  • Eight interfaces at the default reserve half a gigabyte of kernel memory.

Asking the kernel for a large ring can fail with ENOMEM on a small or loaded host. sipnab does not treat that as fatal. Instead it:

• halves the request and retries, down to a 2 MiB floor;
• tells you when it settled for less:

'eth0': the kernel refused a 64 MiB capture buffer; capturing with 16 MiB
instead. This host will tolerate a smaller burst before dropping — watch the
drop counters, and set -B/--buffer explicitly to pin a size.

sipnab honors an explicit small -B exactly and never promotes it upward. -B 1 on a constrained box means 1 MiB.

pcap_set_buffer_size takes a C int. So 2047 MiB is the last whole MiB whose byte count fits. sipnab clamps anything larger and says so:

-B/--buffer 5000 MiB exceeds the 2047 MiB ceiling (pcap_set_buffer_size takes a
C int); capturing with 2047 MiB instead.

The clamp matters because the arithmetic underneath used to wrap. --buffer 2148 once handed libpcap a negative byte count. Read the warning as a real limit rather than as advice. No -B above 2047 buys anything.

Two settings carry the word. They sit on opposite sides of the kernel boundary.

• -B sizes the kernel ring.
• --buffer-budget sizes the in-process queue between the capture thread and the processing loop.
  • It is in MiB, and it defaults to 64.
  • sipnab turns that budget into a packet count at an assumed 2 KiB average.
  • It then clamps the result to between 10,000 and 5,000,000 packets.

Raise it when processing lags behind a bursty capture the ring is already absorbing. It does nothing for a ring that overflows. The kernel discards those packets before this queue ever sees them.

The cheapest packet is the one the kernel never gives you.

• A BPF filter runs in the kernel, before the ring.
• So filtered traffic costs no buffer space, no copy, and no parse.

A live capture already has one. Give sipnab no expression of your own, and it does this:

• It compiles one from --portrange (default 5060-5061).
• It then adds an arm that reaches SIP inside one VLAN tag, QinQ, a PPPoE Session header or an MPLS label stack.
• It logs the expression it installed. So -v shows you exactly what the kernel is running.

The baseline is therefore signaling-only already. This section is about narrowing further or about letting media back in.

Your expression replaces that one entirely. sipnab never edits what you typed.

• So an expression of your own drops the encapsulation arm.
• It also makes --capture-tunnels inert.
• Two warnings cover that:
  • one when a port-based expression shows no sign of handling encapsulation;
  • one naming --capture-tunnels as ignored when you passed it beside a filter of your own.

Careful with the flag name:

• --filter is sipnab's own matching language, applied to messages after capture.
• The BPF expression is the trailing argument, or a file named by --bpf-file.

Only SIP signaling, and nothing tagged:

sudo sipnab -N -d eth0 "port 5060 or port 5061"

Signaling plus one media range:

sudo sipnab -N -d eth0 "port 5060 or (udp portrange 10000-20000)"

One customer's traffic:

sudo sipnab -N -d eth0 "host 203.0.113.10"

This is the correct first response to sustained drops with a busy CPU. Halving the traffic that reaches userspace is worth more than any buffer size.

Careful with RTP.

• Filtering to port 5060 alone gives you signaling with no media.
• So every stream turns orphan and every MOS figure disappears.
• If you want quality metrics, the filter must admit the negotiated media ports too.

Default: 65535 — the whole frame.

The obvious reading of --snaplen is "how many bytes of each packet get copied".

• On that reading it looks unimportant. A 1500-byte frame costs 1500 bytes whether the cap is 1600 or 65535.
• That reading is incomplete. On a busy server it is the expensive kind of incomplete.

On Linux, snaplen also determines how many packets your ring can hold — on the TPACKET_V2 ring.

• Which ring you get depends on the run mode.
• The subsection at the end of this section is the rule. Read it before applying the arithmetic here to a headless capture.
• libpcap's create_ring() sizes each slot in the V2 ring from the snapshot length:

frame_size = handle->snapshot;
/* ... clamped for Ethernet ... */
req.tp_frame_size = TPACKET_ALIGN(macoff + frame_size);
req.tp_frame_nr   = (handle->opt.buffer_size + req.tp_frame_size - 1)
                    / req.tp_frame_size;

The slots have a fixed size. So -B buys you buffer_size / frame_size packets, not bytes of useful queue. There is a clamp that can rescue you. But read its guard carefully:

if (handle->linktype == DLT_EN10MB) {
        ...
        if (offload)
                max_frame_len = MAX(mtu, 65535);
        else
                max_frame_len = mtu;
        max_frame_len += 18;
        if (frame_size > max_frame_len)
                frame_size = max_frame_len;
}

The clamp only applies to DLT_EN10MB — real Ethernet.

• sipnab's default capture device on Linux is any (src/capture/device.rs:38-40). The reason for that choice: SIP servers often listen on loopback.
• any is DLT_LINUX_SLL2, not DLT_EN10MB.
• So on the default configuration no clamp runs at all. The slot stays at the full snaplen:

So out of the box — default device, default snaplen — a 64 MiB ring holds about a thousand packets.

• That is milliseconds of slack.
• It is roughly forty times less than the same memory would buy at a smaller snaplen.
• Before this raised the default from 2 MiB, that same arithmetic gave 31 slots.

Note what this means:

• Naming an interface explicitly and disabling offloads is worth far more than either alone.
• Only that combination reaches the clamp.
• Which interfaces: any versus named is the decision guide for the device half of that. It covers:
  • what leaving any gains you;
  • what it costs;
  • how to check you did not drop a call leg on the way.

# Signaling-focused capture: ~36,000 slots in the same 64 MiB
sudo sipnab -N -d eth0 --snaplen 1600

Three ways out, and they compose:

• Cap the snaplen (above).
  • Immediate, no root beyond capture.
• Name the interface (Which interfaces: any versus named).
  • It is the only way to reach DLT_EN10MB at all.
  • It is the prerequisite for the clamp below.
  • It is a coverage trade. So read Which interfaces: any versus named before making it.
• Turn the offloads off (Interface and driver drops).
  • This is independently correct for capture fidelity.
  • GRO/LRO hand you reassembled super-frames that were never on the wire.

--capture-profile picks the number for you.

• signaling asks for a snaplen of 1500 — one MTU.
  • That keeps a whole INVITE carrying a long Record-Route set, ISUP encapsulation or a fat SDP offer.
  • It drops the bulk of every RTP packet.
• full is 65535, the default.
• An explicit --snaplen wins over the profile. Someone who typed a number has already answered the question the profile asks.

About that profile:

• It is a named profile rather than a smaller default because truncation is not free, as the note below says.
• It stops at 1500 rather than a tighter 200 to 400 bytes. One INVITE with a full Record-Route set, a long Contact, ISUP encapsulation or a fat SDP offer passes 400 bytes routinely.

A snaplen that cuts a header stops the message parsing. That reports the peer that sent a valid message as broken. sipnab_capture_snapped_frames_total counts how many frames arrived truncated.

sudo sipnab -N -d eth0 --capture-profile signaling

Truncation is lossy, and not everything survives it.

• A small --snaplen breaks audio reconstruction.
  • --retain-audio, Opus decode, the TUI's WAV save and the MCP export_audio tool all need whole RTP payloads.
• It degrades -O capture re-emit to truncated frames.
• sipnab tracks captured versus original length per packet. So truncation is visible rather than inferred.
• But choose the value deliberately, not reflexively.

To limit only how much sipnab parses without truncating what it captures, use -S/--limitlen instead. That is a parser bound, not a capture bound.

Modern libpcap on Linux can use the block-based TPACKET_V3 ring.

• It sizes its blocks independently of the snapshot length.
• So it does not have the capacity cliff described above.
• One flag decides whether you get it, and sipnab decides that flag for you.

The rule is in libpcap's prepare_tpacket_socket():

/*
 * The only mode in which buffering is done on PF_PACKET
 * sockets, so that packets might not be delivered
 * immediately, is TPACKET_V3 mode.
 *
 * The buffering cannot be disabled in that mode, so
 * if the user has requested immediate mode, we don't
 * use TPACKET_V3.
 */
if (!handle->opt.immediate) {
        ret = init_tpacket(handle, TPACKET_V3, "TPACKET_V3");

So immediate mode reads like a latency preference. It is really a ring-format choice. sipnab answers it by asking who consumes the packets (immediate_mode_for() in src/app/bootstrap.rs):

V3 is not free of its own trap. sipnab pays for it explicitly rather than inheriting it.

• libpcap copies the read timeout into req.tp_retire_blk_tov.
• It then polls with -1.
• So the timeout becomes added delivery latency rather than a poll bound.

The interactive 100 ms would have meant up to 100 ms before a block retires. The batched path therefore uses its own BATCHED_READ_TIMEOUT_MS = 5 (src/capture/live.rs).

Shutdown responsiveness (--duration, Ctrl-C) never depended on either:

• the handle is non-blocking;
• an empty ring returns TimeoutExpired;
• the wait is sipnab's own bounded wait_readable().

So the snaplen arithmetic above binds the TUI, not a headless capture.

• On a headless run, --snaplen and the offload settings still matter for capture fidelity and for copy cost.
• But they are not what decides how many packets the ring holds.

Unverified on hardware.

• Whether the kernel selects V3, and what that is worth, rest on reasoned from libpcap's source and not measured.
• Two checks would settle it:
  • strace -e trace=setsockopt for PACKET_VERSION;
  • KERNEL_DROPPED under load against a V2 baseline.

Default on Linux: any — every interface at once, loopback included.

That default is deliberate. It is a correctness choice, not a performance one. find_default_device() returns "any" on Linux (src/capture/device.rs:35-40), for the reason written beside it:

// On Linux, "any" captures all interfaces, which is what we want.
// SIP servers often listen on loopback, so capturing only eth0 misses traffic.

That is a real hazard, not a hypothetical one. Capture eth0 alone and these legs are simply absent:

• a B2BUA talking to a registrar over 127.0.0.1;
• a containerized stack bridging SIP across docker0;
• a proxy handing calls to a media server over a veth pair.

A dialog missing one leg does not come back smaller. It comes back wrong, exactly as Are you dropping packets? describes for dropped packets. any is the setting that does not lose calls.

It is also the slowest and least capable device sipnab can open, on four counts.

• Three are performance.
• One is a second correctness cost that runs the other way.

• It forfeits about 40x of your ring capacity — on the V2 ring, so on the TUI.
  • Snapshot length has the mechanism and the run-mode rule that scopes it.
  • libpcap's create_ring() sizes each TPACKET_V2 slot from the snaplen.
  • The clamp that cuts a slot down to MTU+18 sits behind if (handle->linktype == DLT_EN10MB).
  • any reports DLT_LINUX_SLL2, so that clamp never runs.
  • Every slot is the full 65535-byte snaplen regardless of interface MTU.
  • No ethtool setting can reach it. The guard tests the link type, not the offloads.
  • At the 64 MiB default that is ~1,000 slots against ~41,000 for a named Ethernet interface with offloads off.
  • Before the default buffer still defaulted to 2 MiB, the same arithmetic gave any just 31 slots.
• It cannot go promiscuous.
  • The capture loop in src/capture/live.rs computes let use_promisc = config.promisc && device != "any".
  • The pseudo-device does not support promiscuous mode. So sipnab does not ask for it.
  • Promisc is on by default for a named device, and --no-promisc turns it off. On any there is nothing to turn off.
  • The consequence is that any misses traffic not addressed to the host.
  • That matters on precisely the deployment where you would want it: a SPAN port or tap feeding mirrored calls the capture host is not a party to.
  • That is a correctness cost. It points the opposite way from the loopback argument. any sees every interface but only the host's own traffic on them.
• It runs one capture thread.
  • Naming devices unlocks --multi-device.
  • That spawns one coordinator thread plus one capture thread per interface (start_multi_capture() in src/capture/native.rs).
  • Each has its own ring and its own drain loop.
  • any is one device. So it is one thread and one ring no matter how many interfaces the traffic actually arrives on.
  • --cores N (On a live device it means capture sockets, not workers) is the other way to get more than one socket. It works on a single device rather than on a list.
• It sweeps interfaces you never wanted.
  • any also picks up loopback, docker0, veth pairs, tunnels and management interfaces.
  • Every one of those packets costs a BPF evaluation.
  • If it passes, it also costs a copy into the same ring the traffic you do want is competing for.

Stay on any when any of these hold:

• You are diagnosing rather than monitoring.
• You do not yet know which interface carries the traffic.
• SIP genuinely crosses loopback or container bridges and you have not enumerated those interfaces.
• The drop counters from Are you dropping packets? read no drops. If it is not dropping, it is not costing you anything worth this trade.

Name your interfaces when any of these hold:

• Are you dropping packets? shows sustained kernel buffer drops.
• You are running a long-lived headless capture on a known topology.
• You need promiscuous mode, because the switch mirrors the traffic to you rather than addressing it to you.
• You have several busy interfaces and one thread cannot drain them all.

Name what you need, open them in parallel, and turn the offloads off so the clamp can run:

First disable the offloads, once per interface you intend to capture:

sudo ethtool -K eth0 gro off lro off gso off tso off

Then name those interfaces and open them in parallel:

sudo sipnab -N -d eth0,eth1 --multi-device -B 256 "port 5060 or port 5061"

Naming the devices buys all four counts at once:

• The slots are now clamped to MTU+18. So the same 64 MiB holds tens of thousands of packets instead of ~1,000.
• Promiscuous mode is back.
• There is one capture thread per interface.
• Nothing goes to docker0.

Neither half works alone:

• any cannot reach the clamp however you set ethtool.
• A named device with offloads on is still stuck at ~65 KB slots (Snapshot length).

Verify you did not lose a leg.

• This is a deliberate trade of coverage for throughput.
• And the failure mode is silent — calls do not error, they just stop being complete.
• Before you keep it, confirm the interface list is right:

ip -brief address

If any SIP endpoint answers on 127.0.0.1 or a container bridge, add lo or docker0 to the -d list rather than accepting the gap.

• --multi-device costs -B per device (The kernel capture buffer).
• So eight interfaces at the default reserve half a gigabyte.
• Name what you need and no more.

• --no-rtp — skip RTP/RTCP entirely when you only care about signaling.
  • RTP is ~93% of carrier traffic by packet count.
  • So this is the largest single reduction available short of a BPF filter.
• --no-dialog — skip dialog reconstruction.
• -N / --no-tui — do not render a TUI you are not watching.
• --quiet-bad-parse — silence per-packet parse notices on dirty links.
• Avoid per-message output (--json, --text-dump) on a high-rate live capture unless you are consuming it.
  • Formatting and writing every message is real work on the capture path.

If interface/driver is non-zero, the packets never reached libpcap. -B is irrelevant. Look outside sipnab:

• NIC ring buffers
  • ethtool -g eth0 to inspect, ethtool -G eth0 rx 4096 to raise.
  • This is the NIC's own ring, distinct from libpcap's.
• Offloads
  • GRO/LRO/GSO make the kernel hand you reassembled super-frames.
  • Those no longer match what was on the wire.
  • For accurate capture: ethtool -K eth0 gro off lro off gso off tso off.
• IRQ affinity and RPS
  • A single core servicing all NIC interrupts is a common ceiling on a busy server.
• A tap or SPAN port that is already oversubscribed
  • If the mirror source is dropping, nothing on the capture host can recover it.

libpcap can read through netmap instead of the kernel's packet socket.

• The device name selects it: --device netmap:eth0.
• The libpcap behind a binary decides whether it can.
• That libpcap differs by artifact.

Where to check:

• Which capture backends an artifact can reach has the table.
• The second line of sipnab --version names the libpcap your binary runs and the alternate backends its banner names.

Version notes:

• On 0.5.183 and earlier, the musl tarballs' netmap module speaks only netmap API 13. Current netmap kernel modules refuse it.
• 0.5.184 moves to API 14.
• That page has the details. They include a crash on 0.5.188 and earlier when a netmap: capture that has captured traffic sees a frame its filter drops.

Warning: use a netmap: device only on an interface dedicated to capture, such as a SPAN or mirror port. Never use it on one the host needs. netmap takes the interface's rings away from the kernel. So the host stops receiving that interface's traffic for as long as the capture runs.

Measured on a veth pair:

• a UDP socket on the host received 0 of 5 datagrams during a netmap: capture;
• it received 5 of 5 during an ordinary capture on the same interface.

On the interface carrying your SIP service, that is an outage.

No other measurement on this page went through netmap.

--cores N names a different resource on each source. So read the one that matches your run.

Reading files (-I), it is N parallel reconstruction workers.

• They are sharded by host pair.
• Each has its own dialog and RTP-stream stores.
• That is the case this section measures.

sipnab -N -I /var/captures/ --cores 4 --report

Measured on the reference corpus (benchmarks):

• 1.13M pkts/s at one core;
• 2.17M at two;
• 3.23M at four;
• 3.24M at eight.

The curve flattens after four rather than declining.

• So four cores buys nearly all of it and eight buys almost nothing more.
• The gap between them is inside the noise of a single sweep.

Read the flattening as the shape it is.

• Four cores is where the returns stop, not a ceiling the eighth core falls off.
• Asking for eight costs the extra workers' memory for a throughput figure that does not move.

The workers compute the frame-provenance digest rather than the sequential reader. So that cost scales with the core count instead of capping every count at once.

--cores N on -d <device> asks the kernel to spread that interface across N AF_PACKET sockets through PACKET_FANOUT.

• Each socket gets its own ring and its own drain loop.
• That is the thing a ring that keeps overflowing actually needs.
• A bigger -B buys a deeper ring and still leaves one thread emptying it.
• Linux only. Elsewhere sipnab captures on one socket and logs the reason.

Three things to know before reaching for it.

• It widens capture, not analysis.
  • Every socket feeds the same channel and the same processing loop.
  • So --cores 8 is not eight cores of reconstruction.
• -B is per socket.
  • N sockets ask the kernel for N rings of that size.
  • So --cores 8 at the 64 MiB default reserves half a gigabyte.
  • sipnab states the total out loud when it opens the group.
• sipnab probes first, then commits.
  • A kernel that refuses PACKET_FANOUT refuses it for every socket.
  • So sipnab tests one throwaway handle, warns, and falls back to a single socket.
  • It does not let N threads each discover it.

sudo sipnab -N -d eth0 --cores 4 -B 32 "port 5060 or port 5061"

Reach for it when both of these hold:

• Are you dropping packets? shows sustained kernel buffer drops on ONE busy interface;
• The kernel capture buffer through Which interfaces: any versus named have not cleared them.

Several interfaces is --multi-device (Which interfaces: any versus named) instead.

sudo sipnab -N -d eth0 \
  -B 256 \
  --snaplen 1600 \
  "port 5060 or port 5061 or (udp portrange 10000-20000)" \
  --report

Note that this already makes the interface choice from Which interfaces: any versus named.

• It names eth0 rather than taking any.
• So confirm no SIP leg lives on loopback or a container bridge before adopting it.

Then read the drop line at the end.

• If it says no drops, stop there. You can walk the settings back to recover fidelity.
• If it does not, work down these in that order:
  • The kernel capture buffer
  • Capture less: BPF filters
  • Which interfaces: any versus named
  • Give sipnab less work
  • Interface and driver drops
• Re-measure after each change rather than applying all of them at once.

• CLI reference — every capture flag
• Configuration — the [capture] section
• Benchmarks — measured throughput and method
• Troubleshooting