Internals Threading - NormB/sipnab GitHub Wiki

The topology below is the reality on main today.

  • The core principle is design decision D2.
  • The packet path is synchronous threads + channels.
  • tokio exists only inside the optional servers.

Topology

                       โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
                       โ”‚                 shared state                     โ”‚
                       โ”‚  Arc<RwLock<DialogStore>>  Arc<RwLock<StreamStore>>
                       โ”‚        (parking_lot, single writer)              โ”‚
                       โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                        write  โ”‚                     read โ”‚ (try_read in TUI)
capture thread(s)              โ”‚                          โ”‚
  live device / file / HEP     โ”‚                          โ”‚
  (capture::start_multi_capture)                          โ”‚
        โ”‚ Packet               โ”‚                          โ”‚
        โ–ผ                      โ”‚                          โ”‚
  capture::channel  โ”€โ”€โ”€โ”€โ–บ processing thread โ”€โ”€โ”€โ”€โ”€โ”€โ–บ TUI event loop (main thread)
  (capped channel)        (TUI mode: pipeline::            crossterm events, render
                           process_packet;                 TUI MODE ONLY
                           batch mode: main.rs loop
                           on the main thread)

  auxiliary threads, by the RUN MODE that starts each โ€” none of them is a
  child of the TUI event loop:

  both modes
    โ”œโ”€โ”€ server runtime thread          app::servers::start_servers
    โ”‚     one shared current-thread tokio runtime hosting every enabled
    โ”‚     async server as a task: api (axum), mcp (rmcp), mcp-http
    โ”œโ”€โ”€ metrics-server thread          app::servers::start_servers
    โ”‚     raw std::net::TcpListener accept loop + one short-lived
    โ”‚     metrics-conn thread per scrape
    โ”œโ”€โ”€ DNS resolver thread            app::build_resolver
    โ”‚     std::mpsc queue; spawned only when reverse DNS is enabled
    โ””โ”€โ”€ rtpengine reconciler           app::relay_reconciler::spawn
          bounded hand-off queue; starts only when --rtpengine-control
          names a relay on a live run

  batch mode only
    โ””โ”€โ”€ scanner-kill worker            process_isolation::spawn_scanner_kill_worker
          a separate PROCESS (this binary, re-executed), started by
          bootstrap::launch before the chroot and privilege drop; two
          threads here serve its pipes; --kill-scanner has no TUI path

The mode column above is the part to keep true. It is worth saying why.

  • An earlier revision of this diagram drew every auxiliary thread as a child of the TUI event loop.
  • Not one of them is.
    • Both run modes start four of them.
    • Batch alone starts the scanner-kill worker.

That is not a cosmetic error.

  • --metrics really did ship wired to the TUI path alone.
  • So every headless -N deployment got no metrics at all.
    • That is every container and systemd unit.
  • This page said nothing that would have contradicted it.
  • A diagram can group threads by who spawns them rather than by which mode runs them.
    • Such a diagram reads as reassurance.
    • It hides exactly that class of defect.

The four spawn sites, none of which is the TUI event loop:

The Prometheus listener is not a task on the shared tokio runtime. The distinction matters when reasoning about blocking.

  • It is a raw TcpListener accept loop on its own thread.
  • It is deliberately independent of tokio and axum.
    • So metrics stay scrapable when the async servers are not compiled in at all.

Each accepted scrape runs on a short-lived metrics-conn thread.

  • A ConnGate bounds those to 16 in flight.
  • Beyond that, new connections get an immediate 503 rather than a thread (SN-02, CWE-770).
  • start_metrics_server() spawns the thread itself.
  • start_servers calls it for both run modes.

Named threads

Every long-lived thread and its spawn site. A name in a backtrace maps straight back to a row here.

Modes is the column to trust when asking "does this run headless?"

  • That is the question --metrics got wrong.
  • both means the thread exists under -N as well as in the TUI.
  • TUI and batch mean it does not.
Thread name Modes Spawned by Role
capture-<device> both capture/native.rs One per live device: pcap loop producing Packets.
capture-stop:<device> both capture/breaker.rs Beside each live capture: calls pcap_breakloop when a stop is due, for reads that never return by themselves (libpcap's netmap module on a silent link). Joined when its capture ends.
capture-file both capture/native.rs Offline pcap reader feeding the same channel as live capture.
capture-hep both capture/native.rs HEP/EEP receiver on the transport --hep-listen-transport names; packets carry asserted addresses and carry a flag HEP-origin. On udp this thread reads the datagrams itself. On tcp and tls it is the accept loop, and the reading happens in the per-connection threads below.
(unnamed per-connection readers) both capture/hep.rs One scoped thread per accepted HEP stream connection under --hep-listen-transport tcp or tls, up to HEP_MAX_STREAM_CONNECTIONS, so the listener serves several agents at once and one silent peer cannot stall the rest. Scoped, so no reader outlives the listener; each polls the stop flag on its read timeout, which is what bounds the scope's join. A TLS connection completes its handshake here rather than in the accept loop, for the same reason.
capture-multi both capture/native.rs Supervisor for a multi-device capture set.
tui-processor TUI app/tui_mode.rs The single store writer in TUI mode: drains the channel and runs the pipeline.
(unnamed workers) batch parallel.rs --cores N reconstruction workers, spawned with bare thread::spawn โ€” they own thread-local stores, so they show as unnamed in a backtrace.
(unnamed per-file readers) batch parallel.rs --cores N over a multi-file -I set: one scoped thread per reader, each reading whole files and handing per-shard batches to the dispatcher, which releases them to the workers in file order. A single-file -I has one file and therefore one reader, so it gets none of these โ€” the calling thread reads it, exactly as before.
servers both app/servers.rs One current-thread tokio runtime hosting api, mcp and mcp-http as JoinSet tasks.
metrics-server both output/prometheus_server.rs Raw TCP accept loop for Prometheus scrapes.
metrics-conn both output/prometheus_server.rs One short-lived thread per accepted scrape, capped at 16 concurrent.
sipnab-dns both names.rs Reverse-DNS resolver draining an std::sync::mpsc queue so the render path never blocks on a lookup. Starts only when the run turns reverse DNS on.
scanner-kill-fwd, scanner-kill-rx batch process_isolation.rs The parent's half of the scanner-kill worker, which is a process of its own: -fwd writes queued kill requests into the worker's request pipe (the one write that can block, so it is never the capture thread's), -rx reads its outcomes into the tally. The worker process runs the decision loop on its main thread between two pumps, kill-worker-in and kill-worker-out; it is the only process that answers an address the capture supplied, and the process parsing captured traffic holds none of its send sockets. --kill-scanner has no TUI path.
rtpengine-reconcile both app/relay_reconciler.rs Asks an rtpengine relay which calls it holds, over the relay's read-only list and query, so a stream the signaling does not explain still gets a Call-ID. Starts only when --rtpengine-control names a relay on a live run, and keeps the round trip off the packet path.
pcap-load TUI tui/controllers/file_open.rs Loads a pcap chosen from inside the TUI, writing the live stores.
clipboard TUI tui/clipboard.rs Holds the X11/Wayland selection alive after a copy without stalling the UI.
crash-probe tests crash.rs Not a production thread. Only mod tests spawns this name, to prove the panic hook records a thread name. It appears in no shipped backtrace. This row exists so the next reader who greps the name does not re-add it as one.

How the capture thread ends

Every fatal exit taken after start_capture() has to go through stop_and_join(). There is no second correct way out.

The reason is where the thread starts.

  • bootstrap::launch() spawns it before these steps:
    • The readiness hand-shake.
    • The chroot.
    • The privilege drop.
  • The reason: the capture source must be open while the process still holds CAP_NET_RAW.
  • A file run (-I) without --chroot is the exception: it drops privileges first and starts the thread after, so the process reads no byte of an input as root. The run opened its files when it planned, and the thread reads them through those descriptors. A failure in the privileged steps of such a run has no thread to stop.
  • Every failure from that point on happens with a capture thread already running and holding its source. Such failures include:
    • An unopenable source.
    • An unusable --chroot.
    • A refused privilege drop.
    • A hardening step the kernel rejects.
    • A companion server that cannot start.
  • std::process::exit joins nothing and runs no destructors.
    • So exiting directly abandons it.

stop_and_join does three things:

  • It sets the global shutdown flag.
  • It drops the packet receiver.
  • It joins.

Both signals matter, and neither is redundant.

  • Dropping the receiver ends a thread blocked on a send.
  • The flag ends one blocked elsewhere in its loop.
    • A live capture waiting out its read timeout reaches the flag check first.
  • One read never times out.
    • libpcap's netmap module waits inside pcap_next_ex until a frame arrives.
    • So on a silent link the flag check is never reached.
    • For that read, the capture's capture-stop:<device> thread sees the same flag and calls pcap_breakloop.
    • The loop takes the broken read as a stop.
  • The join has no timeout.
    • That matches the one the batch receive loop already performs at end of capture.
  • A HEP listener blocked on its socket returns in milliseconds rather than hanging it.

One consequence applies to callers that cannot reach the handle.

  • BatchRunner::new() does not own it.
  • So its four fatal paths return a PlanError instead of exiting.
  • batch::run holds both the handle and the receiver. It does the teardown.
  • A function that cannot clean up must hand the failure to one that can.

How a gate holds this.

  • ThreadSanitizer treats thread leak as a fatal finding, not a warning.
  • cli_flag_behavior_test exercises both shapes:
    • A source that never opens.
    • A failure after it opened.
  • So a regression fails there, rather than waiting for the weekly sanitizer run.
  • Before this rule, sipnab -I /nonexistent.pcap (a mistyped filename) leaked a thread.

Threads that exist at the privilege drop

The privilege drop signals every other thread of the process.

  • glibc carries out setgroups, setgid and setuid for the whole process.
    • The calling thread sends each other thread signal 33, and each one makes the same call.
  • A thread that waits in a call with a time limit at that moment gets EINTR.
    • Examples: poll, and a socket read with a receive timeout.
    • Measured on a development host: a poll with a 3 s limit returned EINTR after 0.30 s, when another thread called setgid.
  • A read with no time limit starts again.
    • In the same measurement a read on an empty pipe returned its byte, not an error.

Four calls make a drop.

The table lists, for each way a run starts, the threads that exist at its drop and what an interrupted wait does there. No row ends a run, logs an error or reports a timeout that did not happen.

Start path Threads at the drop What each waits in An interrupted wait
Live capture (-d) capture-<device>; one capture-<device>-<n> per extra capture socket on that device; capture-multi for several devices (src/capture/native.rs, src/capture/live.rs) poll on the capture descriptor for at most 100 ms. The pcap handle is non-blocking, so the read itself never waits. capture-multi waits in a channel receive and a join. poll_outcome returns the same answer as for an elapsed wait, and the loop reads again. The loop reports nothing for either.
Live capture, stop watcher capture-stop:<device> (src/capture/breaker.rs) park_timeout in a loop. The loop checks its two flags and parks again.
File (-I) capture-file Reads of regular files (the reader refuses a pipe, a FIFO, a device and a socket), and a send on the packet channel. Neither has a time limit.
HEP listener, udp capture-hep (src/capture/hep.rs) recv_from with a 100 ms receive timeout. recv_datagram returns no datagram and the loop goes round.
HEP listener, tcp and tls capture-hep, and a reader for each connection a peer made before the drop A non-blocking accept with a 20 ms sleep; reads with a 100 ms timeout; the TLS handshake's reads and writes. is_transient_recv_error counts Interrupted with the two timeout kinds: the accept loop, the reader and the handshake step each continue.
uprobe capture capture-uprobe A 20 ms thread::sleep between reads of its rings. The standard library sleeps again for the remaining time.
--kill-scanner scanner-kill-fwd, scanner-kill-rx (src/process_isolation.rs) A channel receive; a pipe read with no time limit. read_exact_or_eof reads again after Interrupted.
--cores none run_cores_file opens the set and drops before it starts a reader. Does not apply.
Scanner-kill worker process none start calls harden before serve starts kill-worker-in and kill-worker-out. Does not apply.
sipnab fleet serve none prepare reads the keys, binds and drops; serve builds the runtime after it. Does not apply.
sipnab fleet inventory none main reads every token file and drops before it starts a request thread. Does not apply.
--vcon-forward, --vcon-fetch, --vcon-audio-fetch none Each reads its files, drops, then creates its directory and connects, on the one thread. Does not apply.
--relay-holdings, --relay-sessions none run_relay_listing drops before the fan-out of src/relay/fan_out.rs starts. Does not apply.

Everything else starts after bootstrap::launch returns, so after the drop.

  • batch::run and run_tui_mode start these, or start what starts them:
    • The servers runtime with the REST API and MCP listeners.
    • metrics-server, fleet-member, rtpengine-poll and rtpengine-reconcile.
    • The hep-send delivery thread of --hep-send.
    • The vCon fetcher that --api-allow-vcon-fetch and --mcp-allow-vcon-fetch build.
    • sipnab-dns, tui-processor and the --cores workers.
  • sipnab fleet and --vcon-fetch are commands of their own. Each drops before it starts a thread.
  • The archive password command's reader and the relay snapshot's fan-out end before the capture starts.

Commands that end before the launch

A command that ends before bootstrap::launch does not reach the capture run's drop.

  • src/early_exit.rs holds one row for each such command, with what it does when root starts it.
    • A listener reads its key files and binds as root, then drops before it opens its store or accepts.
    • A network client reads its secrets as root, then drops before it creates a file or connects.
    • A keeper keeps the caller's privileges, and its row says why.
    • A command that parses a file the caller named opens and reads its inputs as root, then drops before it unpacks or parses them. Its readers take the open descriptor, so a capture only root can read is still read. The step that parses takes a value only the drop can make (early_exit::Confined).
  • tests/threads_at_privilege_drop_test.rs reads every line of src/main.rs that ends the process.
    • A new exit fails the test until it has a row.
  • Capture with the least privilege says what an operator sees.

How a gate holds this.

  • tests/threads_at_privilege_drop_test.rs lists every source file that starts a thread.
    • Each file is in one of three lists: at the drop, not at the drop, test code.
    • A file in none of them fails the test, and its message says what to read.
  • The same file fails when a fourth call drops privileges, and when a start path changes its order.
  • Each wait's own test holds its answer:
    • an_interrupted_poll_is_a_timeout_not_a_failure.
    • eintr_and_read_timeouts_are_transient_not_fatal.
    • an_interrupted_read_is_retried_rather_than_reported.
  • Tests that call setgid run in a process of their own, so they do not signal the other tests' threads.
    • no_test_changes_credentials_in_the_shared_process in privilege.rs holds that.

The second writer: pcap-load

Opening a pcap from inside the TUI spawns a second writer.

  • The pcap-load worker writes the same stores the render thread is reading.
  • That is why every render-side access is try_read() and never read().
sequenceDiagram
    autonumber
    participant UI as TUI event loop
    participant Load as pcap-load worker
    participant DS as DialogStore
    participant SS as StreamStore

    UI->>Load: spawn on O (file open dialog)
    UI->>DS: try_read for the frame
    DS-->>UI: guard, or skip the frame
    Load->>DS: write lock
    DS-->>Load: release
    Load->>SS: write lock
    SS-->>Load: release
    Load-->>UI: progress via async_messages
    Note over UI,SS: the render pass never blocks โ€” a contended try_read skips a frame
Loading

--cores N offline mode replaces the single processing thread with a dispatcher + N workers.

  • The dispatcher does a cheap host-pair peek.
  • It shards raw packets over bounded channels.
  • Each worker owns a private PacketProcessor + thread-local DialogStore/StreamStore.
    • There are no locks on the hot path.
  • The stores merge at EOF.
  • Flow correctness holds because a flow's packets share a host pair.
    • Therefore they share a worker.
  • A call's SIP and its RTP can run between different hosts, and so on different workers.
    • The dispatcher also sends a copy of each packet that can carry SDP to every other worker, at the same place in that worker's sequence.
    • A worker reads a copy only for the media sockets its SDP names, and counts nothing from it.
    • So each worker's StreamStore sees a later call take over a socket and an SSRC at the point in the capture where it did, and starts that call's stream there, as the single-threaded path does.
    • The same holds for a relay's control plane: the dispatcher also copies a HEP datagram to the mirror port (9060) and every datagram to or from --rtpproxy-control. Each worker pairs rtpproxy's commands and replies in its own table (relay::rtpproxy::with_worker_pairings).
  • The merge rebuilds what the single store holds under its limits.
    • It admits streams in the read order of their first packet, under --max-streams (StreamStore::merge_in_read_order).
    • It replays dialogs in read order. When they do not fit under --limit, it replays every message through process_message (DialogStore::merge_replaying).
    • Each reader records where the single-threaded path would sweep idle dialogs (SweepMark), and the merge compacts each dialog at those sweeps.

Lock discipline

  • Parse outside the lock.
    • SIP/RTP/SDP parsing happens before any store lock appears.
    • classify_packet() touches no store at all.
    • Each store is then locked for writing once per packet.
    • That makes writes the most frequent lock operation in the process, not the rarest.
  • Lock ordering:
    • When a path needs both stores, dialog store first, then stream store.
    • If it also needs the alert engine, that comes last.
    • The ordering is what carries the safety. The guards are not always disjoint.
    • Two paths each take one store at a time and release it before the other. "briefly" is accurate there.
    • The batch applier (batch.rs) does not work that way.
      • It holds both write guards across the entire per-packet body.
    • This page previously said both write locks were never held at once. That was never true of the batch path.
  • The guards queue side effects rather than performing them.
    • These accumulate while the code holds the guards:
      • Alert findings.
      • Per-message output.
      • The --alert-exec / --on-dialog-exec / --on-quality-exec spawns.
    • DeferredEffects::drain replays them after both drop.
    • So none of these happens inside the critical section:
      • A fork/exec.
      • A stdout write.
      • An AlertEngine lock.
    • Until that change, the batch loop took the alert engine's write lock nested inside both store guards.
      • That was a third lock deep in the hot path, with a posix_spawn beside it.
    • The full rule and what would re-create that edge are in Invariants, "Dialog before stream, then alerts โ€” one consistent order".
  • The TUI never blocks: all render-side store access is try_read().
    • On contention the frame renders with the previous data.
    • Counts may be one frame stale. This is deliberate.
    • An adaptive 10 fps active / 2 fps idle tick bounds staleness.
  • Pause is an AtomicBool checked by the processing thread. No lock.
  • mcp/ denies clippy::await_holding_lock.
    • parking_lot guards must never live across an .await.

Channels in use

Edge Flavor
capture โ†’ processing capture::channel โ€” a count-capped permit pool over an unbounded crossbeam channel, so idle memory returns to ~0 while backpressure still blocks the sender. A live source pays one channel item per packet (the kill path reacts to single packets); a file source sends batches of 128 through the same channel, one item per batch, with the slot pool divided to keep the same in-flight packet cap
batch main loop the same capture::channel wrapper, not a bare bounded channel: batch.rs receives on a capture::channel::PacketRx exactly as the TUI processor does
--cores reader โ†’ workers, file input crossbeam bounded::<Vec<Packet>>(64) carrying batches of 128 (run_offline_parallel_file()) โ€” same ~8192 in-flight packet cap, one channel hop per 128 packets instead of per packet
--cores per-file reader โ†’ dispatcher, multi-file input crossbeam bounded carrying one batch per item (shard_set_parallel()). The queue depth is not the bound that matters: READ_AHEAD_BYTES caps in bytes how far a LATER file's reader may run ahead of the file the dispatcher holds, because a batch is 128 packets of any size and the default snaplen is 65535
scanner-kill request / response crossbeam bounded(256) from the capture thread to scanner-kill-fwd, then pipes to and from the worker process, then bounded(256) for the outcome stream (process_isolation.rs)
packet path โ†’ rtpengine reconciler std::sync::mpsc::sync_channel(1024) (orphan_channel()) โ€” the capture path offers a relay-side socket and never waits on it: a full queue drops the offer and counts it, because stalling packet processing on a relay round trip is the trade --rtpengine-control exists to avoid
DNS resolve queue std::sync::mpsc (names.rs)
inside api/mcp servers tokio (axum/rmcp internals)
โš ๏ธ **GitHub.com Fallback** โš ๏ธ