Internals Threading - NormB/sipnab GitHub Wiki

The topology below is the reality on main today.

• The core principle is design decision D2.
• The packet path is synchronous threads + channels.
• tokio exists only inside the optional servers.

                       ┌──────────────────────────────────────────────────┐
                       │                 shared state                     │
                       │  Arc<RwLock<DialogStore>>  Arc<RwLock<StreamStore>>
                       │        (parking_lot, single writer)              │
                       └───────▲──────────────────────────▲───────────────┘
                        write  │                     read │ (try_read in TUI)
capture thread(s)              │                          │
  live device / file / HEP     │                          │
  (capture::start_multi_capture)                          │
        │ Packet               │                          │
        ▼                      │                          │
  capture::channel  ────► processing thread ──────► TUI event loop (main thread)
  (capped channel)        (TUI mode: pipeline::            crossterm events, render
                           process_packet;                 TUI MODE ONLY
                           batch mode: main.rs loop
                           on the main thread)

  auxiliary threads, by the RUN MODE that starts each — none of them is a
  child of the TUI event loop:

  both modes
    ├── server runtime thread          app::servers::start_servers
    │     one shared current-thread tokio runtime hosting every enabled
    │     async server as a task: api (axum), mcp (rmcp), mcp-http
    ├── metrics-server thread          app::servers::start_servers
    │     raw std::net::TcpListener accept loop + one short-lived
    │     metrics-conn thread per scrape
    ├── DNS resolver thread            app::build_resolver
    │     std::mpsc queue; spawned only when reverse DNS is enabled
    └── rtpengine reconciler           app::relay_reconciler::spawn
          bounded hand-off queue; starts only when --rtpengine-control
          names a relay on a live run

  batch mode only
    └── scanner-kill worker            process_isolation::spawn_scanner_kill_worker
          a separate PROCESS (this binary, re-executed), started by
          bootstrap::launch before the chroot and privilege drop; two
          threads here serve its pipes; --kill-scanner has no TUI path

The mode column above is the part to keep true. It is worth saying why.

• An earlier revision of this diagram drew every auxiliary thread as a child of the TUI event loop.
• Not one of them is.
  • Both run modes start four of them.
  • Batch alone starts the scanner-kill worker.

That is not a cosmetic error.

• --metrics really did ship wired to the TUI path alone.
• So every headless -N deployment got no metrics at all.
  • That is every container and systemd unit.
• This page said nothing that would have contradicted it.
• A diagram can group threads by who spawns them rather than by which mode runs them.
  • Such a diagram reads as reassurance.
  • It hides exactly that class of defect.

The four spawn sites, none of which is the TUI event loop:

• start_servers() for the server runtime and the metrics listener.
• build_resolver() for the reverse-DNS worker.
• spawn_scanner_kill_worker() for the kill worker.
• relay_reconciler::spawn() for the rtpengine reconciler.

The Prometheus listener is not a task on the shared tokio runtime. The distinction matters when reasoning about blocking.

• It is a raw TcpListener accept loop on its own thread.
• It is deliberately independent of tokio and axum.
  • So metrics stay scrapable when the async servers are not compiled in at all.

Each accepted scrape runs on a short-lived metrics-conn thread.

• A ConnGate bounds those to 16 in flight.
• Beyond that, new connections get an immediate 503 rather than a thread (SN-02, CWE-770).
• start_metrics_server() spawns the thread itself.
• start_servers calls it for both run modes.

Every long-lived thread and its spawn site. A name in a backtrace maps straight back to a row here.

Modes is the column to trust when asking "does this run headless?"

• That is the question --metrics got wrong.
• both means the thread exists under -N as well as in the TUI.
• TUI and batch mean it does not.

Every fatal exit taken after start_capture() has to go through stop_and_join(). There is no second correct way out.

The reason is where the thread starts.

• bootstrap::launch() spawns it before these steps:
  • The readiness hand-shake.
  • The chroot.
  • The privilege drop.
• The reason: the capture source must be open while the process still holds CAP_NET_RAW.
• Every failure from that point on happens with a capture thread already running and holding its source. Such failures include:
  • An unopenable source.
  • An unusable --chroot.
  • A refused privilege drop.
  • A hardening step the kernel rejects.
  • A companion server that cannot start.
• std::process::exit joins nothing and runs no destructors.
  • So exiting directly abandons it.

stop_and_join does three things:

• It sets the global shutdown flag.
• It drops the packet receiver.
• It joins.

Both signals matter, and neither is redundant.

• Dropping the receiver ends a thread blocked on a send.
• The flag ends one blocked elsewhere in its loop.
  • A live capture waiting out its read timeout reaches the flag check first.
• One read never times out.
  • libpcap's netmap module waits inside pcap_next_ex until a frame arrives.
  • So on a silent link the flag check is never reached.
  • For that read, the capture's capture-stop:<device> thread sees the same flag and calls pcap_breakloop.
  • The loop takes the broken read as a stop.
• The join has no timeout.
  • That matches the one the batch receive loop already performs at end of capture.
• A HEP listener blocked on its socket returns in milliseconds rather than hanging it.

One consequence applies to callers that cannot reach the handle.

• BatchRunner::new() does not own it.
• So its four fatal paths return a PlanError instead of exiting.
• batch::run holds both the handle and the receiver. It does the teardown.
• A function that cannot clean up must hand the failure to one that can.

How a gate holds this.

• ThreadSanitizer treats thread leak as a fatal finding, not a warning.
• cli_flag_behavior_test exercises both shapes:
  • A source that never opens.
  • A failure after it opened.
• So a regression fails there, rather than waiting for the weekly sanitizer run.
• Before this rule, sipnab -I /nonexistent.pcap (a mistyped filename) leaked a thread.

Opening a pcap from inside the TUI spawns a second writer.

• The pcap-load worker writes the same stores the render thread is reading.
• That is why every render-side access is try_read() and never read().

sequenceDiagram
    autonumber
    participant UI as TUI event loop
    participant Load as pcap-load worker
    participant DS as DialogStore
    participant SS as StreamStore

    UI->>Load: spawn on O (file open dialog)
    UI->>DS: try_read for the frame
    DS-->>UI: guard, or skip the frame
    Load->>DS: write lock
    DS-->>Load: release
    Load->>SS: write lock
    SS-->>Load: release
    Load-->>UI: progress via async_messages
    Note over UI,SS: the render pass never blocks — a contended try_read skips a frame

--cores N offline mode replaces the single processing thread with a dispatcher + N workers.

• The dispatcher does a cheap host-pair peek.
• It shards raw packets over bounded channels.
• Each worker owns a private PacketProcessor + thread-local DialogStore/StreamStore.
  • There are no locks on the hot path.
• The stores merge at EOF.
• Flow correctness holds because a flow's packets share a host pair.
  • Therefore they share a worker.
• A call's SIP and its RTP can run between different hosts, and so on different workers.
  • The dispatcher also sends a copy of each packet that can carry SDP to every other worker, at the same place in that worker's sequence.
  • A worker reads a copy only for the media sockets its SDP names, and counts nothing from it.
  • So each worker's StreamStore sees a later call take over a socket and an SSRC at the point in the capture where it did, and starts that call's stream there, as the single-threaded path does.

• Parse outside the lock.
  • SIP/RTP/SDP parsing happens before any store lock appears.
  • classify_packet() touches no store at all.
  • Each store is then locked for writing once per packet.
  • That makes writes the most frequent lock operation in the process, not the rarest.
• Lock ordering:
  • When a path needs both stores, dialog store first, then stream store.
  • If it also needs the alert engine, that comes last.
  • The ordering is what carries the safety. The guards are not always disjoint.
  • Two paths each take one store at a time and release it before the other. "briefly" is accurate there.
    • process_packet() on the live path.
    • run_pcap_load() for the file-open worker above.
  • The batch applier (batch.rs) does not work that way.
    • It holds both write guards across the entire per-packet body.
  • This page previously said both write locks were never held at once. That was never true of the batch path.
• The guards queue side effects rather than performing them.
  • These accumulate while the code holds the guards:
    • Alert findings.
    • Per-message output.
    • The --alert-exec / --on-dialog-exec / --on-quality-exec spawns.
  • DeferredEffects::drain replays them after both drop.
  • So none of these happens inside the critical section:
    • A fork/exec.
    • A stdout write.
    • An AlertEngine lock.
  • Until that change, the batch loop took the alert engine's write lock nested inside both store guards.
    • That was a third lock deep in the hot path, with a posix_spawn beside it.
  • The full rule and what would re-create that edge are in Invariants, "Dialog before stream, then alerts — one consistent order".
• The TUI never blocks: all render-side store access is try_read().
  • On contention the frame renders with the previous data.
  • Counts may be one frame stale. This is deliberate.
  • An adaptive 10 fps active / 2 fps idle tick bounds staleness.
• Pause is an AtomicBool checked by the processing thread. No lock.
• mcp/ denies clippy::await_holding_lock.
  • parking_lot guards must never live across an .await.