Internals Index - NormB/sipnab GitHub Wiki

Documentation for people changing sipnab, not people running it.

  • Operator documentation lives one level up in docs/.
  • The polished site is sipnab.com.

Pages here link directly into the source tree. That is deliberate.

  • Docs and code share one repository.
  • So a claim about the code should be one click from the code.
  • A moved file or renamed function should fail the build rather than rot quietly.
  • tests/dev_docs_drift_test.rs enforces it.

Start here

A reading order, not a table of contents:

  • Domain primer — the SIP and RTP model the code assumes you already have.
    • Start here if you are a Rust engineer rather than a VoIP engineer.
    • Nearly every subtle bug in this tree is a protocol-semantics bug wearing a Rust costume.
  • Subsystem guide — one packet's journey from the wire to the screen, across all four packet paths.
  • Invariants — the rules that must not break.
    • Read before your first pull request.
    • Each entry names what enforces it.
  • Testing — the test tiers and the self-enforcing gate tests.
    • Read when one of them fails you.
  • Walkthroughs — ordered checklists for the common changes: a new TUI view, a new detector, a new CLI flag, a new MCP tool.
  • Build, CI and release — features, workflows, hooks, and how to cut a release.

Already written, and narrower:

  • Threading — thread topology, the channels between them, and the lock discipline.
  • Zero-copy payloads — the bytes::Bytes spine from capture to output, including a performance claim the page itself refutes.
  • The rtpengine control plane — naming media captured on a standalone relay. It covers:
    • the ng wire format;
    • why a CAPTURED Call-ID arrives over HEP rather than off the control socket;
    • the RelayControl action and the six appliers it names;
    • how a paired positive and negative capture together prove the claim.
  • Uprobe and eBPF TLS capture — reading SIP plaintext out of a process's TLS library with kernel uprobes, and recovering the peer with a real eBPF program. It covers:
    • banded fetches;
    • the wipe that keeps adjacent heap from leaving;
    • why this input can never transmit;
    • the one thing tracefs cannot do without eBPF.
  • TUI testing — snapshot and state testing for the terminal UI.
  • The vCon exporter — writing one observed dialog as a vCon container. It covers:
    • which sipnab type feeds each section;
    • the completeness caveat carried in two surfaces from one value, and the test that fails if they diverge;
    • the deterministic UUIDv8 and its collision window;
    • a credential filter that removes nothing today on purpose.

Other starting points:

The corpus: Live vs archaeological

The long design documents live in docs/design/ and docs/research/. The codemap is one level up in docs/ itself.

  • They are not all current.
  • Reading the wrong one as current is the main trap here.

Live:

Document What it is
../architecture.md The codemap: module layout, data flow, and the design decisions that still hold. Maintained; a phantom flag in it fails docs_drift_test.
../design/maintainability-perf-spec.md The rationale behind the current shape of the code — why one pipeline replaced four, why main.rs broke up into src/app/. Sections 0–9 are the 2026-07-03 review of v0.4.18 and read as history; the unnumbered section at the end, "WS8 — 0.5.16 benchmark re-validation follow-ups", is the only live section — read it before any performance work.
docs/design/backlog.md How the backlog works, and the four states an item can be in. The working list itself is local and uncommitted (docs/design/backlog.local.md), so this page carries the convention rather than the items.
../design/lessons.md Four defects that reached a release, each with the rule derived from it: TUI state no renderer read, feature flags gating nothing, config parsed and never used, and the 2026-05-05 audit that found four blocking and ~17 major doc drifts accumulated since 0.3.1. Its cheap-regression greps still hold, though one of them carries a field count that has drifted: it calls 30 current, while FIELD_NAMES in ../../src/sip/dsl.rs lists 33 and parse_field accepts every one of them. Count the constant, not the comment.
../research/codex-analysis.md Adversarial security review of 698585e (2026-07-22). Findings SN-01/02/03, all fixed; the analysis of why each was reachable is still the best description of the HEP trust boundary.
../research/capture-performance.md The packet-capture throughput roadmap: four phases ordered cheapest-first, each after the first carrying an explicit trigger condition so the complexity is only paid once the previous phase proves insufficient. Research, not committed work — one item carries a done mark, the auto-grow capture channel. Its baseline section names symbols rather than line numbers, and says why: the ranges it cited had all rotted by the time the work below it landed.

Archaeological — kept for the determination record, superseded in places:

Document Read it for Do not trust
../design/implementation-plan-v6.md The design-decision catalog D1–D21 and the original phase plan. Its feature tables. The tls-wolfssl, tls-openssl and grpc features described there were never implemented and are not in Cargo.toml. D14's pluggable crypto backend never happened: one backend ships (ring + rustls).
../design/implementation-plan-phases-8-10.md The MCP, HEP and observability designs, and the "Resolved Decisions" section that formally retires parts of v6. Its D-numbering. It defines its own D20 (infrastructure-optional integration) and D21 (capture vs enrichment sources), which collide with v6's D20 and D21. Always say which document a D-number comes from.
../design/compact-headers-spec.md Why sipnab accepts all 19 RFC 3261 / IANA compact header forms, and the y: STIR/SHAKEN evasion case that motivated it. Nothing — the code implements it and tests pin it.
../design/kill-target-spoofing-spec.md The scope and ethics of sending a scanner-kill response from the victim's ip:port rather than an ephemeral one. Nothing — but read --kill-scanner's guard rails in cli.rs alongside it.
../design/dialog-tracking-modes.md Why --dialog-track keys on Call-ID or on Call-ID plus top-Via branch, what the branch view costs everything downstream of the store, and the rejected alternatives. Nothing outstanding. Its status line read "spec, not yet implemented" for six releases after the flag shipped in 0.5.54 — recorded here and left — and now reads IMPLEMENTED. an_unimplemented_design_doc_does_not_name_a_shipped_flag fails if a design doc calls itself unimplemented while naming a flag Cli accepts, so this column no longer has to carry that job.

Glossary

Identifiers that appear in commit messages, code comments, and the backlog without expansion.

D1–D21 — design decisions. The catalog in ../design/implementation-plan-v6.md. The ones cited most often in code:

  • D2 (synchronous core, async only at the edges — the packet path in pipeline.rs never awaits)
  • D3 (zero-copy payload spine)
  • D10 (feature gates keep the binary small)
  • D11 (key material is toxic waste — crypto.rs zeroizes)
  • D13 (RTP is first-class: stream_store.rs discovers streams with no SIP at all)
  • D15/D16 (privilege drop and process isolation)
  • D17 (warn and continue on malformed input)
  • D18 (localhost default for every listener)

Beware the numbering collision noted above.

D22, D23, and D24 also exist, but only in ../design/implementation-plan-phases-8-10.md. v6's catalog stops at D21.

  • D22 is competitive-feature-borrowing discipline. Its prompt-injection rule is the one cited in src/mcp/server.rs.
  • D23 makes documentation a tier-1 deliverable. It lands in the same pull request as the code it describes.
  • D24 makes tests a phase-completion gate. That is why every code-bearing sub-phase in that plan carries a Tests — X.Y deliverables block beside its Gate and Docs blocks.

WS0–WS8 — workstreams. The refactor program in ../design/maintainability-perf-spec.md.

  • WS0 was a batch of independent quick wins.
  • WS1 unified the per-packet pipeline into the single classify_packet() router.
  • WS2 decomposed main.rs into src/app/.
  • WS3–WS5 were structural and performance work.
  • WS6–WS7 hardened the API surface.
  • WS0–WS7 shipped in v0.5.0.
  • WS8 (performance) is the only live section.

P0–P5 — backlog priority tiers, used by the local backlog and by the identifiers in commit messages:

  • P0 panics and security
  • P1 wrong results in real use
  • P2 robustness and efficiency
  • P3 code health
  • P4 test quality
  • P5 features and exploratory work

A "P1" in a commit message means the commit fixed something that produced a wrong answer. It does not mean that it was merely important.

SN-01/02/03 — security findings from ../research/codex-analysis.md, all fixed:

  • SN-01: unauthenticated HEP metadata driving active network responses.
    • This is the reason parse_packet() tracks HEP origin per packet.
    • It is the reason scanner-kill refuses HEP-origin packets without --hep-allow-kill.
  • SN-02: an unauthenticated non-loopback metrics bind.
  • SN-03: crash-report creation that followed symlinks.

The gate suite — the self-enforcing checks that run without anyone asking.

The numbered gates in .githooks/pre-commit:

  • formatting, Vale and codespell;
  • clippy;
  • the full test suite;
  • no unwrap()/expect() or abort macro in production;
  • WASM exports in sync;
  • the homepage test count still generated rather than hand-set;
  • sub-gate 5b for the site version — a different claim from the crate version;
  • no TODO stubs;
  • an advisory notice when a commit touches code these pages cite.

The TODO scan and that notice are the two advisory gates. They print WARN/REVIEW and let the commit through.

Version markers are not in that list.

  • One Rust test asserts them.
  • It runs here and in CI.
  • The reason: two implementations of one rule diverge. The shell copy the hook once carried did.

Also twelve in .githooks/pre-push, each marked # -- Hard gate in the hook:

  • fmt;
  • clippy --workspace --all-features --all-targets;
  • cargo doc with -D warnings;
  • a fuzz workspace check;
  • the release-delivery tests;
  • the reduced feature combinations;
  • the non-Linux arm of every platform cfg;
  • the YANG module and its RFC 7951 exports;
  • the test count a commit message claims;
  • the refusal to tag v* at a commit whose CI is not green;
  • the prose linters;
  • a zola build of the website.

Plus the conditional corpus gate, and the CI jobs behind them.

The drift tests — the subset of the gate suite that compares documentation and configuration against the code and fails on divergence:

They exist because every one of them was, at some point, a doc that lied.

The smoke fuzz floor — tests/smoke_fuzz_test.rs.

  • It feeds tens of thousands of random and mutated inputs to every parser reachable from attacker-controlled bytes.
  • It does this under catch_unwind, on the stable toolchain, in every cargo test run.
  • It is the always-on regression floor beneath the coverage-guided targets in fuzz/fuzz_targets/. Those need nightly and run weekly.
  • A panic there is a remote DoS on a capture process. So the floor is not optional.

Conventions for these pages

  • Cite code as a link, never as file:line.

    • Line numbers rot within a commit.
    • A path plus a ()-suffixed symbol in the link text survives.
    • dev_docs_drift_test checks both.

  • Relative links only.

    • An absolute github.com/NormB/sipnab/blob/main/… URL pins a branch and goes stale silently.
    • build-wiki.py rewrites the relative form into a blob URL when publishing to the wiki.

  • Write heading anchors in GitHub's spelling.

    • That is the spelling readers of docs/ see.
    • The generators translate on the way out.
    • build-site-pages.py rewrites an anchor to Zola's slug when it emits a site link.
    • The reason: the two renderers disagree.
      • GitHub drops an em dash and keeps its surrounding spaces (step-0--install-…). Zola collapses the run (step-0-install-…).
      • GitHub keeps an underscore where Zola makes it a dash.
    • generated_site_anchors_resolve_under_zola checks the generated tree under Zola's rule alone.
    • The older anchor_candidates unions all three slug rules. That is right for docs/ and too generous for a page only Zola ever renders.

  • Diagrams are mermaid sequenceDiagram, and a prose line precedes every one that carries the same point. So a page still reads where mermaid does not render.

  • A change to linked code updates the page that links it, in the same pull request. The hard gate is dev_docs_drift_test in CI.

  • These pages publish twice, and this tree is the source of both.

    The same arrangement covers the operator pages.

    • build-site-pages.py renders each entry in its PAGES registry from docs/ into website/content/docs/.
    • site_pages_mirror_is_current gates that.
    • Two entries' sources sit outside docs/: CONTRIBUTING.md and contrib/README.md. The script resolves each page's relative links from that page's own directory.
    • That same script also writes llms.txt and llms-full.txt into website/static/.
      • It builds them from ALL the published pages, docs/internals/ included.
      • llms_aggregates_are_current gates them.

    Which script owns them is the trap.

    • build-site-internals.py does not touch the aggregates.
    • Say you edit an internals page and regenerate only the internals mirror.
      • That satisfies the gate that checks the mirror.
      • It leaves the aggregates stale.
    • Run both generators, or run build-site-pages.py last.

    Every page in that registry got there the same way. Each was hand-maintained on both sides until they diverged. Read the registry for what it holds today, not this sentence.

    • The cookbook shared 2 of its 36 commands with the site copy.
    • The REST API page was 430 lines against the site's 893. Each side held sections the other lacked.
    • The MCP page was 672 lines against the site's 440. Its tool table listed 7 of the 11 registered tools. The site's listed all 11.
    • The rest were the same story. The site's Filter DSL page carried fourteen operational recipes docs/ did not have. So every wiki reader got that page without them.

    benchmarks.md is the deliberate exception.

    • Both copies exist, and no generator writes either one.
    • A gate covers only the measured tables (benchmark_tables_match_between_docs_and_website). The framing around them should differ.
    • The wiki renders from docs/. So it showed whichever copy was thinner as though it were the whole page.
    • Register a page there. Never copy the script.

    The site mirror exists because GitHub's wiki mermaid viewer pins its controls over the diagram. There is no way to move them. The site renders the same diagrams with a viewer this repo controls.