Internals Testing - NormB/sipnab GitHub Wiki

The tests are of three kinds:

  • Over two hundred integration-test binaries under tests/.
    • Every tests/*.rs file is one.
    • That is because nothing turns Cargo's auto-discovery off.
  • Unit tests inside src/.
  • Doctests.

The exact total is not repeated here on purpose.

  • The homepage stat card carries it.
  • A pre-commit gate holds it.
  • So any number written on this page would be wrong within a week.
  • The opening sentence read "fifty" until 2026-09-06.
  • On that date a sweep of these pages counted 238.

What follows is the map:

  • What each tier asserts.
  • How to regenerate its artifacts.
  • Most importantly, the roster of gate tests.
    • These fail you for reasons that have nothing to do with the code you were writing.

Tiers

Tier Files Asserts
CLI surface cli_test, cli_options_test, cli_help_test, cli_defaults_test, cli_flag_behavior_test Parsing, the default value of every parameter, --help grouping across ~160 flags, and behavior for flags that were once untested.
CLI golden cli_goldens .trycmd goldens under tests/cli/ β€” exact stdout/stderr for a command line, run by tests/support/golden_cases.rs (the trycmd crate's format, without the crate). tests/cli/cookbook/ pins the output of every cookbook command check-cookbook.py executes, one case per distinct command, run in a scratch directory because recipes write files.
Integration integration_test, capture_test, rtp_integration_test, pipeline_test, parse_path_test, bootstrap_test, app_servers_test, config_test Capture-to-output pipeline, and the library facades WS2 extracted from main.rs β€” bootstrap planning is a pure Cli + Config β†’ RunPlan function precisely so these tests can reach it. config_test covers the step before planning: config discovery (-f, SIPNAB_CONFIG, --no-config, unknown-key warnings, the missing-file error) driven through the real binary's --dump-config.
TUI tui_snapshot_test, tui_state_test, tui_e2e_test Rendered buffers via ratatui's TestBackend + insta; state-machine transitions; and end-to-end drives of the real binary inside tmux. See TUI testing.
Servers api_test, api_token_test, mcp_stdio_test, mcp_http_test, mcp_token_test, mcp_token_rotation_test, metrics_test, hep_test REST, MCP (stdio and HTTP), signed-token auth and rotation, Prometheus scrapes, HEP ingestion β€” all end to end against a spawned process.
Security security_test, privilege_drop_test, resource_bounds_test, crash_test Audit regressions, the never-continue-as-root guarantee, attacker-keyed map caps, and crash-report handling with the real binary.
Property & fuzz property_test, smoke_fuzz_test, fuzz_corpus_replay proptest invariants; the always-on stable-toolchain fuzz floor; and β€” despite its name β€” a deterministic sweep over an adversarial seed set defined in the file itself, not a replay of fuzz/corpus/ (it opens no files).
Contract json_schema_test, summary_consistency_test, output_behavior_test, error_types_test, api_guidelines_test, wasm_exports_test The shapes other people's code depends on: JSON schemas, cross-surface summary agreement, machine-readable output flags, typed errors, #[non_exhaustive] on growth-prone enums, WASM export list.
Docs enforcement docs_drift_test, dev_docs_drift_test, link_integrity_test, doc_example_coverage_test, config_examples_test, site_journey_test, mockup_alignment_test See the gate roster below.
Governance flag_coverage_test, keybinding_drift_test, feature_gate_test, config_wiring_test Rules about how the project grows, not about a specific behavior.
Meta support_selftest The shared test helpers have their own tests.

Files in a subdirectory of tests/ are not compiled as their own test binaries. So an explicit path attribute pulls shared helpers in:

#[path = "support/run.rs"]
mod run;

That idiom is why the same module appears in several test files with no mod.rs chain. Each test binary compiles its own copy.

Helper Provides
mod.rs Shared normalization (normalize()) used to compare output across platforms; self-tested by support_selftest.
run.rs The canonical binary-spawn helper for CLI/output/config integration tests β€” one place that knows how to find and invoke the built binary. It runs the binary with HOME, XDG_CONFIG_HOME and SIPNAB_CONFIG pointed into an empty directory, so no developer's own config reaches a test. config_isolation is that list, and cli_goldens uses it too.
server.rs REST API spawn harness: start the server on an ephemeral port, wait for readiness, tear down.
mcp.rs The same for HTTP MCP, including the JSON-RPC framing.
teardown.rs terminate(), the one way a harness stops the binary it spawned: SIGTERM, a bounded wait, and SIGKILL only for a child still running after it. Child::kill() is SIGKILL, and a process killed that way never writes its coverage profile. Pulled in with include! rather than #[path].
src/test_scratch.rs ScratchDir, a test directory under the system temp directory with a random name, removed when the guard drops, on a failing test as well as a passing one. The library's unit tests use it as crate::test_scratch; integration tests mount the same file with #[path = "../src/test_scratch.rs"]. temp_path_isolation_test refuses a temp path named by the process id or by a fixed name.
schema.rs JSON-Schema validation against tests/schemas/.
tui_fixtures.rs SIP fixture builders shared by the TUI snapshot and state tests.
executable.rs write_executable(), the one way a test writes a file it then runs: a child process writes it, so this process never holds it open for writing and an exec cannot fail with ETXTBSY ("Text file busy") while another test thread forks. executable_stub_test refuses a file written in-process and then made executable.
child_lifetime.rs tied_to_test(), the one way a test starts a process: call it on the Command before .spawn(). On Linux it sets PR_SET_PDEATHSIG to SIGKILL, so the child ends when the spawning thread ends, even when a signal ends the test process and no Drop runs. See Children end with their test. Pulled in with include!.
root_run.rs root_run::sudo(), the one way a test starts a command under sudo. The kernel clears PR_SET_PDEATHSIG when sudo executes, so a pipe ties the command to the test process: it ends when the test process ends. See Children end with their test.
fuzz.rs A deterministic xorshift PRNG shared by the stable-toolchain fuzzers, so a failure reproduces from its seed.
ports.rs refused_tcp_port(), the one way a test gets a loopback TCP port where a connect fails with ConnectionRefused: a socket bound and not listening, held until the guard drops, so no test running alongside can bind the number. Binding :0, reading the port, and closing the socket leaves the number free for another test to listen on; released_port_scan_test refuses that pattern under tests/.

Children end with their test

A process a test starts ends when that test ends, however the test ends (TEST-CHILD-ORPHAN).

Why. The helpers stop their child in Drop. Drop does not run when something kills the test process: SIGKILL, the kernel's OOM killer, or a harness that signals the test's pid and not its process group. A serving sipnab left that way keeps running, with init as its parent, until someone stops it.

Prerequisites. Linux, for the guarantee. Other systems have no equivalent of PR_SET_PDEATHSIG, so tied_to_test() does nothing there and Drop is the only stop.

How to start a child.

include!("support/child_lifetime.rs"); // once per file, at module level

let child = Command::new(env!("CARGO_BIN_EXE_sipnab"))
    .args(["-N", "-I", "capture.pcap", "--api", "127.0.0.1:0"])
    .tied_to_test()
    .spawn()?;
  • Spawn the child on the thread that holds it. The kernel sends the signal when the spawning THREAD ends, so a child started on a helper thread dies as soon as that thread returns.
  • The child of a parent that died between the fork and the prctl sees a different parent, and its spawn fails with ESRCH instead of running untied.
  • The kernel clears the setting when the child executes a set-user-ID program (sudo) or changes its user or group id (--user). A pipe ties a run under sudo instead: see the next list.
  • The child's own children are not tied: a fork clears the setting.

How to start a command under sudo.

#[path = "support/root_run.rs"]
mod root_run; // once per file

let out = root_run::sudo()? // `sudo -n`, tied to this test process
    .arg(env!("CARGO_BIN_EXE_sipnab"))
    .args(["-N", "-I", "capture.pcap", "--user", "nobody"])
    .output()?;
  • root_run::sudo() is the one place under tests/ that names sudo. The test process holds the write end of one pipe until it ends. The command runs under a small sh program, as root, that holds the read end. When the test process ends, by any cause, that program reads end-of-file and sends the command SIGTERM, and SIGKILL 2 s later.
  • The command keeps its arguments, output and exit code. A command that a signal ended exits with code 128 plus the signal number.
  • SIGTERM, SIGINT or SIGHUP sent to the sudo process reaches the command as SIGTERM.
  • The command's standard input is /dev/null.
  • root_run::tied() runs a command the same way without sudo.

How to verify. cargo test --features full --test child_lifetime_test.

  • an_api_server_ends_when_its_test_process_is_killed, and the same for the HTTP MCP server and the fleet center, run a child role that holds the server, SIGKILL the role's process, and require that the server has ended within 10 s.
  • every_child_a_sipnab_test_spawns_is_tied_to_it reads tests/: every .spawn() in a file that starts the sipnab binary, and every .spawn() in tests/support/, must have .tied_to_test() before it. The exceptions are a command from root_run::sudo() or root_run::tied() and the race test that ties its child by hand.
  • a_sudo_command_ends_when_its_test_process_is_killed and a_pipe_tied_command_ends_when_its_test_process_is_killed run a child role that holds a command, SIGKILL the role's process, and require that the command has ended within 10 s. The first needs a sudo that asks for no password (sudo -n true) and prints a NOTICE line where there is none.
  • only_the_tying_helper_names_sudo reads tests/: only root_run.rs names sudo.

Troubleshooting.

  • these spawns are not tied to their test: add .tied_to_test() before the .spawn() it names, and include!("support/child_lifetime.rs") to the file (include!("child_lifetime.rs") in a file under support/). In a nested module, add use super::TiedToTest as _;.
  • sites that name sudo: a test names sudo directly. Start the command with root_run::sudo()? and drop the -n argument, which the helper passes.
  • A child that ends early with signal 9 while its test still runs was started on a thread that has since ended. Start it on the test's thread.

Fixtures and corpora

Every command below ran against the current tree and left it unchanged.

  • That is the point of documenting them.
  • If one produces a diff, something has drifted.
Artifact Regenerate with
tests/fixtures/ β€” capture fixtures Every capture in the directory comes from tests/support/synthetic_captures.rs, and so do the two SIPp media files in harness/sipp/scenarios/ and the two fuzz seeds in fuzz/corpus/pcap_reader/ named truncated-sip and empty-classic: seventeen in all. cargo run --features native --bin gen_fixture writes them, relative to the CARGO_MANIFEST_DIR Cargo sets (the current directory when run outside Cargo), and synthetic_captures_test fails when a committed byte differs from what the generator builds. The directory's other fixtures, the JSON goldens, ng replies and TLS keys, stay as checked in. tests/PROVENANCE.md records where every capture outside tests/pcap-samples/ came from, and every_committed_capture_is_public_or_synthetic refuses one with no entry, finding captures by their leading bytes rather than their names.
tests/snapshots/ β€” TUI buffers cargo insta test --features tui --accept (needs cargo install cargo-insta)
tests/cli/ β€” .trycmd goldens TRYCMD=overwrite cargo test --features full --test cli_goldens (rewrites only the cases that do not match)
tests/cli/cookbook/ β€” cookbook output goldens python3 scripts/check-cookbook.py --bless writes a command-only case for each executed cookbook command that has none and deletes each case no command produces; then TRYCMD=overwrite cargo test --features full --test cli_goldens records the output. A case's file name is a hash of its command, and it is also the directory the command's output files go to.
.config/implies-no-tui.txt β€” flags that imply -N SIPNAB_BLESS_IMPLIES_NO_TUI=1 cargo test --features full --lib -- the_implies_no_tui_file. Generated from Cli::implying_no_tui, the table Cli::normalize reads; check-cookbook.py reads it to tell a command that raises the TUI (exit-only, no golden) from a headless one such as --count-by without -N (pinned by a golden).
tests/schemas/ β€” JSON Schemas Hand-maintained; json_schema_test validates output against them.
tests/pcap-samples/ β€” capture fixtures Thirteen are synthetic: python3 tests/gen-pcap-samples.py writes ten and python3 tests/gen-link-type-samples.py writes the three link-layer framings (DLT_LOOP, PPPoE inside Linux cooked capture v1 and v2); each script in check mode reports any of its own that have drifted from it. Nothing regenerates the rest; they stay as checked in. New ones arrive through harness/scripts/capture.sh then promote.sh, which record the run in PROVENANCE.md beside them β€” every_committed_capture_fixture_says_where_it_came_from refuses a fixture with no entry, and the 36 that predate that rule sit on an enumerated list which only shrinks.
tests/install-sh/ β€” installer cases Hand-maintained; exercised by the install-sh CI job.
fuzz/corpus/ β€” fuzz seeds Grown by cargo fuzz run (nightly). Note nothing on the stable toolchain reads this directory: fuzz_corpus_replay drives its own in-file seed set. To make a reproducer run in every cargo test, add it to smoke_fuzz_test as well.

The public vCon datasets

The vcon-dev community publishes five vCon datasets as git repositories.

  • They hold 17,443 containers in all.
  • None of them carries SIP signaling.
  • tests/fixtures/vcon-datasets/PINS.tsv pins each one to a commit.
    • It records the license of its LICENSE file at that commit.

Two tests read them.

  • vcon_dataset_subset_test runs on every build with the vcon feature.
    • It reads the four containers committed under tests/fixtures/vcon-datasets/.
    • It pins the findings vcon_schema::validate reports for each.
    • It also checks three more things:
      • The jsonschema reference engine agrees.
      • Each container is valid once the test repairs its recorded dataset issues.
      • The fetcher (--vcon-fetch-kind generic, against a server on 127.0.0.1) saves each one unchanged.
    • Each dataset directory there carries the LICENSE file of its dataset.
    • It also carries a README with the source path and SHA-256 of every file.
  • vcon_dataset_corpus_test reads every container in the five datasets.
    • It runs only when SIPNAB_VCON_DATASETS names a directory that scripts/fetch-vcon-datasets.py filled.
    • Without it, the binary prints one NOTICE line saying its gates did not run, as the capture corpus does.
    • The pre-push corpus gate runs it, with the shell's SIPNAB_VCON_DATASETS, whenever that gate runs. The pre-commit test pool runs with the variable unset.

Fetch the datasets and run the corpus test:

# Run all of these, in order.
python3 scripts/fetch-vcon-datasets.py "$HOME/vcon-datasets"
SIPNAB_VCON_DATASETS="$HOME/vcon-datasets" \
  cargo test --features full --profile corpus --test vcon_dataset_corpus_test

The script runs only git.

  • It fetches each pinned commit.
  • It checks it out detached, with hooks and symbolic links turned off.
  • It refuses a checkout whose HEAD is not the pinned commit.
  • It refuses a checkout whose files have changed.
  • A second run verifies the cache and moves a dataset whose pin changed.
  • The five datasets take about 2.3 GB of disk.
  • The corrected copy described below takes about 2.2 GB more.

The corpus test prints three things for each dataset:

  • The number of containers.
  • The number the schema accepts.
  • Each finding, with the number of containers that carry it.

It asserts these things:

  • Each dataset is at its pinned commit.
  • sipnab and the reference agree on every container.
    • They agree on the verdict.
    • They agree on the instance paths of the findings.
  • The per-dataset counts measured on 2026-10-09.
    • These hold for as long as the pins do.
    • Moving a pin in PINS.tsv means measuring again and rewriting EXPECTED in the test.

What the five datasets get, at their pins:

Dataset Containers Valid Findings, with the number of containers that carry each
vcon-supreme-court-arguments 8,503 0 Attachment Object without mediatype: 8,503
ietf-meeting-vcons 8,181 0 Attachment Object without mediatype: 8,181. Dialog Object with url and no content_hash: 4,079
vcon-dataset-city-of-newport-ri 115 0 Attachment Object without mediatype: 115. Dialog Object with url and no content_hash: 115
fake-vcons 601 0 Attachment Object without mediatype: 601. Attachment Object without start: 601. Dialog Object with a body and no encoding: 264
tadhack-2025 43 0 Attachment Object without mediatype: 43. Attachment Object without start: 43

Each finding is a rule that draft-ietf-vcon-vcon-core-04 states in its prose and in its schema:

The reference engine reports the same findings at the same paths. With those four repaired, every one of the 17,443 containers is valid to sipnab and to the reference engine.

The corrected copy

tests/fixtures/vcon-datasets/CORRECTED.tsv pins NormB/vcon-datasets-core04.

  • It is a copy of the first three datasets.
  • It adds two kinds of key and changes nothing else:
    • mediatype on every inline attachment.
    • content_hash on the 425 IETF audio recordings whose files it downloaded, with an IETF login, and hashed.
  • Its CHANGES.md lists what it leaves open and why.
  • fake-vcons and tadhack-2025 are not in it.
    • The reason is the contact data described in the subset README.
  • The fetch script fetches both pins files unless --pins names others.
  • It accepts --pins more than once.

The corpus test holds the corrected copy to three things. These are beside the tests above that pin the upstream findings.

  • It is at its pinned commit.
  • Each corrected dataset gets the answer measured on 2026-10-11.
    • sipnab and the reference agree on every container.
  • Each corrected container is its upstream container with keys added.
    • The test counts the added content_hash values (425).
    • It requires each to be a SHA-512 token.
    • With them set aside, repairing both containers with the test's own repair gives the same document.
    • The corrected one needs no mediatype repair.
    • That last check compares this repository's repair rule with the corrected repository's own, on all 16,799 containers.
Dataset Containers Valid Findings, with the number of containers that carry each
vcon-supreme-court-arguments 8,503 8,503 None
ietf-meeting-vcons 8,181 4,527 Dialog Object with url and no content_hash: 3,654
vcon-dataset-city-of-newport-ri 115 0 Dialog Object with url and no content_hash: 115

The remaining finding needs a hash of each recording. No fetch returns these recordings to hash.

  • The IETF ones are YouTube pages.
  • The Newport video host refused the download.

Accepting a snapshot or overwriting a golden is a decision, not a fix.

  • Read the diff first.
  • These files are the record of what the tool promised its users.

The gate-test roster

These fail on changes you thought had nothing to do with them. Each one exists because the thing it guards silently rotted at least once.

Gate Trips when
docs_drift_test A --flag named on a published page does not exist in the CLI. That corpus derives from git ls-files rather than a hand-kept list, and covers every tracked docs/ page except design/ and research/, plus website/content/, README.md, SECURITY.md and CONTRIBUTING.md β€” these pages included, so a phantom flag here fails too. A flag belonging to another tool goes in FOREIGN_FLAGS, scoped to the label of each page naming it. Also: a version marker in the docs or man page disagrees with Cargo.toml; the README feature table misses a Cargo feature; a [theme] slot in ThemeConfig has no documentation, or the slot count quoted in either config reference is wrong. Also the benchmark reproducibility contract: the bench/ harness the benchmarks page tells readers to run must exist and be executable, bench/carrier.py must still produce the corpus composition the page quotes (checked at 1/100 scale), and both doc trees must name the same measured artifact and date.
dev_docs_drift_test A page under docs/internals/ links to a path that no longer exists, names a fn that no longer exists, uses an absolute GitHub URL instead of a relative one, is not registered in build-wiki.py, or breaks a mermaid convention. It also builds the wiki and fails if any relative link survives into the output β€” the wiki is flat and has no repo tree, so such a link publishes dead. That check runs the generator rather than reading it: the version that only asserted CODE_LINK_RE appeared in the script passed while ](../bench/) was shipping broken.
link_integrity_test Any relative link or heading anchor in either doc tree does not resolve; Zola content uses a plain relative .md link that would render as a dead URL. On the wiki-source side the scan is the top-level docs/*.md plus docs/internals/ β€” docs/design/ and docs/research/ are planning material outside the published journey and are deliberately not walked, though a link into them from a scanned page still has to resolve. A third scope covers the root community files (README.md, SUPPORT.md, MAINTAINERS.md, CONTRIBUTING.md, SECURITY.md, CODE_OF_CONDUCT.md), which neither doc tree walks β€” these are what GitHub renders in the sidebar, so a rename that broke their cross-references used to go unnoticed.
doc_example_coverage_test A user-facing CLI flag appears in fewer than two documented examples. A ratchet β€” the exemption list may only shrink.
flag_coverage_test A new long flag ships with no test referencing it. Also a ratchet: adding a test for a grandfathered flag fails until you remove it from the baseline list.
keybinding_drift_test A controller handles a key that the F1 help never mentions.
config_wiring_test A config key exists but is never read, or a CLI flag has no config fallback where its peers do.
feature_gate_test A flag whose subsystem is not compiled in fails late or silently instead of fast and clearly.
api_guidelines_test A growth-prone public enum loses #[non_exhaustive], or a shared store stops being Debug.
summary_consistency_test Two serializing surfaces disagree about a dialog or stream field.
wasm_exports_test The WASM binding loses an export the browser analyzer calls.
site_journey_test / mockup_alignment_test A website journey breaks, or a terminal mockup on the site drifts from real output. Also the homepage's advertised numbers: the binary-size ceiling, the glibc floor, the throughput tiles (which must quote a figure that appears on the benchmarks page they link to), and the automated-test count, which the template must read from the file pages.yml generates rather than state as a number. It holds the standards cards to a canonical table naming the code symbol behind every item, the field the program emits for every quality metric, and a citation in docs/ and the built site mirror, one card per standard. It also holds the homepage's four MCP examples to the files gen-mcp-examples.sh generated, and holds each of those to the claim the surrounding copy makes about it. That covers the second half of live binary -> website/data/mcp-examples/*.json -> index.html; demos/gen-mcp-examples.sh --check covers the first and needs a built binary, so CI cannot run it. The homepage's voice-stack section must link exactly the guides in the "Your SIP server" and "Add to your voice stack" groups of the docs navigation, so a guide added to one reaches the other.
config_examples_test A config sample in the docs no longer parses.
doc_commands_run_test A command shown in docs/, the README or a hand-written page under website/content/ is one sipnab refuses. It extracts every line that starts with sipnab from every shell block, including a fence indented under a list item, through the same CommonMark fence rule as scripts/lib_markdown.py (it checks the two parsers agree on every page), and RUNS 447 of the 496 against a capture that ships with the repository, several at a time, each in a directory and a home of its own, failing on a usage error rather than on a non-zero exit -- plenty of documented commands exit non-zero for honest reasons. A device or root example runs with a device name that cannot exist substituted for the one the page names, so its flags still parse in full; a server example runs under a shared wall-clock bound, because clap refuses in milliseconds and a process still alive after it necessarily parsed its arguments. Only two kinds never run, and it names both: a shell program rather than one invocation, and a command that would exec something of its own or attach to the kernel. A command matching neither fails the test. It prints its own coverage, because the difference between "352 ran" and "3 ran, 380 skipped" is the difference between a gate and a decoration.
examples_run_test A program in examples/ no longer runs the way its own comment says, or prints something other than the output the comment shows. It runs every cargo run command the comments document (12, across all seven programs), except one under sudo, and holds docs/library.md's count of the programs to the directory.
release_surface_test A release adds a flag, MCP tool or REST route that no release note names, or a guide's Tested on names a sipnab release older than a flag its commands use. It reads the three lists at every release tag, with extractors it first holds to clap and the MCP router.
homepage_claim_truth_test A claim the homepage states in PROSE stops being true: the MCP tool counts and their read-only split, which released builds carry the bpf feature, what the installer actually puts on a host, a capability the static musl build does not have, the filter language's field and operator counts, a TUI key the page names that nothing binds, and a CLI flag the page names that does not exist, plus the unsafe count and the safety claim resting on it β€” the page said "memory-safe by construction" beside 88 unsafe blocks and no forbid(unsafe_code). site_journey_test gates the tiles; this gates the sentences, because the two disagreed on one page for months.
cookbook_recipes_test A command in docs/examples.md no longer runs or names a flag --help lacks; or a command it executes has no output golden in tests/cli/cookbook/ and no reason in the checker's OUTPUT_UNPINNED table; or a golden there is one no cookbook command produces.
support_selftest The shared normalization helper changes behavior under the tests that depend on it.
released_port_scan_test A test under tests/ binds a loopback port 0, reads the port, and releases the socket before using the number, which a test running alongside can take in between.
pipefail_early_exit_test A tracked shell script, git hook or workflow run: block pipes into a reader that can stop before its writer finishes (grep -q, grep -m, grep -l, head, awk with exit, sed with q, read, a while read loop that breaks) in a shell with pipefail on. The writer then dies of SIGPIPE and the pipeline fails although the reader matched. It names the file and line. The matcher, tests/support/pipefail_scan.rs, reads quoting, comments, here-documents, case patterns, [[ ]] and GitHub's per-step shell, and fails on a defaults: block or a file that mentions pipefail in a format it does not scan. The safe forms are in CONTRIBUTING.md.

The rule for all of them: the gate is not the problem.

  • If flag_coverage_test fails, the flag needs a test.
  • If dev_docs_drift_test fails, a page now lies.
  • Adding an exemption is the last resort.
  • Every exemption list in this repo works as a ratchet, so it cannot quietly grow.

The client example programs

Three checks cover the programs under clients/ that the user pages quote.

  • client_snippets_test holds every block marked with a comment such as <!-- snippet: clients/python/mcp_probe.py#sse-data-lines --> on docs/rest-api.md, docs/client-examples.md, the metrics and MCP pages and the hand-written website/content/docs/api-clients.md to the marked region of its program, byte for byte. It also requires scripts/smoke-clients.sh to run the site's whole clients, the capability examples and the operator tasks.
  • example_parity_test holds the lifecycle table on docs/client-examples.md to the programs in clients/.
  • scripts/smoke-clients.sh runs every program end to end on loopback, against sipnab servers replaying captures in the repository. CI runs it after the build step that builds the binaries and examples. Each program must print its exact output lines and must fail on a failure case: a wrong token, a missing call or customer, a capture with nothing to judge, or no TFPS.

What the script does that a reader of the user page does not need to:

  • It waits for evidence, never a fixed sleep: source_exhausted in /v1/stats before it queries a replaying server, and packets_admitted in /v1/hep/senders before it reads the HEP roster. sipnab discards input it has not processed when it stops.
  • TFPS bans by writing a pinned BPF map as root, which no CI runner allows. The ops server runs with --tfps-ctl naming clients/python/tests/fake_tfps_ctl.py, which answers in the documents tfps_ctl --json prints. clients/python/tests/test_fake_tfps_ctl.py holds it to the fixtures pinned against the real binary. The script checks that the stand-in's block list holds the two banned addresses.
  • evidence_handoff.py runs twice over one capture into two file roots, and the script requires the same output and byte-identical files. The absolute path each answer carries is the one difference, and the program prints it relative to its root. The README and manifest name the sipnab version, so the script compares two runs rather than pinning hashes.
  • agent_triage.py runs over stdio and then over HTTP, against a server started on loopback port 0 with a signing key generated at run time and --mcp-tools core. The signed token must print the same lines as the stdio run. A token signed with another key, a token minted from the same key for the REST API, and a request with no token must each get 401.
  • customer_export.py reads three files cut from sip-problem-call.pcap the way a wrapped tcpdump -C -W ring leaves them, and capinfos and tshark read the export.
  • The BPF half of --uprobe-tls needs root and a kernel with BTF, so no CI runner can run it. examples/tls_plaintext_records.rs runs the decode half on records laid out as the BPF program publishes them.

The development loop

Logging. SIPNAB_LOG is a tracing EnvFilter.

  • So it takes levels and per-module directives (SIPNAB_LOG=debug, SIPNAB_LOG=sipnab::rtp=trace).
  • In TUI mode sipnab suppresses logging unless SIPNAB_LOG says otherwise.
    • The alternative is log lines painting over the interface.
  • -q lowers the default in CLI mode.
  • Configured by init_logging().

Benches. cargo bench.

  • The release profile unwinds, as bench units do, so the cdylib crate type needed for the WASM build and the benches share one panic strategy.
  • --profile profiling adds full symbols, for a profiler.
  • The profiling profile is release codegen with panic = "unwind" and debug symbols kept.
    • That is also what callgrind wants.
  • Baselines live in benches/BASELINES.md.

The pin gates, before the hook. Some tests hold a count of something the repository tracks: pages, links between them, code links, workflows, error templates. A new page, script or workflow moves those counts, and the tests that hold them run among 16,000 others at the end of the pre-commit hook, about 25 minutes in. scripts/pin-gates.py runs only those tests, in under a minute once their binaries exist.

  • Purpose: meet every count pin a change moves in one short run, before the hook.
  • Prerequisites: cargo-nextest at the version the hook pins, and the change staged. The tests list files with git ls-files, which reads the index, so a file that is not staged is not counted.
  • Steps: git add <paths>, then python3 scripts/pin-gates.py. On the shared build host, run it through heavy-run --quick.
  • Verify: the run ends with the suite's own summary lines and exit status 0. A failing test prints the count it found and the constant to change.
  • Troubleshoot: exit status 2 with untracked, so not counted by any gate names files to git add; --allow-untracked runs anyway, for a scratch file that stays out of the commit. Exit status 2 with neither run nor excluded means a new test counts tracked files: add its name to .config/pin-gates.txt.

The script runs the tests the way the hook does: the same nextest profile, feature set and environment, so it builds no binary the hook does not build. It names each binary to cargo with --test. A nextest filter alone would build every test target first. scripts/tests/test_pin_gates.py holds the list to the tree: a test under tests/ that declares a count constant (a usize whose name begins EXPECTED_), or reads the list of tracked files, must appear in the list, as a binary to run or as an exclusion with a reason.

Where a hook run spent its time. The pre-commit hook appends one line per step to sipnab-pre-commit-steps.log in the checkout's git directory: the run, the step under the name its heading prints, whole seconds and the exit status, separated by tabs. The last line of a run is the step total, counted from the start of the hook. scripts/hook-steps.py prints the table.

  • Purpose: read which step of the hook costs the time, from a file the hook already wrote, with no extra hook run.
  • Prerequisites: one full hook run in this checkout since the hook gained the log. A run that a pass stamp skips whole writes no line.
  • Steps: python3 scripts/hook-steps.py prints the newest run; --runs N prints the newest N runs, oldest first; --mean N prints each step's mean and longest time over the newest N runs that passed; --log FILE reads another checkout's log.
  • Verify: the table ends with the line total, and its first line says passed, or FAILED at with the step that failed. The line (outside the steps) is the time before the first heading and between steps: reading the pass stamps is most of it.
  • Troubleshoot: NOT ENDED in the first line means the run has no total line: a signal ended the hook, or the hook is still running. Exit status 1 with cannot be read means no full run has written the log yet. A step that printed CACHED is in the log with the seconds its stamp read took.

The step Tests holds the build of the test binaries, the nextest run and the doctests together. The hook cuts the log to its newest 2,000 lines once it passes 4,000, and a line that it cannot write does not fail the hook. A run whose suite passed also leaves a note in the log, a line of three fields: the run, the word nextest, and the Summary line nextest printed (the tests run, passed and skipped). hook-steps.py prints it under the table of its run, so two passing runs compare by their totals. Before, the hook kept a summary only for a failed run. scripts/tests/test_hook_steps.py runs the hook in its sandbox and reads the log. scripts/tests/test_hook_steps_table.py holds the table.

nextest. The suite runs under cargo-nextest, in the pre-commit hook and in CI, through scripts/parallel-tests.py.

  • cargo test bounds no single test, so one hung test held the hook until its 30-minute watchdog fired, with no name attached. nextest's slow-timeout stops a test after three 60-second periods and fails it by name.
  • .config/nextest.toml defines three profiles:
    • default (no retries, 60 s slow timeout, stopped after 180 s), for local runs.
    • suite, what parallel-tests.py runs: the same deadline, no retries, no fail-fast, and a JUnit report. Three tests have a deadline of their own. every_documented_command_runs_or_says_why_not, whose binary took 191.85 s on CI's macos-latest runner, has six periods. test_pre_commit_sh_leaves_the_hooks_repository_alone and test_pre_push_sh_leaves_the_hooks_repository_alone each run a whole hook harness: about 45 s of processor time in short processes, one after another. Their run time is that work divided by the share of a core the host gives them, so no number of seconds fits every load: the suite stopped both at 180 s at a load average of 44 to 50, and at 300 s at a load average of 38. The two tests end a harness themselves when its log stops growing for 120 s or passes 4 MiB (HookLimits in tests/hook_test_scripts_git_env_test.rs), which does not depend on the load. Their nextest deadline is 30 periods, the 30 minutes of the hook's own watch on the suite, and only covers a failure of that watch.
    • e2e (2 retries, 60 s timeout) for the timing-sensitive tmux tests.
  • nextest does not run doctests, so parallel-tests.py runs them after it with cargo test --doc.
  • The script renders nextest's JUnit report as cargo test's text, one block per test binary. The hook's failure report and scripts/published-test-count.py read that text unchanged.
  • nextest-version in the config pins the version exactly. parallel-tests.py and scripts/check-hook-prerequisites.py refuse any other version, and name the install command: cargo install cargo-nextest --locked --version 0.9.148.
  • CI installs it through .github/actions/nextest: the release tarball, checked against its SHA-256, from the actions cache or, on a miss, from GitHub.
  • Measured on 2026-11-06, on one tree, in the host's heavy-run slot, against the pool it replaced, which ran cargo test's binaries side by side: the test step took 202 s and 158 s under nextest against 164 s and 129 s, plus 11 s of doctests.

Rate limits in tests. A test of a fixed-window rate limiter does not count on the wall clock to keep a burst inside one window.

  • The REST limiter's windows are one second long. A burst of 101 requests at the shipped cap of 100 fits in one window only if the host sends it in under a second.
    • Measured on 2026-11-08 against sipnab --api, at a load average of 50 to 88 on 14 cores: in 5 of 20 bursts the window ended mid-burst and the first 503 came at request 122 to 200 instead of 101. Unloaded, the limiter refused 5 of 5 bursts at request 101.
    • rate_limiter_rejects_when_per_ip_budget_exhausted sent such a burst and failed once in CI that way.
  • Unit tests build the limiter with RateLimiter::with_clock and move the clock themselves, so only the test decides when a window ends.
  • A test of the real clock, in-process or against the binary, sends at most three requests at a cap of 1 and times them. A span shorter than one window touches at most two windows, so three requests in it put two in one window, and the limiter must refuse one of them. The test fails with the measured span if the host took a whole window for them.

The docker lab. harness/ is a docker-compose stack for generating real traffic.

  • It holds OpenSIPS, rtpengine and SIPp.
  • make up in that directory builds and starts it.
  • make down tears it down.

WASM. One pre-commit gate covers the browser analyzer. It checks that website/static/wasm/sipnab.js still exports every function the site calls.

A second gate once demanded a freshly built bundle alongside any staged src/wasm.rs.

  • It no longer exists, nor does the binary it guarded.
  • The published analyzer went eleven releases stale while that gate stayed green.
    • The reason: src/wasm.rs had no commits in the window.
    • The interface held still while the implementation behind it moved.
  • The Pages workflow builds the bundle at deploy time now.
    • So the build produces what ships, rather than the tree carrying it.
    • It cannot go stale.

Feature matrices. A green cargo test --features full is not proof.

  • CI also builds reduced feature sets.
  • Code behind #[cfg(not(feature = "tui"))], or any other cfg(not(feature)), is invisible to the full build.
  • For exactly this reason, the pre-push hook runs two checks before pushing:
    • cargo clippy --workspace --all-features --all-targets -- -D warnings.
    • The fuzz workspace check.

Failure-path coverage

The tests and audits behind the operator fault model. That page says how sipnab behaves under each fault class. This section records what pins each behavior.

Parser panic surface

Two harnesses exercise every parser reachable from packet or file bytes:

  • Coverage-guided fuzzing: fuzz/fuzz_targets/ (cargo-fuzz / libFuzzer).
  • Continuous fuzzing: the same targets run under ClusterFuzzLite from .clusterfuzzlite/.
    • Batch fuzzing and corpus pruning run daily. A coverage report runs weekly.
    • The difference from the weekly run above is the corpus. It persists between runs instead of restarting from the tracked seeds. So what libFuzzer learns accumulates instead of starting over.
  • Always-on smoke fuzz: tests/smoke_fuzz_test.rs runs in cargo test (no nightly needed).
    • It runs ~40k random + structurally mutated inputs per entry point under catch_unwind.
    • It covers the same parser set plus:
      • the full link-layer decap chain (parse_packet across several link types),
      • the pcap file reader.
    • A caught panic fails the test. The offending input is hex-dumped for a repro seed.
    • Calling every fuzz entry point also compile-checks their signatures. So a target that drifts out of step with its parser breaks the test build. It does not drop out of the fuzz suite unnoticed.
    • Example: fuzz/fuzz_targets/sip_parser.rs had passed a &str transport argument where parse_sip takes TransportProto. The whole suite stopped compiling, with nothing in cargo test to say so.

The smoke layer is the regression floor. It found a UTF-8 char-boundary panic in the TLS keylog hex decoder (src/capture/tls.rs). Nobody had ever run the committed fuzz target against that decoder.

  • The decoder checked byte-length parity, then sliced &hex[i..i+2] as a str.
  • So a multi-byte UTF-8 char (for example €, 3 bytes) split by the 2-char window panicked. The message was "byte index is not a char boundary".
  • That was a remote DoS via a crafted SSLKEYLOGFILE line.
  • The decoder now works on raw bytes with explicit nibble validation instead. Any non-ASCII or non-hex byte gives a clean parse error naming its offset.
  • The regression test decode_hex_multibyte_utf8_does_not_panic and the corpus seeds in fuzz/corpus/keylog_line/ pin it there.

Contained panics: the fuzzers look for panics. These tests hold what happens when one gets through anyway (A parser that panics). No known real input makes these parsers panic, so each test hands its reader a parser that does:

tests/contained_entry_points_test.rs holds the wiring: each reader's entry point reaches its contained half, and only one place under src/capture calls crash::contain. The SIP and RTP readers outside it reach that place through crate::capture::contained.

Property tests: tests/property_test.rs (proptest) asserts the semantic invariants the fuzzers cannot:

  • a SIP message built from generated fields parses back to those fields,
  • SDP survives a buildβ†’parseβ†’rebuild round trip,
  • the filter DSL is a total function on arbitrary text (parse then evaluate never panic).

Verified bounded in code (confirmed by audit, exercised by

fuzz): SIP header count (≀200) and fold size (≀8 KB); websocket payload (≀64 KB); TLS record length (≀18432); IP/GRE encapsulation depth (≀5); RTP CSRC / RTCP report counts (5-bit fields); all length-driven

Vec::with_capacity sites bounded by the actual slice.

  • RTP/RTCP length-field arithmetic (u16 * 4) cannot overflow usize.
  • Each multiply carries a bounds ensure! after it.

Resource bounds

Store Bound proven by
DialogStore tests/resource_bounds_test.rs (50k unique Call-IDs, both modes, len() ≀ cap at every step)
StreamStore same test (50k unique SSRCs)
per-dialog messages dialog_store.rs unit tests
per-stream audio frames stream_store.rs
IP/TCP reassembly reassembly.rs

Capture, sink and concurrency audit

Audited, found sound (true-positive findings: none):

  • Capture file (pcap_reader.rs):
    • Magic/length/block-truncation all return clean errors.
    • Tests cover them (too_short_file, invalid_magic, truncated_epb_block_no_panic). The smoke fuzz also exercises them.
    • The FILE path implements the same exit-status rule as the live path. It did not always.
      • Reading a set deliberately continues past a truncated member and returns Ok.
      • So the join never saw it, and a truncated pcap exited 0 with a whole-looking report.
    • ReadTally::report in src/capture/file.rs is the one place both the single-threaded and --cores readers converge on.
      • It records the same lost predicate that decides its log severity into the run-integrity record in src/output/run_integrity.rs.
      • The exit-code sites and the --json-dialogs/--report markers read that record.
      • One place emits both the human sentence on stderr and the machine record. So they cannot disagree.
    • A --plugin that would not load is the same class and lands the same way (exit 1, plugins.failed in the record).
    • The same record also carries idle compaction. Idle compaction deliberately does NOT move the exit status. A retention policy doing what you configured it to do is not a failed run.
  • Live loop (capture/live.rs):
    • Device-open and BPF-compile failures return clean errors via the ready channel.
    • Receiver-dropped breaks cleanly.
    • A transient recv() error is fatal to the capture thread, and untested.
    • The thread ends by returning the error rather than panicking.
    • The exit status is the only place the partial-read distinction survives. Downgrade that join to a warning and exit 0, and an incomplete run reads exactly like a whole one to anything checking $?.
  • event_exec: sipnab reaps every child, and a test covers the escaping.
  • Log/JSON sinks:
    • serde escaping.
    • \r\n stripped from alert log values (log-injection guard, tested).
    • Writes use ?, no unwrap.
  • Shutdown:
    • Atomic-flag signal handlers (signals.rs, tested).
    • parking_lot locks cannot poison.
    • Closed-channel sends checked.
  • unsafe: libc and other FFI calls, with RAII/Drop-guarded fd ops.
    • Every block states its own soundness argument.
    • Cargo.toml sets clippy's undocumented_unsafe_blocks to warn.
    • Both the pre-push hook and CI run clippy with -D warnings. So a block with no adjacent // SAFETY: comment fails the build.

Untested failure paths

  • Live-capture transient recv error: needs a fault-injecting capture source.
  • Interface-down mid-capture: funnels into the same fatal recv path.
  • Opus decode (rtp/opus_decode.rs) delegates to the external libopus. Its FFI input is length-checked. The codec itself is not fuzzed here.
  • No disk-full test for the pcap/wav writers. Writes propagate ?, so there is no panic, but no test covers the error path.

Test matrix: dialog times (updated_at and latest_message_at)

A dialog keeps two times of its newest activity. updated_at is the timestamp of the message processed last, and the live vCon export and the tail cursors read it to detect change. latest_message_at is the newest timestamp among the dialog's messages, and durations, idle compaction, the active counts and the newest-first sorts read it. They differ when sipnab reads an older-stamped message after a newer one, as on an -I set whose files overlap in time. Which dialog time to read is the user-facing how-to.

The tests, by what they check. SUCCESS: a reader gives the right answer. FAILURE: sipnab refuses a wrong answer, or it does not happen. EDGE: a boundary or a degenerate input.

Where SUCCESS FAILURE EDGE
src/sip/dialog_store.rs (store, compaction, active counts, call tree, tail page, merge oracle) 9 6 7
src/output/model.rs (DialogSummary) 3 0 0
src/sip/dsl.rs (duration) 1 1 0
src/output/dialog_report.rs and src/output/call_report.rs 4 0 0
src/security/fraud_detect.rs (short calls) 1 1 0
src/sip/endpoint.rs and src/sip/sdp_origin.rs 2 0 0
src/tui/mod.rs and src/tui/call_list.rs (Duration column and sort) 2 0 0
src/app/batch.rs (live vCon export) 1 1 0
tests/latest_message_at_test.rs (CLI, worker counts, schema) 5 4 4
tests/openapi_contract_test.rs 1 0 0
tests/cores_parity_test.rs (idle compaction at --cores 1, 2, 4, 8) 0 1 2
Total, 56 29 14 13

The cases the matrix covers:

  • Files out of order and overlapping. The two-file fixture callid_across_files in tests/support/synthetic_captures.rs: sixteen calls whose newest messages are in the file read first. Every call lasts 2.505 s; read to the last message it lasted 0.905 s.
  • Identical timestamps, in one file and across two.
  • A single message.
  • A late retransmission with an older timestamp: still counted, and the duration stays at the newest message.
  • A dialog read newest first: no negative duration.
  • Idle compaction at exactly 600 s from the newest message (not compacted), one millisecond past it (compacted), and past 600 s only from the message read last (not compacted), in one store and end to end at --cores 1, 2, 4 and 8. The --cores merge oracle and the shortcut of the merge (compaction_due) cover the last case.
  • The live vCon export writing a call again after a message stamped older than its newest. This is the guard for keeping updated_at the last message read.
  • The output schema: tests/schemas/dialog.schema.json and the OpenAPI DialogSummary component require latest_message_at as a date-time, and refuse a summary without it or with a number in it.

Existing gates hold the field on every surface as well: every_remaining_schema_agrees_with_the_projection_it_describes in tests/json_schema_test.rs, the_dialog_summary_component_declares_exactly_what_the_projection_emits in tests/openapi_contract_test.rs, every_dialog_summary_field_is_classified_as_fenced_or_verbatim in tests/mcp_untrusted_fencing_test.rs, and the merge oracle's observable, which compares latest_message_at across every assignment of a call's legs to workers.

Test matrix: readers that stop early under pipefail

Under set -o pipefail, a pipe into a reader that stops early fails the pipeline when the writer still has output: the writer dies of SIGPIPE (141). The 2026-11-08 sweep found 53 such sites in tracked scripts and workflows and changed every one. The gate and its matcher keep the pattern out.

The tests, by what they check. SUCCESS: the gate accepts a safe form, or it runs correctly. FAILURE: the gate flags an unsafe form, or it loses the race. EDGE: text that only looks like a pipe, or a position that must stay exact.

Where SUCCESS FAILURE EDGE
tests/pipefail_early_exit_test.rs (matcher, workflow shells, the repository gate) 17 30 19
tests/pipefail_early_exit_test.rs (the race: each old form fails all 200 runs, each new form passes all 200) 5 5 0
tests/tag_signature_gate_test.rs (the release check with a git that prints 1 MiB after the signature) 1 1 0
Total, 78 23 36 19

The race tests use a writer that prints MATCH and then 1 MiB, sixteen times a pipe's capacity, so the outcome does not depend on timing. 256 KiB was not enough: mawk 1.3.4 reads that far ahead before it acts on exit.

Test matrix: engines asked at once, and the refilling budget

Every question sipnab puts to all listed rtpengine engines asks them at once (src/relay/fan_out.rs), and each engine's reconciliation budget refills (src/relay/budget.rs). What an engine that does not answer costs and The control-transaction budget, and how it refills are the user-facing how-to.

The tests, by what they check. SUCCESS: sipnab asks the engines and they answer, or the budget grants what it should. FAILURE: an engine is dead or answers cut short, a budget refuses, or a read fails. EDGE: answers out of order, limits, clock readings out of order, and degenerate inputs.

Where SUCCESS FAILURE EDGE
src/relay/fan_out.rs (ask_each: order, concurrency, threads, panics) 4 1 4
src/relay/budget.rs (the token bucket) 5 5 8
src/relay/reconcile.rs (refill, forgetting, startup at once) 6 4 7
src/relay/engines.rs (the lists of an unseen call) 1 2 1
src/rtpengine/control.rs (statistics_of_each, read_until) 2 4 2
src/output/query/relay_holdings.rs and relay_sessions.rs 0 3 1
src/output/api.rs, src/mcp/tools/relay.rs and src/tui/relay_stats.rs 0 3 0
src/app/batch.rs (--relay-stats-interval) and src/app/relay_reconciler.rs (closing line) 0 1 2
Total, 66 18 23 25

The cases the matrix covers:

  • Dead engines. The fake engines are local UDP sockets (test_engines::FakeEngine in src/test_engines.rs): one answers, one never answers, one answers late, one answers with a reply cut short. Three dead engines and a live one cost one client timeout (1.5 s in the tests), checked as at least one timeout and less than two. Asked in turn they cost three.
  • Asked at once, without timing. A gate that opens only when every engine is waiting at it (test_engines::Gate): asked in turn, the first engine waits alone and gives up.
  • Out of order. Engines that answer last first; the answers stay in the configured order on every surface.
  • All dead, a partial answer, and one engine's budget spent while the others keep theirs.
  • The budget: empty, refilled, at its cap after a week or a century, never above the burst, the part of a token already earned carried, a reading earlier than the last, no refill, and a run of three hours that the budget grants exactly the burst plus three hours of refill. The tests drive a hand-moved Instant; nothing waits for a refill.
  • A read a signal interrupts: the client reads again with the rest of its timeout (read_until). Under the full library suite a signal reached the control sockets' threads, the read failed with Interrupted system call (os error 4), and 1.5 s waits ended after 0.7 s. That run found the defect.
⚠️ **GitHub.com Fallback** ⚠️