Internals Testing - NormB/sipnab GitHub Wiki

The tests are of three kinds:

  • Over two hundred integration-test binaries under tests/.
    • Every tests/*.rs file is one.
    • That is because nothing turns Cargo's auto-discovery off.
  • Unit tests inside src/.
  • Doctests.

The exact total is not repeated here on purpose.

  • The homepage stat card carries it.
  • A pre-commit gate holds it.
  • So any number written on this page would be wrong within a week.
  • The opening sentence read "fifty" until 2026-09-06.
  • On that date a sweep of these pages counted 238.

What follows is the map:

  • What each tier asserts.
  • How to regenerate its artifacts.
  • Most importantly, the roster of gate tests.
    • These fail you for reasons that have nothing to do with the code you were writing.

Tiers

Tier Files Asserts
CLI surface cli_test, cli_options_test, cli_help_test, cli_defaults_test, cli_flag_behavior_test Parsing, the default value of every parameter, --help grouping across ~160 flags, and behavior for flags that were once untested.
CLI golden cli_goldens trycmd goldens under tests/cli/ — exact stdout/stderr for a command line. tests/cli/cookbook/ pins the output of every cookbook command check-cookbook.py executes, one case per distinct command, run in a scratch directory because recipes write files.
Integration integration_test, capture_test, rtp_integration_test, pipeline_test, parse_path_test, bootstrap_test, app_servers_test, config_test Capture-to-output pipeline, and the library facades WS2 extracted from main.rs — bootstrap planning is a pure Cli + Config → RunPlan function precisely so these tests can reach it. config_test covers the step before planning: config discovery (-f, SIPNAB_CONFIG, --no-config, unknown-key warnings, the missing-file error) driven through the real binary's --dump-config.
TUI tui_snapshot_test, tui_state_test, tui_e2e_test Rendered buffers via ratatui's TestBackend + insta; state-machine transitions; and end-to-end drives of the real binary inside tmux. See TUI testing.
Servers api_test, api_token_test, mcp_stdio_test, mcp_http_test, mcp_token_test, mcp_token_rotation_test, metrics_test, hep_test REST, MCP (stdio and HTTP), signed-token auth and rotation, Prometheus scrapes, HEP ingestion — all end to end against a spawned process.
Security security_test, privilege_drop_test, resource_bounds_test, crash_test Audit regressions, the never-continue-as-root guarantee, attacker-keyed map caps, and crash-report handling with the real binary.
Property & fuzz property_test, smoke_fuzz_test, fuzz_corpus_replay proptest invariants; the always-on stable-toolchain fuzz floor; and — despite its name — a deterministic sweep over an adversarial seed set defined in the file itself, not a replay of fuzz/corpus/ (it opens no files).
Contract json_schema_test, summary_consistency_test, output_behavior_test, error_types_test, api_guidelines_test, wasm_exports_test The shapes other people's code depends on: JSON schemas, cross-surface summary agreement, machine-readable output flags, typed errors, #[non_exhaustive] on growth-prone enums, WASM export list.
Docs enforcement docs_drift_test, dev_docs_drift_test, link_integrity_test, doc_example_coverage_test, config_examples_test, site_journey_test, mockup_alignment_test See the gate roster below.
Governance flag_coverage_test, keybinding_drift_test, feature_gate_test, config_wiring_test Rules about how the project grows, not about a specific behavior.
Meta support_selftest The shared test helpers have their own tests.

tests/support/

Files in a subdirectory of tests/ are not compiled as their own test binaries. So an explicit path attribute pulls shared helpers in:

#[path = "support/run.rs"]
mod run;

That idiom is why the same module appears in several test files with no mod.rs chain. Each test binary compiles its own copy.

Helper Provides
mod.rs Shared normalization (normalize()) used to compare output across platforms; self-tested by support_selftest.
run.rs The canonical binary-spawn helper for CLI/output/config integration tests — one place that knows how to find and invoke the built binary.
server.rs REST API spawn harness: start the server on an ephemeral port, wait for readiness, tear down.
mcp.rs The same for HTTP MCP, including the JSON-RPC framing.
teardown.rs terminate(), the one way a harness stops the binary it spawned: SIGTERM, a bounded wait, and SIGKILL only for a child still running after it. Child::kill() is SIGKILL, and a process killed that way never writes its coverage profile. Pulled in with include! rather than #[path].
schema.rs JSON-Schema validation against tests/schemas/.
tui_fixtures.rs SIP fixture builders shared by the TUI snapshot and state tests.
executable.rs write_executable(), the one way a test writes a file it then runs: a child process writes it, so this process never holds it open for writing and an exec cannot fail with ETXTBSY ("Text file busy") while another test thread forks. executable_stub_test refuses a file written in-process and then made executable.
fuzz.rs A deterministic xorshift PRNG shared by the stable-toolchain fuzzers, so a failure reproduces from its seed.
ports.rs refused_tcp_port(), the one way a test gets a loopback TCP port where a connect fails with ConnectionRefused: a socket bound and not listening, held until the guard drops, so no test running alongside can bind the number. Binding :0, reading the port, and closing the socket leaves the number free for another test to listen on; released_port_scan_test refuses that pattern under tests/.

Fixtures and corpora

Every command below ran against the current tree and left it unchanged.

  • That is the point of documenting them.
  • If one produces a diff, something has drifted.
Artifact Regenerate with
tests/fixtures/ — capture fixtures Every capture in the directory comes from tests/support/synthetic_captures.rs, and so do the two SIPp media files in harness/sipp/scenarios/ and the two fuzz seeds in fuzz/corpus/pcap_reader/ named truncated-sip and empty-classic: seventeen in all. cargo run --features native --bin gen_fixture writes them, relative to the CARGO_MANIFEST_DIR Cargo sets (the current directory when run outside Cargo), and synthetic_captures_test fails when a committed byte differs from what the generator builds. The directory's other fixtures, the JSON goldens, ng replies and TLS keys, stay as checked in. tests/PROVENANCE.md records where every capture outside tests/pcap-samples/ came from, and every_committed_capture_is_public_or_synthetic refuses one with no entry, finding captures by their leading bytes rather than their names.
tests/snapshots/ — TUI buffers cargo insta test --features tui --accept (needs cargo install cargo-insta)
tests/cli/ — trycmd goldens TRYCMD=overwrite cargo test --features full --test cli_goldens
tests/cli/cookbook/ — cookbook output goldens python3 scripts/check-cookbook.py --bless writes a command-only case for each executed cookbook command that has none and deletes each case no command produces; then TRYCMD=overwrite cargo test --features full --test cli_goldens records the output. A case's file name is a hash of its command, and it is also the directory the command's output files go to.
tests/schemas/ — JSON Schemas Hand-maintained; json_schema_test validates output against them.
tests/pcap-samples/ — capture fixtures Thirteen are synthetic: python3 tests/gen-pcap-samples.py writes ten and python3 tests/gen-link-type-samples.py writes the three link-layer framings (DLT_LOOP, PPPoE inside Linux cooked capture v1 and v2); each script in check mode reports any of its own that have drifted from it. Nothing regenerates the rest; they stay as checked in. New ones arrive through harness/scripts/capture.sh then promote.sh, which record the run in PROVENANCE.md beside them — every_committed_capture_fixture_says_where_it_came_from refuses a fixture with no entry, and the 36 that predate that rule sit on an enumerated list which only shrinks.
tests/install-sh/ — installer cases Hand-maintained; exercised by the install-sh CI job.
fuzz/corpus/ — fuzz seeds Grown by cargo fuzz run (nightly). Note nothing on the stable toolchain reads this directory: fuzz_corpus_replay drives its own in-file seed set. To make a reproducer run in every cargo test, add it to smoke_fuzz_test as well.

The public vCon datasets

The vcon-dev community publishes five vCon datasets as git repositories.

  • They hold 17,443 containers in all.
  • None of them carries SIP signaling.
  • tests/fixtures/vcon-datasets/PINS.tsv pins each one to a commit.
    • It records the license of its LICENSE file at that commit.

Two tests read them.

  • vcon_dataset_subset_test runs on every build with the vcon feature.
    • It reads the four containers committed under tests/fixtures/vcon-datasets/.
    • It pins the findings vcon_schema::validate reports for each.
    • It also checks three more things:
      • The jsonschema reference engine agrees.
      • Each container is valid once the test repairs its recorded dataset issues.
      • The fetcher (--vcon-fetch-kind generic, against a server on 127.0.0.1) saves each one unchanged.
    • Each dataset directory there carries the LICENSE file of its dataset.
    • It also carries a README with the source path and SHA-256 of every file.
  • vcon_dataset_corpus_test reads every container in the five datasets.
    • It runs only when SIPNAB_VCON_DATASETS names a directory that scripts/fetch-vcon-datasets.py filled.
    • Without it, the binary prints one NOTICE line saying its gates did not run, as the capture corpus does.
    • The pre-push corpus gate does not run it. That gate selects tests by the name SIPNAB_CORPUS.

Fetch the datasets and run the corpus test:

# Run all of these, in order.
python3 scripts/fetch-vcon-datasets.py "$HOME/vcon-datasets"
SIPNAB_VCON_DATASETS="$HOME/vcon-datasets" \
  cargo test --features full --profile corpus --test vcon_dataset_corpus_test

The script runs only git.

  • It fetches each pinned commit.
  • It checks it out detached, with hooks and symbolic links turned off.
  • It refuses a checkout whose HEAD is not the pinned commit.
  • It refuses a checkout whose files have changed.
  • A second run verifies the cache and moves a dataset whose pin changed.
  • The five datasets take about 2.3 GB of disk.
  • The corrected copy described below takes about 2.2 GB more.

The corpus test prints three things for each dataset:

  • The number of containers.
  • The number the schema accepts.
  • Each finding, with the number of containers that carry it.

It asserts these things:

  • Each dataset is at its pinned commit.
  • sipnab and the reference agree on every container.
    • They agree on the verdict.
    • They agree on the instance paths of the findings.
  • The per-dataset counts measured on 2026-10-09.
    • These hold for as long as the pins do.
    • Moving a pin in PINS.tsv means measuring again and rewriting EXPECTED in the test.

What the five datasets get, at their pins:

Dataset Containers Valid Findings, with the number of containers that carry each
vcon-supreme-court-arguments 8,503 0 Attachment Object without mediatype: 8,503
ietf-meeting-vcons 8,181 0 Attachment Object without mediatype: 8,181. Dialog Object with url and no content_hash: 4,079
vcon-dataset-city-of-newport-ri 115 0 Attachment Object without mediatype: 115. Dialog Object with url and no content_hash: 115
fake-vcons 601 0 Attachment Object without mediatype: 601. Attachment Object without start: 601. Dialog Object with a body and no encoding: 264
tadhack-2025 43 0 Attachment Object without mediatype: 43. Attachment Object without start: 43

Each finding is a rule that draft-ietf-vcon-vcon-core-04 states in its prose and in its schema:

The reference engine reports the same findings at the same paths. With those four repaired, every one of the 17,443 containers is valid to sipnab and to the reference engine.

The corrected copy

tests/fixtures/vcon-datasets/CORRECTED.tsv pins NormB/vcon-datasets-core04.

  • It is a copy of the first three datasets.
  • It adds two kinds of key and changes nothing else:
    • mediatype on every inline attachment.
    • content_hash on the 425 IETF audio recordings whose files it downloaded, with an IETF login, and hashed.
  • Its CHANGES.md lists what it leaves open and why.
  • fake-vcons and tadhack-2025 are not in it.
    • The reason is the contact data described in the subset README.
  • The fetch script fetches both pins files unless --pins names others.
  • It accepts --pins more than once.

The corpus test holds the corrected copy to three things. These are beside the tests above that pin the upstream findings.

  • It is at its pinned commit.
  • Each corrected dataset gets the answer measured on 2026-10-11.
    • sipnab and the reference agree on every container.
  • Each corrected container is its upstream container with keys added.
    • The test counts the added content_hash values (425).
    • It requires each to be a SHA-512 token.
    • With them set aside, repairing both containers with the test's own repair gives the same document.
    • The corrected one needs no mediatype repair.
    • That last check compares this repository's repair rule with the corrected repository's own, on all 16,799 containers.
Dataset Containers Valid Findings, with the number of containers that carry each
vcon-supreme-court-arguments 8,503 8,503 None
ietf-meeting-vcons 8,181 4,527 Dialog Object with url and no content_hash: 3,654
vcon-dataset-city-of-newport-ri 115 0 Dialog Object with url and no content_hash: 115

The remaining finding needs a hash of each recording. No fetch returns these recordings to hash.

  • The IETF ones are YouTube pages.
  • The Newport video host refused the download.

Accepting a snapshot or overwriting a golden is a decision, not a fix.

  • Read the diff first.
  • These files are the record of what the tool promised its users.

The gate-test roster

These fail on changes you thought had nothing to do with them. Each one exists because the thing it guards silently rotted at least once.

Gate Trips when
docs_drift_test A --flag named on a published page does not exist in the CLI. That corpus derives from git ls-files rather than a hand-kept list, and covers every tracked docs/ page except design/ and research/, plus website/content/, README.md, SECURITY.md and CONTRIBUTING.md — these pages included, so a phantom flag here fails too. A flag belonging to another tool goes in FOREIGN_FLAGS, scoped to the label of each page naming it. Also: a version marker in the docs or man page disagrees with Cargo.toml; the README feature table misses a Cargo feature; a [theme] slot in ThemeConfig has no documentation, or the slot count quoted in either config reference is wrong. Also the benchmark reproducibility contract: the bench/ harness the benchmarks page tells readers to run must exist and be executable, bench/carrier.py must still produce the corpus composition the page quotes (checked at 1/100 scale), and both doc trees must name the same measured artifact and date.
dev_docs_drift_test A page under docs/internals/ links to a path that no longer exists, names a fn that no longer exists, uses an absolute GitHub URL instead of a relative one, is not registered in build-wiki.py, or breaks a mermaid convention. It also builds the wiki and fails if any relative link survives into the output — the wiki is flat and has no repo tree, so such a link publishes dead. That check runs the generator rather than reading it: the version that only asserted CODE_LINK_RE appeared in the script passed while ](../bench/) was shipping broken.
link_integrity_test Any relative link or heading anchor in either doc tree does not resolve; Zola content uses a plain relative .md link that would render as a dead URL. On the wiki-source side the scan is the top-level docs/*.md plus docs/internals/ — docs/design/ and docs/research/ are planning material outside the published journey and are deliberately not walked, though a link into them from a scanned page still has to resolve. A third scope covers the root community files (README.md, SUPPORT.md, MAINTAINERS.md, CONTRIBUTING.md, SECURITY.md, CODE_OF_CONDUCT.md), which neither doc tree walks — these are what GitHub renders in the sidebar, so a rename that broke their cross-references used to go unnoticed.
doc_example_coverage_test A user-facing CLI flag appears in fewer than two documented examples. A ratchet — the exemption list may only shrink.
flag_coverage_test A new long flag ships with no test referencing it. Also a ratchet: adding a test for a grandfathered flag fails until you remove it from the baseline list.
keybinding_drift_test A controller handles a key that the F1 help never mentions.
config_wiring_test A config key exists but is never read, or a CLI flag has no config fallback where its peers do.
feature_gate_test A flag whose subsystem is not compiled in fails late or silently instead of fast and clearly.
api_guidelines_test A growth-prone public enum loses #[non_exhaustive], or a shared store stops being Debug.
summary_consistency_test Two serializing surfaces disagree about a dialog or stream field.
wasm_exports_test The WASM binding loses an export the browser analyzer calls.
site_journey_test / mockup_alignment_test A website journey breaks, or a terminal mockup on the site drifts from real output. Also the homepage's advertised numbers: the binary-size ceiling, the glibc floor, the throughput tiles (which must quote a figure that appears on the benchmarks page they link to), and the automated-test count, which the template must read from the file pages.yml generates rather than state as a number. It holds the standards cards to a canonical table naming the code symbol behind every item, the field the program emits for every quality metric, and a citation in docs/ and the built site mirror, one card per standard. It also holds the homepage's four MCP examples to the files gen-mcp-examples.sh generated, and holds each of those to the claim the surrounding copy makes about it. That covers the second half of live binary -> website/data/mcp-examples/*.json -> index.html; demos/gen-mcp-examples.sh --check covers the first and needs a built binary, so CI cannot run it. The homepage's voice-stack section must link exactly the guides in the "Your SIP server" and "Add to your voice stack" groups of the docs navigation, so a guide added to one reaches the other.
config_examples_test A config sample in the docs no longer parses.
doc_commands_run_test A command shown in docs/, the README or a hand-written page under website/content/ is one sipnab refuses. It extracts every sipnab … line from every shell block, including a fence indented under a list item, through the same CommonMark fence rule as scripts/lib_markdown.py (it checks the two parsers agree on every page), and RUNS 447 of the 496 against a capture that ships with the repository, several at a time, each in a directory and a home of its own, failing on a usage error rather than on a non-zero exit -- plenty of documented commands exit non-zero for honest reasons. A device or root example runs with a device name that cannot exist substituted for the one the page names, so its flags still parse in full; a server example runs under a shared wall-clock bound, because clap refuses in milliseconds and a process still alive after it necessarily parsed its arguments. Only two kinds never run, and it names both: a shell program rather than one invocation, and a command that would exec something of its own or attach to the kernel. A command matching neither fails the test. It prints its own coverage, because the difference between "352 ran" and "3 ran, 380 skipped" is the difference between a gate and a decoration.
examples_run_test A program in examples/ no longer runs the way its own comment says, or prints something other than the output the comment shows. It runs every cargo run command the comments document (12, across all seven programs), except one under sudo, and holds docs/library.md's count of the programs to the directory.
release_surface_test A release adds a flag, MCP tool or REST route that no release note names, or a guide's Tested on names a sipnab release older than a flag its commands use. It reads the three lists at every release tag, with extractors it first holds to clap and the MCP router.
homepage_claim_truth_test A claim the homepage states in PROSE stops being true: the MCP tool counts and their read-only split, which released builds carry the bpf feature, what the installer actually puts on a host, a capability the static musl build does not have, the filter language's field and operator counts, a TUI key the page names that nothing binds, and a CLI flag the page names that does not exist, plus the unsafe count and the safety claim resting on it — the page said "memory-safe by construction" beside 88 unsafe blocks and no forbid(unsafe_code). site_journey_test gates the tiles; this gates the sentences, because the two disagreed on one page for months.
cookbook_recipes_test A command in docs/examples.md no longer runs or names a flag --help lacks; or a command it executes has no output golden in tests/cli/cookbook/ and no reason in the checker's OUTPUT_UNPINNED table; or a golden there is one no cookbook command produces.
support_selftest The shared normalization helper changes behavior under the tests that depend on it.
released_port_scan_test A test under tests/ binds a loopback port 0, reads the port, and releases the socket before using the number, which a test running alongside can take in between.

The rule for all of them: the gate is not the problem.

  • If flag_coverage_test fails, the flag needs a test.
  • If dev_docs_drift_test fails, a page now lies.
  • Adding an exemption is the last resort.
  • Every exemption list in this repo works as a ratchet, so it cannot quietly grow.

The development loop

Logging. SIPNAB_LOG is a tracing EnvFilter.

  • So it takes levels and per-module directives (SIPNAB_LOG=debug, SIPNAB_LOG=sipnab::rtp=trace).
  • In TUI mode sipnab suppresses logging unless SIPNAB_LOG says otherwise.
    • The alternative is log lines painting over the interface.
  • -q lowers the default in CLI mode.
  • Configured by init_logging().

Benches. cargo bench --profile profiling.

  • Plain cargo bench cannot build.
    • The cdylib crate type needed for the WASM build forces panic = "abort" into the lib unit.
    • Bench units build with unwind.
    • So shared dependencies get built twice with incompatible panic strategies.
    • They fail to unify.
  • The profiling profile is release codegen with panic = "unwind" and debug symbols kept.
    • That is also what callgrind wants.
  • Baselines live in benches/BASELINES.md.

nextest. .config/nextest.toml defines three profiles:

  • default (no retries, 30s slow timeout).
  • ci (no retries, no fail-fast, immediate-final output).
  • e2e (2 retries, 60s timeout) for the timing-sensitive tmux tests.

Worth knowing: no workflow currently invokes nextest.

  • CI runs plain cargo test.
  • The config is there for local use.
  • It is also there for the day the e2e shim goes away.

The docker lab. harness/ is a docker-compose stack for generating real traffic.

  • It holds OpenSIPS, rtpengine and SIPp.
  • make up in that directory builds and starts it.
  • make down tears it down.

WASM. One pre-commit gate covers the browser analyzer. It checks that website/static/wasm/sipnab.js still exports every function the site calls.

A second gate once demanded a freshly built bundle alongside any staged src/wasm.rs.

  • It no longer exists, nor does the binary it guarded.
  • The published analyzer went eleven releases stale while that gate stayed green.
    • The reason: src/wasm.rs had no commits in the window.
    • The interface held still while the implementation behind it moved.
  • The Pages workflow builds the bundle at deploy time now.
    • So the build produces what ships, rather than the tree carrying it.
    • It cannot go stale.

Feature matrices. A green cargo test --features full is not proof.

  • CI also builds reduced feature sets.
  • Code behind #[cfg(not(feature = ...))] is invisible to the full build.
  • For exactly this reason, the pre-push hook runs two checks before pushing:
    • cargo clippy --workspace --all-features --all-targets -- -D warnings.
    • The fuzz workspace check.