Internals vCon Exporter - NormB/sipnab GitHub Wiki

This page covers three things:

What src/output/vcon.rs holds.
Which sipnab type feeds each part of the container.
The two mechanisms an editor has to respect:
- the duplicated completeness caveat;
- the deterministic uuid.

Read Export one observed call as a vCon for the operator's view. Read the Phase 0 decision for why the module refuses what it refuses. This page is about the code.

Shape of the module

The module is one file behind the non-default vcon feature. The full feature carries it.

It has no submodules and no state.
Every input arrives as an argument.
Nothing in it reads a process global except the node name.

The public surface has these parts:

A struct per vCon section: Vcon, Party, Dialog, Attachment, Analysis.
BlindSpot and CaptureCompleteness for the caveat.
ExportContext for everything the dialog itself does not carry.
Three functions: export_dialog(), export_dialog_at() and dialog_uuid().

export_dialog_at() is the pure form. Every test drives it.

Its third argument is the container's created_at.
So two exports compare byte for byte.
Any difference between them came from the capture, not from the clock.
export_dialog() is the same function with Utc::now() supplied.

Serialization is serde's. Each struct maps one-to-one onto its vCon object.

skip_serializing_if = "Option::is_none" does the work of "sipnab did not observe this".
The result is an absent field, never a null.

Where each section comes from

vCon section	Read from	Built by
`vcon`, `extensions`	module constants	inline in `export_dialog_at()`
`uuid`	the dialog's `Call-ID` and clock, the capture id, the node name	`dialog_uuid()`
`created_at`	the caller's clock, not the dialog's	the `exported_at` argument
`parties[0]`, `parties[1]`	`SipDialog`'s `from_`/`to_` fields, the opening message, the first response	`observed_parties()`
`parties[].tel`	the SIP user part, only when it is an RFC 3966 global number	`tel_uri()`
`parties[last]`	`node_name()` and the crate version	`observer_party()`
`dialog[0]`	the `Call-ID`, the observed tags, and `final_status_code()`	`dialog_object()`
`attachments[0]`	the message ladder, through `message_to_json_value()`	`message_trace_attachment()`
`attachments[1]`	`CaptureFacts` and the ranked `CaptureAnalysis`	`completeness_attachment()`
`analysis[0]`	`diagnose_signaling()` and the same completeness value	`report()`

Three of those rows carry a decision rather than a mapping.

tel is narrow because a wrong index entry beats no index entry only in appearance.

A conserver indexes parties by tel, mailto and name and by nothing else.
So a container with none of the three answers only to its UUID.
tel_uri() supplies one for + followed by digits and for nothing else.
A bare digit run is an RFC 3966 LOCAL number needing a phone-context.
In SIP captures it is usually an extension. 1001 is not a telephone number.

The message trace reuses --json's projection.

One serializer produces both.
So a vCon and an NDJSON line describing one message cannot disagree about it.
The cost of that reuse is the credential strip, further down.

The report reuses the signaling diagnosis.

It is already the wire shape the dialog-JSON surface emits.
A second projection of one analysis is two definitions waiting to drift apart.

ExportContext exists for the same reason CaptureFacts does:

It makes visible, in one place, exactly which facts a completeness claim stands on.
It lets a test state the capture it describes instead of mutating a global.

The completeness carrier

The whole module exists for this mechanism. An edit is most likely to break it.

sipnab's ranked problem list puts incompleteness findings inside the list.

They sit at Severity::Blind, above every call fault.
So "no problems found" is structurally unreachable for an incomplete read.

vCon offers no position with that property.

analysis[] is a list of conclusions.
A caveat placed there is one entry among others.

Why duplication rather than one field

The obvious repair is a custom extension carrying the caveat. The format rules it out. An extension is one of two kinds:

compatible: a consumer that does not implement it ignores it safely;
critical: a consumer that does not implement it MUST NOT process it except to reject.

There is no third level meaning read this before trusting the contents.

Pick either and the caveat fails.

Ignorable: the consumer that most needs it drops it.
Critical: an ordinary vCon reader refuses the whole container.
- So sipnab has emitted something nobody can read.

So the caveat goes into two surfaces a consumer walks past anyway:

capture_completeness inside analysis[0].body, beside the diagnosis.
- body is the JSON object itself.
- So a reader indexes into it directly. See below.
An attachment with purpose sipnab-capture-completeness.
- Its party index names the sipnab observer.

One value, embedded twice

completeness_of() builds one CaptureCompleteness.

export_dialog_at() then hands the same value to completeness_attachment() and to report().
Nothing rebuilds it.
Nothing formats it a second time.

That distinction is the whole rule.

Two strings built twice drift.
A container whose two caveats disagree reads as authoritative while contradicting itself.
That is worse than one carrying no caveat at all.

The audio exporter learned this the expensive way.

It built its embedded note and its printed summary separately.
A clause reached one and not the other.
A comparison test caught it.
This module copies that shape deliberately.

The divergence gate is the_two_completeness_surfaces_carry_one_value.

It exports one dialog against facts carrying two distinct losses.
It then asserts that the attachment body and the report's capture_completeness compare equal as JSON values.
Rebuild the caveat in either place and it fails.

Two more tests hold the ends of it:

the_completeness_carrier_discriminates_a_lossy_run_from_a_clean_one:
- It exports one dialog twice.
- One run lost seven captured messages. The other run lost none.
- It requires the two notes to differ.
- Without it the carrier could be decoration: identical on every run, and equal to itself.
- Deleting the messages_evicted clause fails it.
an_absent_analysis_is_not_a_clean_bill:
- It separates "nobody looked" from "looked and found nothing".
- blind_spots stays absent when no analysis reached the export.
- It becomes [] when one ran and ranked nothing.
- Collapsing the two lets a run that skipped the analysis read as clean.

The spec is the gate, not our reading of it

Two decisions in this module came from reading two sources:

the draft;
the reference implementation.

They did not come from what looked natural in Rust.

Both were wrong in the first cut.
A hand-written test agreed with the wrong answer.
The same misreading produced both.

body takes the shape its encoding defines.

Section 2.3.2 of the core draft defines "json" as "The value of the body parameter is a JSON value (object, array, number, string, or one of the literals true, false, or null)".
The CDDL in Appendix C states that body is any JSON value when encoding is "json", and a string otherwise.
Attachment::body and Analysis::body are serde_json::Value.
- Every json body sipnab writes is the object itself.
Earlier releases wrote the object's text as a string.
- That is a JSON string value, not the object the encoding describes.
The readers still accept that shape.
- answer_body() keeps a json body from an enrichment service as whatever JSON value it is.
- The -I reader parses a string json body as JSON text.
The gate is every_body_has_the_shape_its_encoding_defines, in the ingest contract test named under the schema gate below.

core-03 had two defects on this one field, and core-04 settled both.

The first was a vocabulary gap.
- None of the five type values described a conversation known to have occurred whose content the container does not carry or reference.
- That is the ordinary result for an observer that retains no media.
The second was a prose/schema inconsistency.
- core-03's Dialog Object section said "it is possible to have a Dialog Object with no parameters in it".
- Its schema required type and start.
- section 7.1 of the vCon design page, "Section numbers", records where that text went.
From 0.5.128 sipnab emitted such an object with no type.
- Its vendored schema removed type from required as a documented deviation.

Section 4.3 of core-04 replaced that sentence.

It covers a dialog known to have occurred, with little or nothing captured from it.
The object for it is a placeholder "which contains only the type parameter".
- Its type is recording if the call reached setup.
- Its type is incomplete if it did not.
start became a SHOULD (section 4.3.2).

So sipnab dropped the deviation.

Both vendored schema files are the publisher's core-04 file, byte for byte.
Every Dialog Object sipnab writes names its type.

The type follows what sipnab observed.

incomplete, from section 4.3.1.5 of core-04, names a call that "failed to be setup".
That is a claim about the CONVERSATION.
sipnab emitted it for every signaling-only export until 0.5.128.
So a successful call shipped a container reporting a setup failure.
Read months later beside a switch's CDR showing a connected ninety-second call, the container is the thing that looks wrong.

sipnab now keeps incomplete for a dialog whose final response it OBSERVED to be a failure.

Every other signaling-only object is the recording placeholder.
dialog_object() decides the type and the disposition in one expression.
The reason: section 4.3.11 of core-04 couples them.
- An incomplete object MUST name a disposition.
- A disposition is only nameable when sipnab saw a failure.
The media path fills the placeholder in place when audio actually arrives.
- It clears disposition with it.

A recording placeholder carries no body and no url.

A conserver transcription link that selects type == "recording" and reads dialog["url"] with a bracket raises on it.
Under core-03 the type-free object raised in the same links at dialog["type"].
The converse costs as much.
- Every type == "recording" selector skips audio left on an incomplete object.
- So the WAV would sit in the container unreachable.

audio_never_rides_on_an_object_typed_incomplete and a_recording_without_content_is_a_bare_placeholder pin both directions.

The second lives in the SIGNALING-ONLY test file on purpose.
Over a media fixture every object has a body.
So the assertion passes vacuously and the mutation survives.

The repository carries the schema, so the gate runs offline.

tests/schemas/vcon.schema.json is the working group's core-04 schema, vendored unchanged.
tests/schemas/publisher/vcon_json_schema.json holds the same bytes.
the_vendored_vcon_schema_is_the_publishers_core_04_file pins both to the publisher's commit and SHA-256.
So a re-vendor is a recorded change rather than an edit.

a_container_validates_against_the_working_group_schema in tests/vcon_ingest_contract_test.rs validates a real export against it.

That one test found six violations.
The hand-written assertions around it had all missed them.
Those assertions encoded what we believed the format required.
Add a field and this gate, not a reviewer, tells you whether the format agrees.

Media enriches the signaling object rather than appending a second one.

A dialog is one conversation.
A container that carries a recording object beside a separate signaling object describes it as two.
When audio decodes, export_dialog_at() fills the existing object's fields in place: start, duration, parties, mediatype, encoding, content_hash and body.
The one exception is a wrapped ring buffer.
- There the retained audio genuinely is a subset of a longer call.
- The object becomes a recording-set and the audio hangs beneath it.
- That is what that type is for.

Adding a field without breaking the divergence gate

The gate compares whole JSON objects. So the rule is short: add the field to CaptureCompleteness, and to nothing else.

Add the field to the struct.
- Both surfaces serialize the same value, so both gain it at once.
- The equality assertion still holds.
Populate it in completeness_of(), from ExportContext and from nothing else.
- Reading a process global here is what stops an export from reproducing.
If the field describes a loss, add a clause to completeness_note().
- Then the prose surface says what the structured surface counts.
- Write it as a measurement of the run, never as a conclusion about the call.
- "the capture missed the answer" is a fact about the tap.
- "the call went unanswered" is a fact about the traffic.
- This module exists because vCon cannot tell them apart.
Extend the_completeness_carrier_discriminates_a_lossy_run_from_a_clean_one with a run that moves the new field.
- Without that, the clause is unproven.

What breaks the gate is building a value inside report() or inside completeness_attachment().

Both take a &CaptureCompleteness and serialize it.
Keep it that way.

Two adjacent rules an edit trips over:

Never put a caveat in subject.
- The core draft defines it as the topic of the conversation.
- So a disclaimer there arrives styled as a fact about the call.
- the_container_declares_its_version_and_signs_nothing asserts subject stays absent.
- The same holds for signatures, payload, protected, jwe, jws and consent.
Never widen failure_disposition() below 400.
- A 3xx redirect did not fail.
- A container claiming it did sends an operator after a fault that is not there.
- The redirect case sits inside the mapping test on purpose.
- So widening the range fails there rather than shipping.

The uuid: deterministic by construction

dialog_uuid() writes a UUIDv8 laid out like a UUIDv7:

a 48-bit millisecond timestamp;
a 12-bit rand_a;
a 62-bit rand_b.

Three choices make it a function of the observation rather than of the run:

Field	Source	Why
48-bit timestamp	the dialog's `created_at`	the export clock here would mint a new identifier on every re-export
`rand_a`, 12 bits	SHA-256 of `Call-ID`, a `0x1e` separator, then `capture_id`	ties the value to the dialog and the capture it came from
`rand_b`, 62 bits	the high 62 bits of SHA-256 over the node name, a `0x1e` separator, the `Call-ID`, another separator, then `capture_id`	host-derived, as the draft asks, and dialog-derived as well — see the collision window below

The separator byte matters. A bare concatenation collides ("ab", "c") with ("a", "bc").

The version and variant bits follow RFC 9562:

Version bits go in the high nibble of octet 6.
The variant 10 goes in the two high bits of octet 8.
rand_b shifts down by two rather than masking its top bits off.
So the variant bits overwrite nothing the draft asked for.

created_at still carries the export time, where the draft puts it.

Two exports of one dialog days apart therefore share a uuid.
They differ in created_at.
the_uuid_separates_dialogs_captures_and_clocks asserts exactly that pair.

The collision window, and how it closed.

Until 0.5.125 this keyed rand_b on the node ALONE.
- It spent 62 of the 74 available bits on a value identical for every dialog on the box.
Two dialogs opening in the same millisecond on one node had 12 bits between them.
- So roughly one pair in 4096 collided.
section 4.1.2 of the core draft makes the uuid globally unique.
- The reason: a store KEYS on it.

A collision raises nothing.

It overwrites the record already there.
That loses one capture with no error anywhere.

An earlier version of this page called that "inherent in the layout the draft asks for". It is not.

The draft asks for a host-derived value in rand_b.
It never asks for a value derived from the host and NOTHING ELSE.
Folding the dialog and the capture into the same digest keeps every property the layout wanted.
It takes the discriminating bits from 12 to 74.

Determinism is the property that had to survive, and it did.

The digest is over stable inputs.
So re-exporting one dialog from one capture still yields one identifier.
two_dialogs_in_one_millisecond_on_one_node_get_different_uuids holds a real colliding pair found by brute force.
It asserts the precondition that they still share the old 12-bit bucket.
So it fails loudly rather than passing vacuously if the seed ever changes.

One more substitution to know about: the draft fills rand_b with SHA-1 of a host name alone, and the default does not.

UUIDv8 constrains only the version and variant bits.
- This function writes both.
What changes is which bits land in rand_b.

[vcon] uuid_domain follows the draft's form.

UuidDomain holds the name and the high 62 bits of SHA-1 over it, from the sha1 crate the vcon feature already carries for RSA-OAEP.
bootstrap::declare_process_globals declares it once, so every door mints the same uuid for one dialog.
rand_b is the domain's 62 bits. rand_a keeps the default's seed.
That leaves 12 bits per millisecond per domain, so UuidSlots records the rand_a values given out per millisecond and steps a colliding seed to the next free value.
- It is per process and capped at UUID_SLOT_CAP identities, oldest milliseconds dropped first.
revision_uuid takes the same path for a new-uuid chain.
Tests: the_default_uuid_is_byte_identical_to_before pins the default from before the key existed. a_domain_uuid_follows_the_section_4_1_2_layout recomputes each field. two_dialogs_in_one_millisecond_get_different_domain_uuids holds a seed collision found by search. tests/vcon_uuid_domain_test.rs covers the shipped binary.
See The uuid under a domain.

The credential strip, and what it does today

strip_credentials() walks a JSON value at every depth. It drops any key matching CREDENTIAL_HEADERS:

Authorization;
Proxy-Authorization;
WWW-Authenticate;
Proxy-Authenticate.

It compares case-insensitively.

SIP header names are case-insensitive on the wire.
A filter keyed on exact case is one an ordinary peer walks through.

Proxy-Authenticate is on the list although the SIP-signaling extension names only three.

It is the proxy-side twin of WWW-Authenticate.
It carries the same realm and nonce.
Leaving it off would hold the rule for one hop and not the other.

It is load-bearing as of 0.5.125. It was not before.

The honest history matters more than the current state.
The reason: the shape of the mistake recurs.

The filter used to guard a projection that could not carry a credential.

message_to_json_value() had no raw header map.
So no message reached it with a banned key.
no_credential_survives_an_export planted a real Authorization and a real Proxy-Authorization in its fixture and passed.
- It passed not because the filter worked, but because the field did not exist.
A regression gate for something that could not regress reads exactly like a working control.
- That holds in a coverage report and to a reviewer.

The old text here predicted its own resolution: "giving that projection a headers field is an ordinary, sensible change to a debugging surface, and it would start putting digest credentials into a container that leaves this machine."

That is what 0.5.125 did.
The two halves landed in ONE change, deliberately.

The reasons:

The sip-signaling extension names headers in its message structure.
A SIP trace without them is a summary of a trace.
So the field was going to arrive.
A field that publishes credentials must never land in a release ahead of the filter that removes them.

The filter is now mutation-proven end to end.

Empty CREDENTIAL_HEADERS and no_credential_survives_an_export fails.
That is the thing it always claimed to check.

Three tests keep the distinction visible:

the_trace_carries_the_headers_the_filter_exists_to_clean is the half that keeps the other two honest.
- Drop headers from the trace and the filter has nothing to remove again.
- So it fails rather than letting the pair go quietly back to proving nothing.
the_credential_filter_removes_banned_headers_at_every_depth runs against a hand-built value.
- So it covers nesting depths an ordinary export never produces.
- It is mutation-proven: deleting the retain fails it.
- Its anti-vacuity assertions fail a filter that emptied the object instead.
no_credential_survives_an_export is the end-to-end half.
- It passes on the strength of the projection rather than of the filter.
- It is what refuses the digest response the day that changes.

Reading a container back

The reader is a separate module, capture/vcon_input.rs. It is an input source rather than part of the exporter.

It rebuilds each message of a sip-message-trace attachment, or each message/sip attachment, as wire text.
It hands it to the packet path as a pre-parsed packet.
- So nothing downstream has a vCon branch.
What it cannot rebuild it records in a VconProvenance.
- The capture analysis turns that into the rebuilt_from_vcon finding.
The inline analysis behind the MCP analyze_vcon tool and POST /v1/vcon/analyze is output/vcon_analysis.rs.
- It is one function for both surfaces.

A schema finding never stops a read. It rides in the provenance.

The reader refuses only two things:
- a container whose SIP it cannot rebuild;
- a JSON object with none of the members a vCon has.
A change to the trace the exporter writes is a change to what the reader can rebuild.
So tests/vcon_input_test.rs exports a capture with the built binary and reads it back.
- It compares every stored field.

What the tests cover

Unit tests live in the module and reach its private helpers. tests/vcon_export_test.rs is the other half. It proves two things the unit tests structurally cannot:

that the whole export is reachable through the crate's public surface;
that a dialog off a real parsed capture carries what the hand-built fixtures carry.

A projection can agree with a synthetic dialog and disagree with the parser.

Both halves gate on the feature. So a build without vcon compiles neither.

The schema validator in vcon_schema.rs also runs over containers sipnab did not build: the public vCon datasets.

A committed subset runs on every build.
All 17,443 containers run on request.
Testing says how to fetch them and what they get.