Historical report imported 2026-10-10 from docs/experiments/benchmarks/catalog-transition-ownership-findings.md. Original SHA-256: 6ad5703f40a73b433733bbf6f04fb19f4da211e4ac92b7a4549f4905964d339a. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Three labs turned source-level risks into executable counterexamples and fixes. The useful common result is that a physical transition must preserve its logical obligations: changing files cannot change a page's evidence, losing a caller cannot free capacity still used by its scan, and dropping a server future cannot leave a storage owner running indefinitely.

This is new implementation and evidence in Fabric, not a claim to have invented RAII, semaphores or snapshot isolation. The frontier opportunity is to combine truthful query evidence with predictable resource ownership for human and automated clients, then optimize that combination under measured load.

The transition protocol (local checkout reference docs/experiments/benchmarks/catalog-transition-ownership-protocol.md; not present at origin/main) and witnessed HTTP schedule (local checkout reference docs/experiments/benchmarks/catalog-body-admission-protocol.md; not present at origin/main) were committed separately before their executions. No oracle, wire/persisted format, sync order, product promise or qualification policy changed.

Lab	Before, reproduced on unchanged production	Implemented correction	Targeted evidence
Lifecycle	Aborting a listening TLS server left every journal reopen attempt at WouldBlock for three seconds; occupied-port failure released it	A private worker owner signals shutdown on drop; already-scheduled cleanup joins sealer and commit off the executor, even if awaiting cleanup is cancelled	Both unchanged integration cases passed; three ownership controls passed, including partial startup and worker panic
Query	After publication, four Scan/Walk old-page chains reported receive/freshness 900 instead of their snapshot's 200	Borrow manifest aggregates only for entirely covered Segments; derive metadata from snapshot-contained raw entries for crossing Segments	Ten full chains passed the unchanged Python oracle; two missing-row controls were rejected; two retained tokens became Gone after removal
Boundedness	Sixteen batch/two query bodies were known to have started extraction; excess incomplete requests still waited until the 600 ms read deadline	Independent pools admit 16 batch/two query requests before body extraction; a query permit stays inside the blocking task	Excess bodies received 503 with Retry-After 1; other pool/auth controls remained 200/401; cancellation restored slots; three blocking-work lifetime controls passed

The snapshot fixture supplies exact producer bytes and known timestamps before storage. It exercises real FrameLog, publication, journal reclamation, restarted History readers and retention, through the public query API. It is not an HTTP query-pressure test. The admission fixture uses actual plaintext HTTP/1 with Expect: 100-continue as its admission witness. The lifecycle fixture uses TLS. The existing native collector/TLS lifecycle remains part of workspace checks.

Workers (local checkout reference crates/fabric-server/src/lifecycle_workers.rs; not present at origin/main) owns the stop signal, HTTP handle, a bounded single-sealer handoff and a cleanup completion handle immediately after commit startup. Drop only signals and closes the handoff; a blocking task already owns the joins. Ordinary return waits for both workers. A sealer panic cannot skip the commit join. Startup/bind errors retain priority while secondary cleanup errors remain visible. This costs one waiting blocking cleanup task per running server; it does not impose a deadline on uninterruptible filesystem I/O.

HTTP admission acquires a Rust-owned permit before Bytes extraction. The batch handler drops its original Bytes after copying into the admitted Submission. The query handler moves its permit into the same blocking closure as the scan; the permit is released after completion or unwind, regardless of the async waiter's lifetime. This bounds these admitted request/work counts and prevents one saturated pool consuming the other's slots. The 16/two limits are conservative fixed choices, not measured throughput optima. Connection buffers, TLS handshakes, response-body retention, other admin routes and each scan's own memory remain separate resource dimensions. A peer can still hold an admitted partial body open; admission deadlines and early authentication are subsequent investigations.

Snapshot evidence uses Cow<BTreeMap<...>>: the common full-Segment path borrows existing metadata, without a new map clone or raw decode. Crossing Segments use the one-row raw scanner, verify raw entry digests and project one included Batch at a time. Metadata stays local until the full scan succeeds. A raw error marks unavailability and cannot fall back to future or partially accumulated metadata. The reader still owns Parquet page buffers; a one-row batch is not a one-row process-RSS guarantee. The extra boundary scan is a correctness cost whose repeated-page CPU/I/O needs measuring.

Root serialized every command through python3 -B tools/resource_group.py --, then tools/bench/labs/completion/run_job.py and tools/bench/labs/catalog/coupled_admit.py. The driver (local checkout reference tools/bench/labs/catalog/transition_ownership.py; not present at origin/main) records exact argv, exit codes and source hashes. Coordinator receipts include full source, working-tree diff, registered protocols, resource observations and cleanup.

Job	Result and command evidence	Coordinator evidence
catalog-transition-repro-01	Receipt (local evidence omitted from stage): cancellation and snapshot Rust checks each exited 101; exact counterexamples classified, driver exited 0	17.52 s receipt (local evidence omitted from stage)
catalog-admission-repro-01	Receipt (local evidence omitted from stage): HTTP test exited 101 with both expected WouldBlock responses; controls succeeded, driver exited 0	9.51 s receipt (local evidence omitted from stage)
catalog-transition-fixed-01	Receipt (local evidence omitted from stage): all eight Cargo test invocations exited 0, covering 17 targeted Rust tests	28.58 s receipt (local evidence omitted from stage)

The fixed job ran cargo test --offline --locked -p fabric-server separately for integration targets cancellation, query_snapshot_transition, admission, startup, delivery, and library filters lifecycle_workers::tests, http::admission_tests, query::snapshot_evidence_tests. This counts ten integration tests and seven unit controls; the 10 oracle chains and their rejection/retention outcomes are inside one integration test. Source hashes match before and after each execution.

All three jobs used mounted scratch/build storage below $FABRIC_DATA, verified memory high 16 GiB/max 20 GiB, no swap and the normal outer deadline. Original failures are preserved as full small fixture archives; every member was decoded and compared byte for byte before scratch removal. Altered/missing/duplicate archive controls and repaired counterexample trace controls were rejected. No frontier or evidence budget was reset, and no remote host was loaded.

The first full verification, catalog-transition-checks-01, exited 1: all workspace tests and 15 other fast gates passed, but Clippy rejected the new inline test module before production items. Manual documentation checks were not run. Its failed receipt (local evidence omitted from stage) and 319.09 s coordinator record (local evidence omitted from stage) remain unchanged. The test body was moved to its own module with all assertions preserved. No production behavior changed in that correction.

The separately registered retry (local checkout reference docs/experiments/benchmarks/catalog-transition-checks-retry-protocol.md; not present at origin/main), catalog-transition-checks-02, records exact commands, actual exits and cleanup in its driver receipt (local evidence omitted from stage) and coordinator receipt (local evidence omitted from stage). It also archives/readbacks the previous launcher's small remaining fuzz scratch before removing that owned failure tree; original failure receipts stay intact. The fast command is cargo xtask checks --profile fast; the manual documentation profile runs the existing Bun checker, its tests and the Python hook tests. Those receipts, rather than the planned dispatch, establish final outcomes.

The range-evidence and work-ownership research packet derives the next alternatives from the counterexamples. In particular, min/max summaries cannot be subtracted to recover a clipped range. It prioritizes a metadata-column projection comparison before introducing a persistent index.

The next research queue should pursue useful capabilities, with correctness held fixed before any resource claim:

Lab	Candidate	Hypothesis and rejection condition
Query	Storage-owned range summaries for boundary metadata	A bounded summary keyed by group range can avoid repeatedly decoding crossing Segments while preserving the exact oracle envelope through append, publication and expiry. Reject if cache bytes grow with history, stale metadata survives retention, or total CPU/I/O does not improve against this raw-scan baseline.
Boundedness	Work-weighted admission and finite body deadlines	Distinguish cheap selective queries from broad scans without allowing cancellation to erase outstanding work. Compare fixed pools with explicit measured work estimates under mixed ingestion/query demand; reject on starvation, worse ACK tails, unbounded retained state, or incorrect overload/custody reporting.
Lifecycle	Deterministic accepted-work cancellation cuts	Pause after Intake acceptance and before ACK, disconnect the client, stop/restart and retry exact bytes. Require an independent producer/recovery ledger and unchanged delivery oracle. An arbitrary send/abort race is insufficient evidence that the acceptance boundary was crossed.

An additional source-only lead is the loopback trace receiver's blocking channel send before its acknowledgement timeout. A full queue may consume all connection workers before that timeout begins. Reproduce with an explicitly blocked consumer before selecting nonblocking refusal or a bounded admission deadline. This lead has not been executed and is not a fixed defect in this record.

These directions preserve the intended product distinction: a person can inspect why an answer is incomplete or stale, and an agent can use the same structured facts and overload response to make a bounded next decision. Full deployment capacity, long-soak behavior, installed-service behavior and physical power-loss durability remain unqualified.