Historical report imported 2026-10-10 from docs/experiments/benchmarks/catalog-native-lifecycle-findings.md. Original SHA-256: cba6bcecf2815ad1a7a6bc6bc347882919015ed78b64ef2b34489455759e0535. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Native lifecycle reliability and startup recovery

This round connected actual file collection and Spool custody to TLS delivery, restart, Segment publication and complete HTTP queries. It also reproduced and fixed a startup resource leak. The startup registration (local checkout reference docs/experiments/benchmarks/catalog-startup-recovery-protocol.md; not present at origin/main) and native lifecycle protocol (local checkout reference docs/experiments/benchmarks/catalog-native-lifecycle-protocol.md; not present at origin/main) preceded execution. These are finite reliability results, not performance or deployment qualification.

What changed

Boundary	Before	Result
Invalid TLS configuration	serve spawned commit/sealer workers before loading TLS; early TLS errors left the journal writer owned	TLS is validated before Store/worker ownership; both failure fixtures reopen the journal on their first attempt
Native lifecycle evidence	Real collection/TLS restart tests checked bytes; separate history tests used manufactured Batches for query/sealing coverage	One additive fixture now follows real collected data through these boundaries and checks producer bytes before oracle grading
Expected query population	A replay-only expected population can omit a producer Batch lost before recovery	An exact producer/recovery bijection precedes the timestamp join used by the unchanged query oracle

The production correction is constructor ordering in server composition. Valid TLS startup and existing delivery tests continue through the same storage and serving paths. Collection, row projection, wire bytes and durability order are held fixed.

Preserved startup counterexample

Both desired-behavior tests first ran against the original production source. The reproduction receipt (local evidence omitted from stage) records Rust exit 101, separately from the driver's successful identification of the expected counterexample. A missing certificate produced NotFound; a malformed private key produced Other with private key format not supported. Every subsequent same-process Store reopen failed with WouldBlock throughout the two-second retry window. Each fixture had opened/dropped its Store successfully before serve, isolating the ownership left by the failed startup.

The actual traces (local evidence omitted from stage), exact source copies, complete fixture archive (local evidence omitted from stage) and byte-readback manifest (local evidence omitted from stage) remain. A classifier control containing a successful reopen was rejected. The cause is the sealer's live Intake keeping the commit thread alive while its stop flag was never set on the early TLS return. Validating TLS first avoids acquiring those owners in these error cases.

The unchanged startup regressions (local checkout reference crates/fabric-server/tests/startup.rs; not present at origin/main) then both exited 0. Each returned the expected TLS error and opened the Store on its first retry, as the corrected output (local evidence omitted from stage) records. No arbitrary cancellation or all-worker-failure recovery claim follows.

End-to-end evidence

The native fixture (local checkout reference crates/fabric-server/tests/native_lifecycle.rs; not present at origin/main) passed in 10.19 seconds after compilation. The summary (local evidence omitted from stage) records:

• Seven exact producer Batches, on one Strand with contiguous sequences.
• Four bounded skip passes through a 3 MiB source line, with node/server restart after pass two and ACK 2 restored. Resending Batch 2 returns duplicate ACK 2.
• Forty-one source-derived Unicode log bodies, exactly one oversized-line gap, real host metrics and one span committed through public commit_traces.
• One published Segment and complete reclaim of sealed journal files after a real collection crossed the configured 64 KiB rotation boundary.
• Thirty full query chains: three storage/restart stages, two plans, five query forms. Whole Scan/Walk answers matched and the unchanged Python oracle graded logs, substring logs, metric history, rates and spans.
• Twelve rejected dropped/duplicated-row answer controls, plus the missing-record custody comparison. Archive validation rejected altered, missing and duplicate members before preserving the real fixture.

Exact Spool frames were captured before each send. Recovered identities and bytes had to biject that producer population before recovery could supply receive timestamps to oracle inputs. The fixture also independently checks the final decoded log bodies against constructed source lines. Thus server replay and query agreement alone cannot hide loss relative to the producer ledger.

The full archive (local evidence omitted from stage) contains TLS fixtures, source file, Spool, server state, producer ledgers, attempts, complete page chains, oracle inputs/verdicts and numbered observations. Its manifest (local evidence omitted from stage) records exact byte readback of all 9,556,104 decoded file bytes before cleanup. This exceeded the initial informal roughly 6 MiB estimate, demonstrating why admission used a 16 MiB full-state allowance rather than predicted success compression. Driver evidence occupied 659,456 allocated bytes after compression and source capture.

All four existing TLS delivery integration tests also exited 0, including the existing OTLP listener path, restart, rejection and output-rate controls. The new lifecycle fixture itself uses public trace commit; it does not test an SDK exporter or the node CLI loop. Its one-node 2 MiB Spool/ journal, 64 KiB rotation and 4 MiB retention settings exercise transitions with small inputs. They do not qualify dev/small profile capacity, retention pressure, old-token continuation through publication, abrupt process death or physical power loss. No new throughput, application RSS or latency gain was measured.

Execution, resources and cleanup

The execution receipt (local evidence omitted from stage) includes every command/exit and exact relevant source hashes. The registered launch was:

python3 -B tools/resource_group.py -- python3 -B tools/bench/labs/completion/run_job.py --id catalog-native-lifecycle-01 --lab coordinator --stage preparation --seconds 360 -- python3 -B tools/bench/labs/catalog/coupled_admit.py --reserve-mib 18 -- python3 -B tools/bench/labs/catalog/native_lifecycle_run.py --mode lifecycle

It exited 0 in 28.845 seconds, with 24.641 cgroup CPU seconds, 1,583,755,264 bytes cgroup memory peak and 17,337,061 sampled scratch bytes. These include compilation, all integration work and cache; they are not application RSS or per-query CPU. The coordinator receipt (local evidence omitted from stage) records zero high/max/OOM events, zero swap, no surviving descendants and successful scratch removal. All workloads/validators use one containing cgroup, 20 GiB max, 16 GiB high, no swap, the 30-minute outer deadline and data-drive scratch/build paths. CPU has the launcher's default unrestricted quota. No remote work ran.

The initial successful fixture emitted a generic failure retained cleanup message when intentionally leaving evidence for the root driver. Its exit and all assertions were successful; the message was corrected before final checks to distinguish intentional evidence retention from a failed test. That logging change does not alter the measured production source or semantic assertions.

The capacity lab's exact snapshot reclamations 02 (local checkout reference docs/experiments/benchmarks/catalog-protocol-dedup-round2-protocol.md; not present at origin/main) and 03 (local checkout reference docs/experiments/benchmarks/catalog-protocol-dedup-round3-protocol.md; not present at origin/main) completed 1,506 and 1,405 hardlink replacements. All 2,927 logical paths and content bytes were read back; original metadata is preserved in manifests and snapshots must remain immutable. Net observed reclaim was 14,233,600 and 10,543,104 allocated bytes, about 23.6 MiB combined. Receipt 02 (local evidence omitted from stage) and receipt 03 (local evidence omitted from stage) retain the full accounting, controls and inactive producer checks. This funded failure preservation within the unchanged aggregate caps.

Final fast/manual commands, source hashes and actual outcomes are in the verification receipt (local evidence omitted from stage), with coordinator resources and cleanup (local evidence omitted from stage). The temporary Bun executable uses exact existing-archive readback/reference before removal. Original failed startup tests remain recorded as failed; all elapsed work remains charged to the original frontier and stage budgets.

Consequences for the research direction

The useful improvement here is a failure boundary: validate configuration before acquiring long-lived worker ownership. The useful verification improvement is to compare producer custody before trusting recovered records as oracle inputs. Both complement the earlier extraction speedups without changing query meaning.

Next reliability questions are cancellation after worker startup, page-token continuation while publication/retention changes physical storage, and bounded overload followed by drain on actual dev/small service budgets. The broader C5 matrix, target-system installation and long soak remain separately scoped; this finite screen neither completes them nor resolves the historical soak gate.