Experiment benchmarks catalog scan discovery race findings - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/catalog-scan-discovery-race-findings.md. Original SHA-256: 8cd1b12a00ecf3c722ef152f32d66d8a6233cbe9b0ce9b159b9bd1d00745b7b7. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Scan discovery-gap counterexample

Status: failure reproduced; unchanged regression passed on Scan candidate repair. Additive protocol (local checkout reference docs/experiments/benchmarks/catalog-scan-discovery-race-protocol.md; not present at origin/main), receipt (local evidence omitted from stage) and test transcript (local evidence omitted from stage) identify frozen source, commands and result. The separate Walk failure/repair does not establish Scan correctness, because Scan uses a distinct acquisition implementation.

Root executed, through the contained serial coordinator: cargo test --offline --locked -p fabric-server --lib query::discovery_race_tests::publication_and_reclaim_between_scan_discovery_steps_preserve_coverage -- --exact --test-threads=1 --nocapture. Cargo exit 101, job 6.013060746 seconds. This audit ran no new workload.

Seed 0xCA7A10A1; fixture origin scan discovery gap real publication checkpoint reclaim; no retention. Pre-IO ledgers contain 24 exact Batches and 144 explicit log rows, eight groups/three nodes. Thirty independent baseline oracle calls exited 0, and no-cut Scan selected custody matched the producer ledger.

The private hook executes after Scan's Segment list/frontier selection but before journal enumeration. Actual bounded publication and actual Store/sealer checkpoint and reclaim completed with unlimited retention: Segment 1 present, journal 1 gone. The copied cut.json marker says after_segments_before_extend; in this Scan fixture that marker denotes the equivalent pre-journal-enumeration cut, not a call to Walk's tail extend. The source hook and protocol specify the exact placement.

The SAME Scan acquisition returned Ok with 0 selected Batches, no Segments, eager Groups, tail paths/entries or blocks, and oldest_group=9. It returned no Interrupted. Expected custody remained 24 Batches. Bytes remained in the durable published Segment; this establishes source-acquisition omission, not durable loss or an independently observed HTTP response during the gap.

Preserved fixture: $FABRIC_DATA/evidence/fabric-work-3ee3e93fc53149d3a82bc86622bbe415/recovery/catalog-scan-discovery-race-01/scratch/storage-364508-0. Retain full state, origin/cut/source summary, selected custody, independent ledgers, 30 oracle directories and coordinator snapshots before byte-verified cleanup. Root owns archival/cleanup; none performed for this failure here.

Whole-job peak 830,476,288 bytes (792 MiB), sampled scratch 1,961,344 bytes, zero swap; failure scratch retained by launcher. These include compilation and grading/cache, and establish no product memory bound. The regression permits exact acquisition or explicit movement followed by exact fresh acquisition. Candidate correction must pass this unchanged expectation on a separately captured run. No additional schedule, performance comparison, partial-table oracle rule or qualification claim is introduced by this finding.

Separate candidate verification

catalog-scan-discovery-race-02 passed the same selector/expectations, Cargo exit 0, job 6.516439895 seconds. Its receipt (local evidence omitted from stage) and artifacts (local evidence omitted from stage) record 30 accepted baseline oracle calls and explicit Interrupted at the cut: segment coverage changed during discovery. The ordinary fresh acquisition selected Segment 1, oldest group 1 and all 24 exact expected Batches. Source label sets are compared before/after journal discovery before returning Sources; the existing bounded History retry handles movement. Initial labels include all known published labels before page frontier filtering. No expectation was relaxed.

Whole-job peak 862,162,944 bytes, sampled scratch 2,174,543 bytes; successful scratch removed. The original failed run remains preserved unchanged. Additional discovery may introduce safe retries or latency under publication; that cost is unmeasured. Uncached named directories without readable manifests now fail with bounded movement errors instead of being silently absent; no general source-corruption or table-specific completeness claim follows from this repair.