Experiment benchmarks catalog maintenance findings - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/catalog-maintenance-findings.md. Original SHA-256: 0ebbe4a05d70bd25c8e1ab14057573e69993469f862764da51c8df0949534a39. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

CQ2 maintenance findings

The completed comparison did not support proactive refresh at any registered query rate. All 15 corrected slices, 33 trials and 1,826 actual complete oracle chains were validated; the original seed-publication failure remains preserved separately. The measurement protocol (local checkout reference docs/experiments/benchmarks/catalog-maintenance-protocol.md; not present at origin/main) and separately registered fixture correction (local checkout reference docs/experiments/benchmarks/catalog-maintenance-fixture-correction-protocol.md; not present at origin/main) define the unchanged experiment and replay/oracle expectations.

The original rate-zero, seed-128 preflight failed before grading: catalog-maintenance-0-preflight-01 had native exit 101 and coordinator exit 1, elapsed 1.003560 seconds. Final recovery returned 30 Groups against the producer ledger's 33. The assertion was preserved; neither the fixture nor this failed trial contributes a successful oracle verdict or a performance comparison. The coordinator receipt (local evidence omitted from stage) and native stderr (local evidence omitted from stage) retain the failure and original frozen-source lineage.

Read-only inspection of the relocated failed evidence found Segment 1 with journal_label=1, first_group=last_group=1, one Batch and 128 log rows. The remaining sealed journals contained Groups 5–8, 9–12, 13–16, 17–20, 21–24, 25–28 and 29–32; the active file contained Group 33. Sealed journal 1 was absent. Together with the Segment, retained custody covered Group 1 and Groups 5–33, explaining the three missing Groups 2–4 exactly. Frame-number inspection read the existing FAB1 payload headers; it was not a new workload. The archive originated under $FABRIC_DATA/evidence/fabric-work-ce966aacfb354a499c6d2b9a2bcecc09/query/catalog-maintenance-0-preflight-01/. Original completion and subsequent archive/cleanup receipts remain distinct.

The probe had published a seed-only Segment while its seed journal remained active. Later appends shared that file; rotation labeled it by first Group 1, and Store reopen reclaimed the sealed file because Segment label 1 existed. The Segment represented only Group 1. This was an invalid publication boundary in the synthetic probe, not evidence of a catalog query-selection defect. The correction seals the seed's active journal at its actual first Group before publishing any seed Segment, outside all timing spans. The unchanged producer ledger and exact replay assertion then remain a regression of this failure. Production code, durability order, default plan and oracle were unchanged.

Corrected freeze 02 (local evidence omitted from stage) captured both variants and the correction protocol with complete source/archive hashes. The corrected plain decoded binary SHA256 is bf0aa76d870abc7d2e9ab22b8fab735146dd534a43dabe821462f24f6ee99f44. Build 01 and failed preflight 01 are retained and cannot be mixed into this campaign.

Completed corrected control Coordinator exit / elapsed Trials / complete oracle chains
Rate 0 preflight 02 (local evidence omitted from stage) 0 / 6.513917 s 3 / 6
Rate 1 preflight 01 (local evidence omitted from stage) 0 / 37.608987 s 3 / 102

Both use seed 128 and one typed Segment, followed by 32 durable appends of 128 rows. Their three trials are lazy acquisition, eager refresh after every append, and eager refresh with refresh 32 deliberately omitted. Recorded refresh counts are 0/32/31, and query counts are 0 for rate zero and 32 for rate one. Each trial retains all measured/quiet first pages and complete chains, with six recorded negative controls: changed/missing/duplicated rows and changed framing/missing/ duplicated chunks. Both corrected slice cleanup receipts report success and owned scratch removed. These two tiny controls preceded the full campaign below.

These preflight controls establish exactness for the bounded tiny fixture and missing-refresh fallback under that schedule; their timings are not the full 64-Segment performance comparison.

Historical full-matrix admission checkpoint

The rate-four preflight subsequently passed all three trials and 390 chains in 134.570 seconds, exit 0, 517.7 MiB rounded whole-job peak, zero swap, scratch removed. All nine preflight trials and 498 chains are therefore collected. The full rate-four counted diagnostic then passed both trials and 260 chains in 541.056 seconds, exit 0, about 2 GiB whole-job peak, zero swap, scratch removed. Its native schedules took 6.587/6.748 seconds (lazy/eager); the much larger coordinator elapsed includes full pagination and independent grading. Counted timing is diagnostic and cannot substitute for the registered plain comparison.

At that checkpoint, query-stage use was 2,571.435 of 7,200 seconds; shared frontier use was 4,642.382 of 14,400 seconds. Three remaining rate-four pairs at the measured cost project 1,623 seconds. Four rate-one slices project about 600 seconds from their smaller chain count, and four rate-zero slices about 80 seconds. Those last two are predictions, not measurements. The resulting roughly 2,303 seconds leaves room for consolidation and variation within the 4,628 seconds of query budget remaining. Root admits the unchanged remaining matrix one slice at a time, retaining each runtime checkpoint and all original counts/thresholds. The launcher still rejects any full requested deadline that cannot fit the remaining allocation. No extra campaign budget is created.

Completed plain comparison

The summary run (local evidence omitted from stage) exited 0 in 207.646835 seconds, with 263.98 MiB whole-job cgroup peak, zero swap, no OOM events and owned scratch removed. Its output (local evidence omitted from stage) validated frozen binary/source/fixture provenance, cleanup, exact ledger and reconstructed answer hashes, all 1,826 retained original complete-chain verdicts, and phase/count consistency. All ten consolidation negative controls rejected missing/duplicate slices or verdicts, wrong answer/count/provenance, missing ACK timing, wrong refresh count, changed reconstruction hashes and missing chunks. Each of 33 original trials also retains six answer/archive negative controls. Consolidation checks retained oracle evidence; it does not rerun the semantic oracle.

Independent read-only inspection of all nine plain pairs' raw total_schedule rows reproduced the ratios below. CPU is process CPU; wall is native schedule wall time. Both include all 32 ACKs, refresh/query/serialization and bookkeeping.

Query rate Pair 1 CPU / wall Pair 2 CPU / wall Pair 3 CPU / wall Registered decision
0 4.026279 / 1.683721 3.960921 / 1.652023 3.839338 / 1.640755 H0-compatible
1 1.066480 / 1.035805 1.111137 / 1.098039 1.086309 / 1.044408 H0-compatible
4 1.029634 / 1.014585 1.027738 / 1.025883 0.994231 / 0.991455 H0-compatible

Every ratio is eager/lazy. H1 required all three plain total-CPU ratios below one; no rate met it. Rate four is mixed, with a 0.58% CPU reduction only in pair three, not a repeatable total-cost benefit. Counted diagnostics added 1,546,521,344 requested refresh allocation bytes at each rate. Query allocation savings at rates one/four were only 46,465,808/46,387,760 bytes; rate zero has no scheduled query. These are nonoverlapping phase sums, not total allocation or peak claims.

At rates zero and one, eager costs more total CPU in every pair. At rate zero, refresh has no scheduled query to benefit. At rate one, query CPU changes by −28.544, +19.330 and −8.718 ms; none offsets the added 133–139 ms refresh CPU. At rate four, query CPU changes by +19.423, +12.649 and −138.530 ms against 134.086/133.838/134.917 ms of refresh. The last pair is consistent with a small net benefit, while the first two are not. Serialization, append and bookkeeping also vary, so the entire total difference cannot be assigned to refresh alone.

The source explains why work need not transfer cleanly between phases. refresh (local checkout reference crates/fabric-server/src/read_catalog.rs; not present at origin/main) lists cached manifests, derives coverage, extends the tail and checks publication labels, then drops its discovered descriptors. Each query independently calls view through sources_walk, repeating discovery, coverage and consistency checks before constructing its sources. With this probe's default owned metadata, segments deep-clones cached manifests on both paths. Tail extend remembers frame offsets and can avoid decoding frames already indexed by refresh, but still lists and opens live journal files. Refresh therefore adds repeated acquisition work while moving incremental tail decoding out of the next query. That is a source-grounded explanation consistent with the measurements; the phase totals do not isolate clone, listing, decoding or block-building costs. This serial probe supplies no evidence of lock contention or server scheduling effects.

Two discriminating followups, neither implemented nor admitted here, would be to separate repeated discovery/metadata-copy cost from newly indexed frame cost, and to compare refresh immediately after an append with refresh on an unchanged frontier. The latter distinguishes fixed rediscovery from incremental preparation without assuming a reusable snapshot or changing correctness expectations. Counted diagnostics do not replace the plain total-CPU criterion. Native ACKs per schedule wall time remain schedule throughput, with no Spool acceptance, concurrency, qualification or adoption claim. Final aggregate resource auditing and documentation checks are separate from this completed measurement result.