Historical report imported 2026-10-10 from docs/experiments/benchmarks/catalog-labs-run-01.md. Original SHA-256: 650db84250aade34a607b6c3dbcd0355c28891b4d7f10422e439163c995d6158. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Storage catalog and Rust labs, run 01

Status: admitted experiments complete; failures and non-admissions retained, 2026-10-07. Owner authorized queuing all experiments and beginning tests. Plan (local checkout reference docs/experiments/benchmarks/storage-catalog-rust-lab-plan.md; not present at origin/main), machine-readable queue (local checkout reference docs/experiments/benchmarks/catalog-lab-queue.json; not present at origin/main) and prior results apply.

Completed measurement outcomes

The following is the latest result; the subsequent admission and checkpoint sections preserve the run's history. A collected experiment is not necessarily a promoted optimization, and successful mechanisms do not clear a failed gate.

Experiment	Completed evidence	Decision
CQ1 storage boundary	18 trials, 1,152 exact complete chains; all three CPU/wall/HWM guards met	Supports the corrected narrow Walk boundary on this workload
CR1 full-builder reuse	48 builds and 14 controls	6.57–7.54% allocation savings missed the 10% target; no promotion
CR2 borrowed rows	12 trials, 768 exact complete chains	24.61–24.68% selective allocation savings; one plain pair's CPU/wall misses block promotion
CR3 shared metadata	48 trials, 96 canonical complete chains and 3,072 exact first-page associations	About 64–67% lower absent-query allocation peak, but 5.3% total allocation savings and two p95 misses fail the full rule
CQ2 catalog maintenance	33 trials, 1,826 exact complete chains; all 15 slices consolidated	No read rate met the all-three-pairs CPU improvement rule; zero/one-query schedules worsened, four-query schedules were mixed
O1–O5 coverage and lifetime	Mixed sources, selected publication/reclaim cuts, fallback, expiration and release controls	Reproduced and corrected silent discovery omission in both Walk and Scan; selected schedules only

Detailed boundary, builder, borrowed-row, metadata, maintenance and integration findings retain raw evidence and limits. The four main query comparisons account for 3,842 complete-chain checks; the 3,072 associated CR3 first pages are not additional independent drains.

F1 service confirmation is not admitted: component optimization guards did not nominate a candidate. No composite, Spool-throughput or deployment claim is inferred. No experimental optimization was enabled by default. The inherited discovery defect is a query omission window, not evidence that durable bytes were lost. History retries the corrected explicit interruption.

Final fast verification remains failed: catalog-fast-03 passed 16 gates and failed workspace tests. catalog-storage-regression-01 captured seven passing tests, one known partial-table metadata/oracle-model failure and one ignored subprocess helper. Neither the oracle nor its expected outcome was weakened. Historical C2 pruning-equivalence failure also remains unresolved.

catalog-final-failure-cleanup-01 exited 0 after verifying archive membership, sizes and SHA-256 for eight retained failure directories, then removing 446,561,074 bytes of scratch. Original failures remain failures. Successful trials cleaned their owned scratch; the shared Cargo cache remains on the data drive. The final aggregate inventory is below 2 GiB. See the closeout below; the capacity lab exceeded its smaller allowance. No limit was retrospectively raised to pass this result.

Final closeout and model review

catalog-campaign-summary-04 ran the registered resource wrapper and python3 -B tools/bench/labs/catalog/campaign_summary.py, exit 0, 2.007 seconds, about 67.3 MiB whole-job cgroup peak, zero swap and owned scratch removed. Its raw audit (local evidence omitted from stage) includes original commands, exits, CPU, sampled process RSS/HWM, cgroup memory, IO and scratch observations. All recorded catalog jobs used 20 GiB maximum, zero swap; highest whole-job cgroup peak was 9,744,941,056 bytes (about 9.08 GiB), with no recorded memory high/max/OOM events. This includes builds and graders; it is not application RSS or a small-deployment memory claim.

The complete persistent catalog footprint was about 1.58 GiB allocated, below the unchanged 2 GiB aggregate. Query's conservative upper bound fits 1 GiB. Capacity's mandatory datasets, baseline and preserved CR3 failure archive total 811,618,689 unique bytes and 815,308,800 allocated bytes, exceeding the 768 MiB allowance by 6,312,321 and 10,002,432 bytes respectively (about 9.54 MiB allocated). This is a confirmed resource-accounting failure, not an inconclusive upper bound. Frozen admission inventories omitted archive categories. Future admission must reserve failure evidence and count the complete inventory before dispatch. The registered attribution (local checkout reference docs/experiments/benchmarks/catalog-capacity-attribution-protocol.md; not present at origin/main) preserves the cap and this failure. No further native workload is admitted.

Audit 01's conservative capacity result was inconclusive. Audit 02 accidentally included partial, unregistered 832 MiB draft helper edits during coordinator/PI overlap; its exit zero does not certify that allocation. Root withdrew the draft, retained its record, and repeated with the original cap. Audit 03 captured the correct original-cap outcome; audit 04 verified the reconciled helper after the PI stopped editing. A leftover output label claiming receipt-owner controls was removed after audit 04; the final reconciled helper is verified in audit 05. No native experiment result or frozen driver was changed.

Final catalog-campaign-summary-05 exited 0 in 2.004 seconds, about 67.4 MiB cgroup peak, zero swap, scratch removed. Its snapshot was 1,693,298,688 allocated bytes; read-only post-completion inode accounting including its own output and receipt was 1,693,552,640 bytes (253,952 bytes added), still below 2 GiB. Original capacity allowance failure remains unchanged. Total recorded frontier elapsed time was 9,192.043 seconds of 14,400; cgroup source capture is outside that recorded elapsed boundary. No catalog receipt remains running and no recorded catalog scratch root remains on disk. Shared build caches are intentionally kept.

Manual documentation validation catalog-docs-03 exited 0. The final documentation pass after the new model review, catalog-docs-04, also exited 0 in 16.041 seconds with about 253.3 MiB cgroup peak, zero swap and scratch removed. Documentation links, checker probes and hook tests passed; Bun CI/hooks remain disabled. The campaign is complete with null results and known verification failures; F1 remains not admitted. This is not release qualification.

The owner's supplied research direction and coupled model are reviewed in coupled performance model review. That review updates stale implementation assumptions, corrects units/model limits, and proposes the next experiments by lab without launching a new campaign.

Dispatch and admission

Lab	First admitted preparation	Dependent queue
Capacity	CR1 isolated real-codec buffer reuse probe	CR2 borrowed Parquet inspection; CR3 metadata snapshots
Query	CR3-P real-Manifest clone/Arc precursor	CQ1 storage read boundary; CQ2 maintenance timing; Q3 attribution integration
Operations	O1–O5 deterministic source coverage fixtures	Partial-table checker design, then capped service tests

Capacity and query probes are isolated lab mechanisms, not production changes. Operations uses actual history/storage fixtures. Root serializes every build, test, measurement and validator in the 20 GiB/no-swap cgroup. Agent preparation may overlap. PIs cannot independently launch workloads.

The shared four-hour frontier budget had consumed 445.192 seconds before this wave, including catalog plan documentation validation. run_job.py now enforces that aggregate deadline allocation for frontier-* and catalog-* job IDs and captures catalog protocols and queue with each source snapshot. Existing stage and scratch/evidence limits still apply. No new four-hour allocation was added.

Registration commits, exact commands, exits, resource observations and cleanup will be recorded here as jobs finish. No outcome is implied by queue admission. The historical C2 and partial-table verification failures remain unchanged.

Workload scope

CR1 tests allocations in the private spill codec, using existing encoding as the baseline. CR3-P tests metadata ownership using real Manifest structures. Neither establishes end-to-end query, ingestion or deployment performance. Benchmark-only allocation instrumentation follows the existing System allocator wrapper pattern; candidate buffer/ownership algorithms use safe Rust.

Each failed control or semantic mismatch stops its dependent comparisons. A negative performance result stops promotion, while independent lab work may continue. Snapshots retaining memory never grant expired records new query rights.

First-wave results

Protocols were separately committed before their workloads: coverage 8e8351b, spill c4b27fe, metadata 71cba75. Each coordinator receipt captures the relevant working-tree source snapshot; a later revision does not inherit that result.

Lab / job	Actual outcome	Scope
Operations / catalog-coverage-01	3 tests passed; 422 oracle acceptances and 3 expected rejections, all artifacts retained	Sequential real active/pending/published/reclaimed states, snapshot continuation, retention Gone and optional-filter fallback
Capacity / catalog-spill-01	All 36 codec trials preserved exact bytes and controls	Combined reuse cut requested allocation bytes 58.26% and allocation calls 28.57%; plain paired median codec wall about 7.6% lower
Query / catalog-metadata-01	All 24 trials and mutation/drop controls completed	Arc acquisitions took about 19.6–19.8% of clone wall cost with one reader, 18.7–18.8% with four, at fixed total work

Detailed coverage findings, spill findings and metadata findings include counterevidence and limitations. Root independently derived paired ratios in data/lab-completion-run-01/coordinator/catalog-summary-01/stdout.txt under containment, exit 0. Spill findings also show ratios of medians; the paired-median summary above is explicitly a different estimand and supports the same nomination.

For CR1, incremental requested peak heap increased from 2698 to 3807 bytes, despite fewer cumulative allocations. Combined reusable capacity was 2218 bytes until drop; sixteen-reader effects still require full-builder measurement. Whole-child CPU improved only about 0.6% by paired median. This nominates reuse for confirmation, not a service speed claim or automatic production promotion.

For metadata, allocation request bytes fell about 99.7%, but the fixture measures metadata acquisition/consumption only. Real query discovery, decoding, locking, IO and publication are absent. Paused handles retain old generations until dropped; object lifetime does not extend logical retention. A catalog prototype must measure those costs before inheriting this result. In one counted lifecycle control, eight held generations increased requested live bytes by 1433560 (about 1.37 MiB), including tracking structures. After dropping the views all tracked Manifest objects were released. This illustrates why the integrated catalog needs a bound on live readers/generations even when individual acquisitions are cheap.

Resource and verification checkpoint

Job	Exit	Job seconds	Whole-job peak reported by systemd	Cleanup
coverage	0	48.331	1.0 GiB rounded (receipt: 1141710848 bytes)	Success scratch removed; 26.96 MB complete oracle evidence retained
spill	0	29.564	963.9 MiB	Success fixture and frozen-binary scratch removed; compressed binaries archived
metadata	0	14.017	179.5 MiB	Success scratch removed; compressed binaries archived
new-target Clippy	0	4.506	285.8 MiB	Removed

No swap was used. Peaks include builders/graders as applicable and are not application RSS. Commands run through resource_group.py and run_job.py are captured in the matching coordinator job directories. Targeted Clippy covered both new examples and catalog_coverage with all features and warnings denied. It does not clear the earlier full-workspace formatting/test/Clippy failures.

That first-wave checkpoint preceded the later CQ1 and operations results below. CQ2 maintenance, CR2 borrowed Parquet and CR3 performance comparisons still await final conclusions, as does service-level F1. Selected O1–O5 schedules now include numeric controls and paused publication/reclaim cuts; exhaustive races and the partial-table availability-model mismatch remain outside their completion.

Documentation validation catalog-docs-01 initially exited 1: a frozen protocol copy inside the evidence directory retained its .md extension, so the checker interpreted its original relative links from the archive location. The copy was renamed to .txt without changing any bytes (SHA-256 preserved in the archive path-change receipt), and future copies use that suffix. No checker rule or experiment result was changed. The original failed validation is retained.

Retry catalog-docs-02 exited 0: documentation links, checker probes and hook tests passed. New-target formatting check also exited 0. Aggregate recorded frontier/catalog job time at this checkpoint is 576.175 seconds of 14400; 13823.825 seconds remain. All launched jobs have terminated. Successful owned scratch was cleaned; the failed documentation job's scratch and earlier failure evidence remain preserved. Required full-workspace failures from the prior wave remain open and are not hidden by these targeted passing checks.

Second-wave admission

The owner explicitly requested an active goal to drive queued experiments to completion within the resource envelope. The same three PIs are preparing full-builder workspace confirmation (capacity), the real catalog boundary and maintenance/snapshot comparisons (query), and remaining coverage/interleaving controls (operations). Root retains exclusive workload admission. Conditional experiments require their prerequisites; a null result is a valid outcome and does not authorize promoting a mechanism or relaxing a decision rule.

Before source extraction, catalog-preboundary-build-01 and catalog-preboundary-build-02 built and archived the current completion_query_probe, plain and allocation/phase-instrumented respectively. Both commands exited 0; these are preparation receipts, not measured CQ1 results. Their source snapshots and binary hashes distinguish the old boundary from the forthcoming candidate. catalog-baseline-freeze-01 also preserves historical example binaries without asserting that they match current source. All three preparation jobs removed their owned scratch.

The native signal harness now accepts an explicit frozen binary directory and query-plan selection, retaining its previous defaults. Composite comparison admission and protocol registration await component evidence; no native trial has yet run through those new options.

Second-wave evidence so far:

• catalog-mixed-sources-01 failed a fixture-only assumption that directory labels arrive sorted. The corrected local set comparison passed as catalog-mixed-sources-02, including 120 independent query acceptances and exact raw replay. The original failure was fully byte-verified into an archive and its 4,223,853 bytes of owned scratch removed by catalog-mixed-failure-cleanup-01 (exit 0).
• catalog-numeric-bounds-01 passed: two absent-statistics answers accepted and four deliberately false-bound answers rejected. This tests fallback and checker sensitivity, not semantic authentication of arbitrary footer statistics.
• catalog-builder-spill-01 completed 48 full builds and 14 controls with exact output. Requested allocation savings were only 6.57–7.54%, below the registered 10% target in every counted pair. Peak heap was effectively unchanged and time effects mixed. The candidate is not nominated. Independent arithmetic ran as catalog-builder-summary-01, exit 0; see builder findings.
• catalog-discovery-race-01 reproduced a real selection omission: publication and reclaim between directory discoveries left an acquisition with zero of 24 committed Batches, without a retry. Durable Segment bytes remained present. A published-label consistency check now returns Interrupted so the existing bounded retry can obtain exact coverage. The unchanged fixture passed in catalog-discovery-race-02. Scan reproduced the same omission in catalog-scan-discovery-race-01 and passed its unchanged regression in run 02 after the analogous correction; both original failures remain recorded.
• catalog-paused-cuts-01 passed real paused manifest/checkpoint cuts, retained Scan/clone/shared readers, reclamation and Gone after age retention. These are selected schedules, not exhaustive concurrency verification.

New protocols were separately registered before their workloads: mixed/bounds b1e9878, builder ed5b91a, paused cuts e977fb8, discovery gap a2625c1. The explicit transcript selector/page-limit adapter was separately registered in 0273c2a; catalog-transcript-controls-01 exited 0. Legacy defaults and the independent query oracle remain unchanged.

• catalog-discovery-cold-controls-01 passed four independent cold/primed and clone/shared one-Batch discovery cuts. catalog-lifetime-01 passed real metadata identity, retention, last-view drop and Tokio cancellation controls. These establish selected ownership behavior, not a bound on arbitrary reader counts or memory bytes.
• catalog-boundary-diagnostic-01 completed six small preflights and three full-size counted trials, exit 0, 411.686 seconds. Whole-job cgroup peak was 4.2 GiB as rounded by systemd, with zero swap; owned scratch was removed. The later three plain paired slices completed, as recorded below. The candidate includes discovery correctness corrections, so this comparison cannot isolate the ownership extraction's overhead alone.

Scan discovery was registered as b56bacc; cold acquisition and lifetime as 4674ef5; CQ1 as 6343059; CR2 as 50185cf. Each result belongs to its captured source and frozen binaries. Ongoing measurements do not clear historical C2 or partial-table verifier-model failures.

Evidence accounting correction

During CQ1 pair 1, root detected that separate slice directories each obeyed profile.check_space but their combined stored evidence exceeded the planned 256 MiB query-lab cap (diagnostic approximately 199 MiB, partial pair 1 approximately 68 MiB). This is an aggregate admission/accounting defect, not an application-memory result. No subsequent workload is admitted until completed archives are compacted and aggregate footprint reviewed. Exact result bytes and all oracle outcomes must survive; logical path bytes and uniquely stored bytes will both be recorded. Earlier campaign archives are inventoried separately rather than silently excluded from the total data-drive footprint.

The separately reviewed persistent allocation (local checkout reference docs/experiments/benchmarks/catalog-evidence-allocation-protocol.md; not present at origin/main) was registered as 818ef87: query 1 GiB, capacity 512 MiB, catalog aggregate 2 GiB. This changes disk evidence allocation only; memory, swap, scratch and time limits are unchanged. The original compaction exited 2 after retaining all exact bytes; it is not retrospectively passed. Under the reviewed allocation, catalog-evidence-compaction-02 exited 0 in 15.513 seconds. The archive admission amendment and CR2 slice accounting were registered as be6ad24.

catalog-boundary-pair1-01 exited 0 in 366.057 seconds, rounded whole-job peak 4.4 GiB, zero swap and successful scratch removal. Its clone/baseline ratios were 0.9590 wall, 0.9602 CPU and 0.9964 measured native HWM. That single pair met the registered guard; pairs 2 and 3 subsequently completed with the same frozen binaries, shared exact objects and aggregate evidence admission.

catalog-discovery-failure-cleanup-01 exited 0. Both failing discovery fixtures were fully archived with byte/size verification before removing 4,304,531 bytes of owned scratch. Their original exit 101 receipts and counterexamples remain. The compaction failure's empty scratch was also cleaned; evidence remains intact.

Consolidated operations/integration checkpoint

CQ1 findings record all 18 trials and 1,152 exact full-chain acceptances, with all three plain CPU/wall/native-HWM preservation guards met. CR1 is a completed null nomination: 48 exact-output builds and 14 controls, but the primary allocation-byte reduction guard missed. The selected operations controls are completed, including mixed sources, numeric mutations, paused cuts, both discovery regressions and cold/lifetime controls; no full race proof follows.

Integration findings retain catalog-fast-02 exit 1: fixture namespace collision and Clippy diagnostics. Unchanged targeted parallel retry catalog-parallel-fixture-02 exited 0, and full workspace Clippy catalog-clippy-03 exited 0 after corrections. Full fast remains pending. CQ2/CR2/CR3 final performance conclusions are not claimed. Maintenance preflight catalog-maintenance-0-preflight-01 exited 1 after a native exit 101 and remains under diagnosis. Historical C2 and partial-table oracle-model failures stay open.