Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/catalog-labs-capacity.md Original SHA-256: 860bab49ac29fb50d3bf44c109da8ea1ef8a3f50f1503c2cab01cdd9bbb2d3a0 Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Storage-map and Rust capacity experiments

Status: planning packet; not registered, implemented or executed. Scope: development/small workloads; bounded memory, CPU, successful Spool acceptance and ACK/query service cost. This packet authorizes no qualification, oracle/policy changes, unsafe code or persistent-format changes. Follow experiment guidance and the frontier controls. Freeze each protocol, baseline revision, candidate diff, seed and decision rule before measurement.

Source observations and order

The spill codec (local checkout reference crates/fabric-server/src/segment/bounded/spill.rs; not present at origin/main) allocates a byte vector for each written/read row; decoded strings/maps remain owned. Merge (local checkout reference crates/fabric-server/src/segment/bounded.rs; not present at origin/main) has sixteen readers/heads and byte-capped runs. Reuse scratch first; it is smaller than changing projected-row ownership or introducing a catalog.

The tail block visitor already uses borrowed ObservationRef over a decompressed buffer. The Walk log execution filters and checks its top-k threshold before making an owned log row. Do not claim a new borrowed tail decoder or repeat that existing optimization. Parquet log loading in segment.rs creates owned bodies/attributes after time filtering, before consumer rejection. WalkState::segments caches manifests but clones them into each query's sources; source loading holds its mutex. Filters/tail blocks already use Arc. These are candidate costs, not measured dominance.

CR1: reusable bounded spill workspace

H1: reusable encode/decode byte vectors reduce allocation traffic and CPU at unchanged spill/output bytes, without increasing retained working memory. H0: owned decode/output costs dominate, or retained capacity cancels the benefit. Baseline: current binary codec and partial merge plan. Variable: encode scratch per run writer; separately, decode scratch per reader. Compare each alone before combining. Bound capacity explicitly; discard oversized buffers at a declared threshold. Sixteen reader buffers must be charged sixteen times, not once.

Fixtures: frozen original C2 inputs/hashes at steady 64/256 MiB and bigrows64; supplement with seed 2703163393, alternating small/large rows and long attributes, and actual 16/17/18-run boundaries. Do not redefine historical C2 acceptance. Three alternating fresh-process pairs per admitted fixture; separate warm reuse cycles from fresh trials. Match journal bytes and row contents/order exactly. Measure allocations/count/bytes, exclusive CPU, observer-off build wall time, incremental owned heap, post-build retained capacity, RSS/HWM, logical spill writes, physical IO, peak scratch and cleanup. Separate accounting instrumentation from timing. Require identical final files/manifests, signed scalars/float bits, stable ties, malformed/truncated-row rejection and raw Batch custody. Counterexamples: one maximum row permanently enlarges every merge reader; reuse occurs before synchronous write completion; stale bytes escape after clear; decode ownership is merely relabeled. Preserve each meaningful failure fixture.

CR2: selective borrowed Parquet materialization

H1: predicate/key inspection on borrowed Arrow values before ownership conversion saves allocations/CPU on rejected rows. H0: most decoded rows survive, or parsing, decompression and output dominate. Baseline: current owned visitor; variable: private scoped log visitor borrowing one RecordBatch, with owned conversion only for selected rows. Start with logs; metrics/rates/spans need separate evidence. Do not change public contracts or bypass malformed attributes/column validation.

Fixtures: seed 2703163393; H64/H256, narrow and broad windows, absent/rare/common text, small/full pages, ties, Unicode and large attribute maps. Hold query plan, request starts/counts and pagination fixed; include a tail-only control where borrowing already exists. Charge complete Arrow buffers and validation work. Measure allocations, retained answer heap, decoded-buffer lifetime/bytes, CPU, latency and IO; preserve all pages against the unchanged independent oracle. No view may outlive its batch or remain in top-k/page state. Owned winners must survive callback return. Borrowing that pins a large batch for a small answer, changes rejection/error behavior or retains attributes longer is a counterexample.

CR3: immutable catalog snapshots and reclamation

H1: sharing immutable manifest/bounds/filter metadata reduces per-query clones and lock contention under fixed demand without accumulating obsolete generations. H0: metadata costs are small, publication cloning costs more, or slow queries pin enough old generations to worsen service memory. Baseline: current mutex-backed Walk cache/source assembly; variable: private Arc metadata snapshots with one explicit publisher. First compare metadata sharing alone; do not move query execution, invent generic storage traits or add a persistent catalog format.

Fixtures: seed 2703163393; H64/H256 at equal total history but differing Segment counts; fixed query requests with 1/4 concurrent consumers; normal/burst/drain, seal and byte/age retention transitions, paused reader and canceled query. Freeze one worker, CPU placement and query plan. Keep page snapshot/expiry, missing-table/index fallback, freshness and custody semantics unchanged. An Arc pins metadata, not filesystem custody: do not silently defer retention or reinterpret a deleted file as readable. Token lifetime is not an instruction to retain an in-memory snapshot indefinitely.

Measure active unique metadata bytes, transient publication-copy overlap, retained obsolete bytes/generation count and age, reader lifetime, lock wait/hold time, directory/manifest reads, allocations, RSS/cgroup anon/file/kernel and pressure. Count shared allocations once by identity; report reader references separately. Pause one reader while publishing multiple generations, then drop/cancel it and observe explicit reclamation and allocator-retained capacity after quiescence. Counterexamples: stale mixed generations, unbounded retired Arcs, expired pages retaining history, or a publisher dropping metadata still borrowed by a reader.

Composite confirmation and selection

For any surviving lever, run matched development/small native profiles with fixed offered queries, producers and SDK signal mix. Keep total completed query work visible. Retain server/Spindle CPU, successful Spool Batches/s AND bytes/s, attempt/error denominator where available, ACK/query latency, backlog slope/drain, journal reclaim, output/scratch/physical IO and exact recovery alongside memory. Register effect/no-regression bounds and finite command/resource budgets first; execute through the resource launcher on the mounted data drive. Advance at most one candidate after independent correctness and service-cost confirmation.

The shared catalog plan (local checkout reference docs/experiments/benchmarks/storage-catalog-rust-lab-plan.md; not present at origin/main) supplies the decision thresholds, stop rules and 45-minute initial lab allowance within the existing four-hour campaign total. CR1 is the first screen; CR2/CR3 remain queued until attribution and remaining budget justify them. Missing fixture parameters and real executable commands must be frozen in a separate protocol before launch.