Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/catalog-labs-query.md Original SHA-256: 2cbf1b4ebb81808de79d890778a2bb966b60ebe26485bf0ceb184ea3df42ecce Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Storage read catalog: query lab planning packet

Status: proposed; unregistered and unimplemented. No execution is authorized by this packet alone. Frontier queue, Q3 preparation (local checkout reference docs/experiments/benchmarks/query-frontier-protocol.md; not present at origin/main) and current prerequisite outcomes apply. Keep Scan as the production default, custody bytes and sync order unchanged.

Existing mechanism and proposed seam

History owns a mutex-protected WalkState; sources_walk discovers Segments, extends the journal index, selects snapshot sources and loads bounds/optional filters. run retries moved sources at most three times. Query ordering, thresholds, pages and rate semantics remain in query/core code, not the catalog. Tail state already caches immutable Segment metadata and derived compressed blocks. visit_block decompresses a block and uses borrowed fabric_observation::decode_view observations; selected rows become owned values. A catalog is therefore an ownership/lifecycle seam, not the invention of borrowed decoding or a new durable index.

Propose one internal fabric-server storage module, without new crates or generic repository traits. A typed CatalogRequest carries the committed-group frontier, optional snapshot floor, table, time window and node/source hints. ReadCatalog::view(request) returns a typed owned CatalogView: source bounds, locators, immutable block handles and retained/freshness/gap evidence. Names are design sketches, not implemented APIs. Final types must distinguish optional index absence from raw-source failure and a moved source from expired history. Logical row filtering, pagination fingerprints and answer construction stay out.

Rust dimensions: enum dispatch for finite source kinds; owned descriptors and Arc block handles for view lifetime; borrowed records restricted to a decode buffer's lifetime; narrow mutex scope. Release the catalog lock before IO/decode callbacks. An Arc keeps memory alive, not retention rights or completeness: validate every page against live retained coverage. No unsafe lifetime extension, trait-object plugin registry, event bus, wire-format or durability change.

CQ1: behavior-preserving storage ownership

H1: extracting existing Walk discovery/cache state behind the seam preserves answers and bounded retries without materially increasing CPU or memory. H0: ownership transfer creates stale/duplicate coverage, changes metadata/errors, or adds material overhead. Baseline is current Walk; candidate changes ownership and API only. Keep discovery cadence, block size 4096 and decoder unchanged.

Reuse Q3's exact fixture: 65536 logs, requested 1024-byte bodies, 128 logs/Batch, seed 42, sorted/shuffled timestamp-rank cells, limits 100/10000, four existing shapes and tail/Segment layouts. Start at 128 rows. Preserve Q3's timestamp assignment caveat, exact commands, plain/counting separation and full chains; do not rerun a duplicate matrix merely to rename the abstraction. Candidate measurements reuse the registered Q3 estimands once its harness gaps are closed.

Add small deterministic lifecycle schedules using completion storage fixtures (local checkout reference crates/fabric-server/tests/completion_storage.rs; not present at origin/main): existing fixture(1,8,2000000) and recorded seed 7, with a predeclared ledger. Freeze cut positions before registration: first page; append beyond snapshot; builder paused before manifest publication; published Segment before reclaim; reclaim; restart; byte/age eviction. Drain the original snapshot at each cut, then query a new snapshot. No duplicate journal/Segment coverage, leaked later groups, expired-page success, false completeness or changed freshness/gaps.

Grade complete answers with the unchanged query/rate oracles; exact custody ledger precedes candidate output. Keep tie-key, quiet-source, expected-gap, metric/rate and span controls from existing suites. Inject an omitted source, duplicate coverage, changed continuation and stale snapshot floor; each must fail its independent check. Missing/corrupt optional filters must scan exactly; missing/corrupt raw sources must retain documented incomplete/error behavior. The current missing-table metadata/oracle mismatch is unresolved: preserve it and block acceptance of affected corruption claims pending a separate decision.

Primary seam guard: no more than 5% regression in matched broad-tail warm CPU, wall cost or measured HWM in each of three fresh alternating baseline/candidate pairs; zero semantic mismatch. This is a preservation screen, not a speed claim. Report descriptor/cache bytes, lock hold/wait time, discovery reads, moved-source retries and lingering block ownership after eviction in separate diagnostics.

CQ2: lazy versus eager catalog maintenance

Depend on CQ1 exactness and registered Q3 attribution. Baseline refreshes lazily on queries. Candidate incrementally refreshes from already committed/publicized storage transitions, outside existing ACK/sync ordering; unfinished maintenance must use exact lazy refresh/fallback. No query may trust an unvalidated stale view. Change maintenance timing only; same indexes, bounds, block size and decoder.

Reuse seed-42 65536-row history and deterministic 128-log append Batches. Freeze three schedules: no queries until 32 append Batches; one query after each Batch; four queries after each Batch. Append 32 Batches total; alternate selective and broad Q3 shapes at limit 10000. These are event schedules, not offered-rate or concurrent-capacity claims. Retain scheduled/started/completed query counts and full chains. Repeat publication/restart/eviction cut cases from CQ1 unchanged.

H1: eager refresh moves work off first queries and amortizes it under repeated demand. H0: it only relocates CPU, wastes work when queries are absent, expands memory or worsens ingestion/ACK latency. Measure maintenance + ingestion + query CPU per appended MiB and per completed query, first/warm latency, ACK timing, refresh lag, decode counts, persistent cache/RSS and post-eviction ownership. Keep global total CPU primary; separately attribute index decoding, block encoding, source discovery, borrowed block traversal and JSON construction. Counterexample: first query improves while total CPU or ACK latency regresses; record the trade-off rather than declare a winner. Inject skipped/stale maintenance and prove exact lazy repair; otherwise fail. Never convert lag into missing rows.

Admission and bounded execution

Register each experiment and required harness/checker changes separately before measurement; checker controls and expected outcomes are trust-boundary changes. No executable catalog flags exist. Reuse completion/Q3 tooling after explicit adaptation; root freezes actual commands, binaries and source/oracle hashes. Initial screen allowance: at most 45 minutes including builds and graders, serial jobs through python3 -B tools/resource_group.py -- COMMAND, each below 30 minutes, mounted-drive scratch at most 8 GiB and evidence at most 256 MiB, 16 GiB free reserve, existing 16/20 GiB high/max limits and zero swap. Three fresh alternating pairs are required for confirmation; missing pairs mean incomplete evidence. Stop on mismatch, OOM, enforcement/clock failure or storage floor. Archive failures before cleanup and retain command/exit/resource/cleanup receipts. No qualification, destructive new fault run or default change is implied.