Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/coupled-query-preparation.md Original SHA-256: 3f9016b7145a0c14dd6850ba7a1f89d03a61b5c5c9e507005b1c8f95337f7256 Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Q4 bounded physical descriptor reuse: preparation

Status: opt-in candidate and additive controls staged; not registered or executed. No workload run by this packet. Q4 and the common plan govern admission. The contract is unchanged exact rows/envelopes, pagination, explicit movement retry, optional-index fallback and retention expiration; Scan remains default.

Mechanism and comparator

refresh (local checkout reference crates/fabric-server/src/read_catalog.rs; not present at origin/main) currently derives coverage then drops descriptors. view repeats manifest acquisition, paths and coverage construction. WalkState::segments caches parsed manifests; it does not cache the assembled discovery result. CQ2 measured added refresh cost; CR3 measured metadata ownership separately. Reusing descriptors is therefore a different variable from replacing deep manifest clones with Arc clones.

Smallest candidate: one storage-owned immutable published-Segment descriptor snapshot, not cached query-specific Sources. It contains sorted labels, physical paths, immutable manifest handles and sorted coverage ranges. It has a local generation and immutable byte accounting. Active/uncovered sealed journal files, pending tail rows and blocks remain in existing WalkState and are extended on every acquisition; no duplicated tail index or retained raw journal payloads. Queries still derive their frontier/window/node/table selection and evidence. Row-group bounds/filter caches and their current fallback behavior stay unchanged.

Fix metadata ownership to shared handles on both experimental arms using existing History::with_shared_catalog; lazy acquisition without descriptor reuse is the matched baseline. Keep default-owned acquisition as an exactness control, not a timing denominator. Otherwise a result confounds CR3 ownership with Q4 reuse. Both arms retain the same manifest identities; candidate additionally retains one Arc descriptor container. Internal Sources must reference that container plus selected indices, rather than clone its descriptors back into a Vec. There is no public query API, generic port, new crate or persisted format change.

Invalidation, bounds and fallback

Every acquisition must freshly list all published labels, before frontier selection. Equal labels permit descriptor-container reuse; a changed set rebuilds paths/manifests/coverage. After ordinary tail extension, re-list labels and retain the existing changed-set Interrupted retry. Never key validity only by committed frontier: retention/publication can change sources without a new append. The existing discovery cut must execute even on a cache hit. Named missing manifests, journal movement and unavailable table behavior remain explicit. This retains directory listings and journal opens; it removes repeated physical descriptor allocation/path creation/coverage sorting on unchanged publication. It does not remove tail decoding after append or query row materialization.

Proposed experimental admission bounds: 4 concurrent snapshot holders, one current container, at most 4 held retired containers, and an 8 MiB conservative candidate charge across generations. Shared metadata is deliberately charged again for each generation, with 1 KiB reserved per BTree entry; this target-stdlib model is not an exact allocator or RSS bound. Hard caps independently bound 256 descriptors/generation, 256 entries per manifest map and 4096-byte string/path capacities. Query index vectors add at most 256 indices per admitted holder and are reported separately. Check conservative size before publishing a new container. Exceeding the bound disables reuse for that acquisition and runs the unchanged lazy path; never evict a live reader or change its answer. Admission accounting and reader guards must release on success, error and cancellation. Reject implementation if bounded accounting cannot be established without creating a larger framework.

Metadata Arc ownership is not a file lease. Retention may remove files while descriptors are held; existing read errors/retry and live oldest-floor Gone checks still apply. Oldest retained must come from freshly validated live sources, not a held old container. A per-process generation is not a durable epoch or atomic filesystem snapshot; no guarantee against arbitrary filesystem mutation or label reuse is inferred from the existing labels-only consistency check.

Correctness before timing

Root can execute these existing commands under the enforced launcher; they are baseline controls until the candidate is added to their fixture constructors.

python3 -B tools/resource_group.py -- cargo test --offline --locked -p fabric-server --lib read_catalog::race_tests -- --nocapture
python3 -B tools/resource_group.py -- cargo test --offline --locked -p fabric-server --lib query::discovery_race_tests -- --nocapture
python3 -B tools/resource_group.py -- cargo test --offline --locked -p fabric-server --lib catalog_lifetime_tests -- --nocapture
python3 -B tools/resource_group.py -- cargo test --offline --locked -p fabric-server --test catalog_coverage -- --nocapture

Candidate-specific gaps: exercise cache-hit publication/reclaim and cold misses; append with unchanged Segment labels; held container across retention; actual task abort; four-holder/cap overflow exact fallback; equal-time continuation. Use existing seeds/producer ledgers and unchanged Python query oracle. Add omitted/duplicate descriptor, accepted stale generation and leaked retired-owner controls; each must fail independently. Keep all historical race/availability failures. Missing-table model conflicts remain O6's gate, not permission to omit envelope checking or promote Q4 while full fast is failed.

Minimal harness route and admission estimate

Before editing production, root freezes the current working tree. Add a hidden experimental History constructor for descriptor reuse and a separate probe/driver; do not modify frozen CQ2/CR3 measurement drivers. Reuse the corrected CQ2 seed publication procedure, seed42/H65536/1024-byte bodies, 64 typed Segments, 32 128-row appends, broad/selective limit10000 and complete original oracle chains. For the initial evidence bound, separately register four appends (512 rows) instead of CQ2's32; do not change the frozen CQ2 fixture or chain grading. Tiny preflight uses H128/1 Segment/1 query per append. First screen is one fresh alternating plain pair at 64 Segments/rate1, after lifecycle controls, with a separate counted diagnostic. This is a screen, not the three-pair nomination. Rates0/4 and 1-Segment timing remain explicitly deferred until that screen fits and warrants expansion. Primary is total ACK+acquisition+query+serialization CPU; also retain native wall/first-page latency, full-chain grading and bounds/counters. No explicit eager refresh in either arm: lazy acquisition versus lazy reuse.

Harness work still required: fixed-ownership selector, bounded generation counters, descriptor hit/rebuild counts, actual source listings/open/decode accounting, exact answer association, and freeze/archive/admission receipts. No executable candidate CLI exists yet, so this packet invents no command for it. Separate protocols must freeze selector, commands, workload identity, metrics and controls before timing. Historical Walk gap/lifetime controls took roughly seconds each; paused controls roughly37s. Budget 180s for build+selected controls and 300s for preflight+one plain pair+counted diagnostic, provisional within Q4's1200s ceiling. Use the authorized catalog-boundary-reuse-* data prefix. Reserve at most16MiB incremental evidence for the initial screen, including source/binary archives and failures; exact compressed/chunked archives and measured aggregate headroom determine admission. These are projections, not measured new results or adopted allowances. Stop or defer if time/evidence limits cannot preserve every complete chain.

Staged correctness handoff

New coupled_catalog.rs and the hidden History::with_descriptor_reuse implement the candidate. Sources now uses an internal indexed descriptor wrapper; query consumers keep the same order and raw reads. New unique coupled control files reuse the original producer/oracle and real lifetime helpers, through additive includes. Root can run the candidate controls (no result claimed):

python3 -B tools/resource_group.py -- cargo test --offline --locked -p fabric-server --lib coupled_ -- --nocapture

Five tests cover actual cache-hit publication/reclaim with explicit Interrupted, exact retry, append without publication and unchanged-container identity, four held readers/overflow exact fallback, byte/structural overflow, stale label rejection, final-drop release, real retention/expired-token Gone, and actual async cancellation. The unchanged Python oracle grades candidate pagination and rejects omitted/duplicate answer-row injections. Optional-index fallback and all full integration schedules still need candidate routing before timing admission; baseline-only integration checks are not candidate evidence.