Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/coupled-labs-operations.md Original SHA-256: 76702842f519029d226c1c098516a52650fb6761d2438e34da7f3430ad988b7a Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Coupled labs: recovery and operations packet

Current disposition: see completion preparation and executed run. The preparation status below is historical.

Status: prepared; all experiments not run. Accountable: operations PI; coordinator owns workload admission. Common plan applies. Own this packet and future coupled-operations-* findings/fixtures; coordinate shared source edits with root. No oracle or product-contract edits in preparation.

O6 — projection availability and answer envelopes

Question: what may remain known when a query projection is missing but raw custody/Manifest metadata remain? H1: an explicit source-availability model reconciles rows and metadata without hiding unavailable data. H0: proposed semantics remain inconsistent or alter an existing promise.

Start from integration findings, completion_storage.rs (local checkout reference crates/fabric-server/tests/completion_storage.rs; not present at origin/main) (missing_query_tables_are_incomplete_with_independently_declared_unavailability), the query oracle, product contract and retained-history view. Known result: zero expected/answered rows still disagree on retained window, freshness and collection gaps. Do not redefine that as a performance failure.

Prepare a truth table for raw/Manifest/log/metric/span projection availability: all intact; one query table missing; optional filter absent/corrupt; raw absent; Manifest absent. Compare Scan/Walk and first/full pagination. State which facts have independent surviving evidence, which become unknown and which force incompleteness. Keep collection gaps distinct from storage availability.

Controls: falsely complete answer, erased retained metadata, invented freshness, dropped/duplicated row, expired continuation. Each expected rejection must derive from the accepted contract; disputed cells stay unresolved. Deliver a minimal counterexample and alternatives (implementation fix, fixture correction, model extension). Any specification/oracle change requires explicit scope and separate registration before implementation. Success is a supported decision, not a green test obtained by removing metadata assertions.

Existing reproduction command, only after admission, inside the common wrappers: cargo test --offline --locked -p fabric-server --test completion_storage --all-features missing_query_tables_are_incomplete_with_independently_declared_unavailability -- --nocapture. Full fast follows a scoped correction; historical failures remain.

O7 — logical exactness versus physical pruning equivalence

Joint reviewer: capacity PI. Start with C2 in completion results and completion_builder.rs (local checkout reference crates/fabric-server/examples/completion_builder.rs; not present at origin/main). Byte-capped row groups read 9,156 versus 20,350 reference rows on the recorded large-row window test. H1: changed physical grouping explains the mismatch while logical answers remain exact; H0: missing coverage, invalid bounds or an ordering defect explains it. This explanation is not permission to relax the existing gate.

Prepare row-to-group and window-to-visited-group ledgers for the minimized case, including equal timestamps, late rows and a large row spanning a cap decision. Check raw bytes, row order, complete answers and conservative bounds independently. Controls must detect an omitted matching group and a falsely narrowed bound; extra reads must be observable separately from wrong answers. Return whether physical identity is an intentional acceptance obligation or needs a separately scoped decision. Existing C2 and dependent qualification remain failed/blocked until that decision and fresh evidence exist.

O8 — one prepared Batch of collection overlap

Joint reviewer: query PI for visibility. Deferred behind resource admission, O6/O7 decisions relevant to the service fixture and ownership design. Source: runtime.rs, Spool and sender tests. H1: preparing Batch N+1 during ACK wait for N lowers CPU-neutral cycle time or raises accepted logical throughput. H0: contention/coordination cancels the gain. Baseline serial path; candidate exactly one prepared, durably spooled next Batch and one sent unacknowledged Batch per Strand. No widened delivery window.

Prepare an ownership/state-transition table for cursors, sequence allocation, Spool capacity, ACK loss/retry, collection errors, graceful stop and cancellation. No worker queues without byte limits. Propose dev 1-node 10/30 logs/s and small 20-node 1,000/3,000 aggregate logs/s from existing profiles, with quiet/normal/burst phases; freeze durations/mixed-signal cadence before registration. Separately measure offered-load latency and capacity: fixed offers cannot prove a throughput increase when both already accept all input. Preserve exact raw retries and order. No crash/fault injection beyond separately admitted scope. Primary capacity or latency metric must be selected before running; no choosing the winner afterward.