Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/catalog-labs-operations.md Original SHA-256: 5bf395d35f9a1cf37c41ecea726f28453a84df2a5dcb6139907aa12b2bdfdf9e Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Status: planning packet; not registered, implemented or executed. Storage operations owns coverage, publication/reclaim cuts and source lifetime; query/capacity labs own scheduling and performance comparisons. This packet changes no product contract, oracle, durability order, default or qualification. Grounding: product contract, history view, ADR-0024, HIST-1–9, completion run and source enumeration.

Proposed object: an immutable per-query catalog of committed source coverage, provenance and checked pruning evidence. It must include active and sealed pending journals as well as published Segments; publication is not needed for visibility. Compare current per-query discovery against an immutable catalog snapshot and a locked mutable catalog whose handles copy ownership before releasing the lock. Reject holding the catalog lock through decoding/IO; it couples intake progress to consumer latency. Do not select a production mechanism from this packet.

The logical snapshot remains the existing oldest/newest group range and row key. An epoch may identify internal catalog generations; it must not redefine page tokens or wire format. Distinguish exact coverage identity from optional bounds. At every cut, each admitted logical group is selected once, even when journal and published Segment bytes coexist. A failed read is unknown/unavailable or an explicit error under current semantics, never proof that a source is empty.

Freeze seed 0xCA7A10A1: three node identities, eight groups each, logs/metrics/ spans, deliberately late observation times and equal-time rows across nodes. Prebuild expected Batch bytes, receive times, gaps and ordered rows independently. Enumerate these cuts deterministically with barriers, not timing sleeps: active append committed; journal rotated; build paused; durable publication; checkpoint before reclaim; reclaim completed; logical retention advanced. Use fresh/reused History instances, both plans, full pagination and restart.

O5 distinguishes malformed/untrusted metadata from semantically lying metadata. Absent/corrupt optional filters must fall back exactly; absent or unreadable raw tables must expose incompleteness/error. Authentication alone cannot prove bounds or filter contents conservative: include a matching row outside a lying bound and a needle rejected by a forged filter. Never silently accept those as empty. Verify late timestamps cannot be pruned by receive-time/append-order assumptions. Equal-time ordering is (time, node_id, sequence, index) across source boundaries; test page limits 1/2/3 and stop-rule equality with full chains.

First introduce private checked newtypes GroupRange, SourceId and SnapshotId in a lab fixture, then an enum for journal/Segment representations. Constructors reject reversed/overflowing ranges; require explicit provenance for bounds. An optional acceleration summary has a distinct absent/untrusted/checked state. Model read outcome as rows or a typed failure, avoiding unwrap_or_default().

Next compare borrowing SourceHandle<'snapshot> from an owned snapshot against cloning Arc<SourceDescriptor> plus owned reader/file handles. Lifetimes prevent use after Rust ownership ends; Arc retains allocations. Neither alone prevents filesystem unlink, validates coverage or grants logical retention entitlement. Propose one bounded in-flight query guard and explicit release/drop accounting; avoid speculative caches or a durable lease that changes reclaim semantics. Any deferred physical unlink proposal needs separate durability/retention scope.

Memory lifetime and logical retention are separate axes: an old descriptor may remain allocated for a bounded admitted read while a subsequent continuation must still return Gone. Register the current in-flight retention/error behavior before comparison; do not invent a successful-read guarantee during eviction. Compile-fail borrowing examples complement runtime cut tests; types cannot prove metadata truth, IO availability or correctness of catalog publication order.

frontier-storage-01 retained a missing-log-table counterexample: its oracle declaration excluded whole records, while surviving manifests/raw Batches/gaps still supplied metadata. Oracle input cannot model table-specific availability. Keep that failure unchanged. Design a separate trust-boundary proposal with independent per-source custody, per-table availability and surviving metadata evidence. Specify expected rows, completeness, retained bounds, freshness and gaps separately; controls erase or alter metadata, fabricate rows, and falsely claim complete. No oracle edits or new acceptance rules are authorized by this planning packet.

Before execution: register fixture ledger/hash, cuts, baseline/candidate diff, checker controls and exact success/error rules. Run controls before candidates. Correctness metrics: missing/duplicate identities, ordering violations, snapshot coverage holes, unknown sources hidden, and Gone outcomes. Resource metrics: catalog/handle bytes, retained generations, open descriptors, drop/release counts, publication lock hold/wait time, cgroup peak/events and sampled scratch bytes. Timing remains descriptive until matched populations and thresholds are frozen.

Existing anchors below are runnable tests, not implementations of O1–O5:

python3 tools/resource_group.py -- cargo test --offline --locked -p fabric-server --test history crash_states_of_sealing_never_serve_a_record_twice -- --exact --test-threads=1
python3 tools/resource_group.py -- cargo test --offline --locked -p fabric-server --test history text_filters_skip_only_groups_without_the_needle_and_fall_back_when_corrupt -- --exact --test-threads=1
python3 tools/resource_group.py -- cargo test --offline --locked -p fabric-server --test completion_storage age_eviction_grades_the_predeclared_retained_ledger_and_expires_pages -- --exact --test-threads=1

All builds/tests/validators require the mounted data drive and resource launcher: 16/20 GiB high/max, zero swap, 1800-second default. Proposed first fixture budget: 900 seconds including grading, 1 GiB owned scratch, 16 GiB free reserve and 16 MiB compact evidence. No workload here. Stop on failed control, custody/query mismatch, unknown evidence hidden, OOM, deadline, storage limit or cleanup survivors; preserve seed/cut/input/output/exit receipts before cleanup. Missing environment means not run. Broader qualification, protocol changes and physical power loss remain out of scope. Root owns serial execution and final admission.