Historical report imported 2026-10-10 from docs/experiments/benchmarks/readiness-labs-run-01.md. Original SHA-256: b4eda1a25ae3bb6921c8a01b153625e0d2c4a8d7030223ef9090bcbfb9209448. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Coordinated development/small investigation, run 01

First wave of the registered plan (local checkout reference docs/experiments/benchmarks/dev-small-readiness-plan.md; not present at origin/main), executed 2026-10-04/05 on milestone/streaming-segment-output, base fbfef819785c5b4f36202281018f45c3bd3cd20d with preserved working changes. Three GPT-6.1-SOL medium workers prepared separate memory, query and recovery labs; the inline coordinator reviewed their evidence and dispatched every workload sequentially. No production algorithm, default, oracle, wire format or ACK/sync ordering changed in this wave.

Outcome: the memory advantage survives the reconstructed 64 MiB shapes, and nine selected custody/query recovery tests passed. The diagnostic shape screen failed because its large-row checker conflated logical rows with physical index groups. Full registered acceptance remains unrun. Retained-file investigation supports a checker error. The controlled query comparison is inconclusive after a 380-second wall/monotonic clock discontinuity in Scan; Walk and confirmation were stopped. No release or deployment qualification follows from this wave.

Memory and physical layout

Lab results (local evidence omitted from stage), derived measurements (local evidence omitted from stage) and frozen protocol (local checkout reference tools/bench/labs/readiness/memory/protocol.md; not present at origin/main) record one reference/bounded pair per reconstructed shape, seed 2703204353, 100 identities, logs, gauge metrics, spans and collection gaps. Each journal reaches 64 MiB of encoded groups, with at most one-group overshoot. These are diagnostic fixtures, not the original registered workload-shape acceptance.

Shape	Reference / bounded incremental heap, MiB	Reference / bounded build wall, s	Reference / bounded build CPU, s	Original comparison
Steady	377.79 / 43.72	1.444 / 1.494	1.37 / 1.42	All applicable gates true
Delayed observations	377.80 / 43.72	1.455 / 1.591	1.37 / 1.50	All applicable gates true
Disordered observations	379.06 / 43.74	1.568 / 1.631	1.46 / 1.50	All applicable gates true
16 KiB log bodies	359.17 / 32.96	1.089 / 1.009	1.01 / 0.93	Manifest comparison failed

All four pairs preserved ordered log, metric, span, gap and raw Batch ledgers; the bounded builder stayed below 80 MiB with no build/run leftovers. These measurements show 88.43–90.82% lower incremental heap on these fixtures. They neither replace nor repeat the earlier 97% result at 256 MiB. Scaling was stopped after the failed shape comparison, so its 10% growth gate is unrun; scale: true in 64 MiB pair files is an inactive baseline check.

Whole-process bounded RSS was 94.45–126.58 MiB, including fixture generation and readback. Build CPU comes from process tick deltas at 100 ticks/s; those boundaries include small sensor reads. The counted allocator instruments both builders. One pair is insufficient to establish a speed improvement. Steady build syscall byte counts grew from 67.19 to 144.60 million read characters and 51.94 to 129.38 million written characters. read_bytes was zero: cached IO accounting does not establish device throughput or scratch amplification.

The failed field was manifest.files["text_filter.bin"].rows: 1 versus 9. It counts physical row-group filters, while logs.parquet.rows remained 4,096. The registered diagnostic permitted different large-row grouping but its checker still required this physical count to match. The minimized counterexample (local evidence omitted from stage) and separate forensic plan (local checkout reference tools/bench/labs/readiness/memory/forensic-plan.md; not present at origin/main) preserve that mismatch without changing the original verdict.

The retained-file validation (local evidence omitted from stage) authenticated both original manifests/files, matched each Parquet group to its filter and checked all 67,100,672 body trigrams in each layout. No actual trigram was rejected; missing-filter and cleared-filter controls were rejected. This supports physical-layout consistency on this fixture, not full query or spans-index acceptance. M2 remains failed; M3 exited 0.

Ingestion and query cost

The query protocol (local evidence omitted from stage) fixed 20 native Spindles, 900-byte bodies, 60/60/60-second phases at 1,000/3,000/1,000 source logs/s, one sealer, 64 MiB journals, fresh state and the same binaries. Query-off disables timed HTTP consumers but retains exact recovery and independent post-run query grading.

Query-off phase	Source logs/s	Successful Spool batches/s	Encoded Spool bytes/s	Mean server CPU, cores
Normal	1,000	20	1,169,872	0.0400
Burst	3,000	20	3,493,638	0.1104
Recovery	1,000	20	1,177,031	0.0401

Query-off (local evidence omitted from stage) exited 0 with 300,000 exact logs, 3,641 recovered/ACKed batches, no retries or gaps, and collection-to-ACK-observation p99 43.59 ms. Offered-phase sampled server RSS peaked at 63.41 MiB; phase-balanced CPU was 0.0635 cores. Its process HWM was 67.38 MiB, demonstrating sampling's limit. Peak pending batches were seven, sampled unread file backlog zero, and final queues drained. An append-attempt denominator is unavailable, so batches/s is successful acceptance throughput, not a percentage of attempted appends. Timed query latency and visibility are explicitly unmeasured in this control.

Query-off timed cgroup peak was 972,050,432 bytes, including generator, Spindles, server, file cache and kernel charges. Launcher peak was 2,384,158,720 bytes and includes post-run oracle/replay work. Neither is server RSS or live builder heap.

Scan (local evidence omitted from stage) also recovered all 300,000 logs exactly, with no retries and exact live/final answers. However, wall-minus-monotonic offset changed by 380,217 ms; coordinator monotonic runtime was 276.62 s while launcher wall runtime was 657.17 s. This is consistent with host suspension or a clock adjustment; the cause is unverified. Clock, producer-lag and sampled-visibility gates failed. A recovery phase had no wall-window CPU samples, and the lab annotation then raised KeyError('processes'). Its original source, raw artifacts, failed summary and exception are retained. Scan's 483.25 MiB sampled whole-run RSS is an observed value from an invalid timing trial, not a valid paired performance result.

Walk and the three-pair confirmation were deferred under the plan's stop rule. No Scan/Walk gain, CPU attribution percentage or general freshness percentile is claimed. The fixed five-second visibility tags are phase-locked to collection even on an otherwise valid run.

The post-run annotation repair passed its real missing-phase regression control (local evidence omitted from stage). It produces null metrics with reasons while preserving failed gates. The first independent accounting audit also found a precision mismatch: dividing a large integer clock offset by floating 1e6 versus integer 1000000 changed its rounded range. That failed audit remains archived. The rounding counterexample (local evidence omitted from stage) and reviewed audit (local evidence omitted from stage) use an integer range and a representation tolerance derived from the offset's floating precision. A deliberately changed 1 ms range is rejected; the 5 ms stability gate is unchanged. Revised accounting validation exited 0 for both archives, while Scan's original failed and performance-ineligible verdicts remain intact.

Recovery and deployment

Recovery results (local evidence omitted from stage) record nine exactly selected tests, all exit 0: corrupt input after spilling, reported Spool sync errors, interrupted append callbacks, full-Spool/source coverage, checkpoint failure and exact retry, synthetic restart states, journal/Segment answer equivalence, independent query grading, and sender/server restart. Zero-selected and ignored-selected test controls were rejected. The largest of these service peaks was 1,340,391,424 bytes, with no swap/OOM and successful owned scratch cleanup.

This supports the existing tested boundaries only. Actual read/write/ENOSPC failures at new spill stages, staged process kills around publication, and physical power loss remain untested. Successful recovery tests removed their per-query oracle scratch; retained cargo assertions cannot retrospectively regrade each individual answer. The visible Fedora environment lacks a supported disposable installation host and verified Docker containment. Installation remains unavailable, not passed.

Synthesis and next queue

Observed: bounded sorting removes most builder allocation without a large wall-time penalty on these particular 64 MiB fixtures. Physical row-group geometry changes index count and size; logical equality and physical consistency need separate assertions. Bursts enlarge Batch payloads while successful Batch cadence stays near 20/s. Query-off CPU returned to its normal level after the burst in this finite run.

Inference to test: query work and retained history may explain much of the previous 485 MiB server RSS and rising recovery CPU. The interrupted Scan comparison cannot establish that attribution. Faster final materialization is only one possible optimization: committed-journal query access already separates availability from completion of a Segment build.

1. Register a memory-checker revision separating logical rows/metadata from physical index cardinality, keeping the existing heap/growth bounds and defect controls. Re-run original shapes/repeats and the 256 MiB comparison.
2. Preserve and reject the clock-discontinuity fixture; report missing phase measurements as null with a reason. Re-run fresh off/Scan/Walk cells on a stable host, then three alternating pairs only if a candidate meets guards.
3. Vary retained history independently of incoming rate. Seed a declared near-rotation journal, then apply development traffic to exercise its first seal cheaply; this supplements rather than substitutes for a fresh default development lifecycle. Compare query CPU/RSS at matched history volumes.
4. Add private stage-specific IO/space and kill fixtures, followed by the unchanged 5,460-second soak and supported installed-profile acceptance. Long protocols need a separately scoped containment deadline; do not shorten them or infer installed limits from this 20 GiB experiment envelope.

No runtime optimization or default is selected from this first wave. Alpha release readiness is still unestablished while bounded-sealer acceptance, stable query comparison, long lifecycle and target installation gates remain.

Reproduction, resource handling and verification

Every workload and validator uses python3 tools/resource_group.py -- followed by the coordinator job runner (local checkout reference tools/bench/labs/readiness/run_job.py; not present at origin/main). Per-job receipts under coordinator evidence (local evidence omitted from stage) contain exact argument arrays, source snapshots/diffs, pre-run protocol hashes, exits, elapsed time, cgroup measurements and cleanup. Memory high is 16 GiB, maximum 20 GiB, swap zero, outer deadline 30 minutes; jobs also enforce 4 GiB scratch and a 4 GiB free-space reserve. Measurements ran sequentially on $FABRIC_DATA; no remote workload ran.

Earlier coordinator receipts retain total cgroup peak/current, memory events, CPU and IO, but not an anon/file/kernel breakdown. Query raw resource samples retain those charges. Later coordinator receipts include the final breakdown and pressure counters; final values are not component-specific peaks.

The original plan hash is 248aa5cc7b8c3a44bc2bbf584c17d31689e71a4d0591903bf00781d0bb0dee69. Git metadata is read-only, so the required separate protocol commit remains missing. Preserved pre-run snapshots/hashes provide provenance but do not replace that policy requirement. The existing dirty implementation is retained.

The cleanup receipt (local evidence omitted from stage) records removal of 247,244,067 bytes of reviewed memory fixtures and 1,005,279,534 bytes of the interrupted Scan fixture, plus empty/41-byte control fixtures. Compact raw observations, source/seed/hashes and minimized failures remain. The injected exit-7 control was rejected by the coordinator and its marker preserved before cleanup. Original failed receipts remain unchanged.

The full cargo xtask checks --profile fast command exited 1: sixteen checks, including workspace tests, passed; Clippy alone rejected % 500 == 0 in the new private fixture. Changing that expression to is_multiple_of(500) preserves its predicate. Targeted formatting and Clippy receipt runs then exited 0; the measured fixture source remains archived. Six resource-launcher tests also passed. The original failure and targeted repair (local evidence omitted from stage) are retained separately from the first verification commands (local evidence omitted from stage).

The twelve first-wave jobs were followed by verification-only repair/closure; no further timed experiment was added. The initial manual documentation attempt could not launch bun from PATH. The final closure uses the existing local runtime copied into owned data-drive scratch, records its version/hash, runs the manual check and removes that copy. Bun CI and the hook stay disabled. The manual documentation check exited 0 with Bun 1.4.2. Remaining reviewed verification scratch and the copied runtime were removed, as recorded in the closure receipt (local evidence omitted from stage). The campaign used 858.14 seconds of coordinator-measured command runtime (excluding wall-clock discontinuities). The largest whole-service memory peak was 6,394,564,608 bytes during workspace verification, below the 20 GiB cap; all recorded cgroup high/max/OOM events were zero. Compact evidence stayed below 50 MiB per lab. These totals include verification, not only the application.