Historical report imported 2026-10-10 from docs/experiments/benchmarks/dev-small-labs-run-02.md. Original SHA-256: f7220032f136309baa29a59fa1ec732eb461536b128c17b810ca282d1b6ed5c0. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Development/small lab screen, run 02

Execution record for the registered screen (local checkout reference docs/experiments/benchmarks/dev-small-labs-screen-protocol.md; not present at origin/main) and campaign plan (local checkout reference docs/experiments/benchmarks/dev-small-lab-plan.md; not present at origin/main). Status: eight native cells completed; C1-6 deferred at admission. This is a local diagnostic screen, not deployment qualification or a release.

Scope and provenance

The user authorized execution of the reviewed initial M0/C1/Q1 wave. Capacity, query/availability and recovery/evidence responsibilities used separate artifact directories. A capacity PI and two bounded Luna assistant tasks prepared the fixtures and checks; the inline coordinator reviewed them and serialized every build, validator and workload. See the dispatch ledger (local evidence omitted from stage). Actual agent token telemetry was unavailable; dispatch allowances are estimates, not measured savings.

The protocol was committed separately as e18897f8b3acbcd05d3afbc68c8ee875d9e5135a; independent checks as 463ff12, and private fixtures as e7af410. The existing dirty production working tree was preserved. Each job records its source snapshot/diff, protocol hashes, command and environment: the commit alone does not identify the measured implementation. Frozen release binary hashes are in binaries.json (local evidence omitted from stage). The planning document gained a results/status pointer only after measurements; the original pre-run plan bytes and hash remain in every timed job's receipt.

All commands use python3 tools/resource_group.py -- and the private tools/bench/labs/dev_small/run_job.py coordinator. Owned scratch and build caches use $FABRIC_DATA; containment is 16 GiB memory high, 20 GiB maximum, no swap and a 30-minute outer deadline. The coordinator additionally enforces the registered per-stage time allocations and 4 GiB scratch ceiling.

Preparation and counterexamples

The first fixture compilation failed with Cargo exit 101 because an OTLP KeyValue initializer lacked key_strindex. The failure receipt and source were preserved; adding Default for remaining fields repaired the private helper. The subsequent offline, locked release build exited 0.

Full fixture/control preparation exited 0. It checked H0, H256 and near-rotation fixtures by independently decoding recovered Batches and comparing body and Batch hash ledgers. Clock discontinuity, suspend gap, missing phase, custody, body-hash, rate and query-verdict controls were rejected as intended. Lossless archive round trips and 36/36/0 ordinary/jittered/off selector counts were checked.

Before C1-5, pre-run review found that fractional offers had no rows on the old normal/recovery probe ticks. The separately registered addendum (local checkout reference docs/experiments/benchmarks/dev-small-labs-screen-r2.md; not present at origin/main), commit f7ebc3e, preserves the 12-target counterexample and changes selection to the first actual write in each bucket. The private correction (1330077) passed controls with 36 targets; high-rate selections and Q1 jitter retain their behavior. No prior result was replaced.

Audit review also found a reporting-key typo: earlier audit summaries displayed a null replay hash, although the canonical query-provenance.json contained recovered_sha256 and the audit checked its presence/format. Commit c874293 copies that checked value into the report, without changing the acceptance predicate. Its positive/control fixture exited 0 before Q1. Earlier audit files remain unchanged; use their canonical provenance files for the actual hash.

Seed input and Batch hashes are deterministic. Production receive timestamps and therefore physical Segment hashes vary; those actual values are retained. H256 adds one old historical identity beyond the live Spindles. Prefix preparation and post-run replay/oracle costs are excluded from timed server measurements but count toward enclosing cgroup resources and the machine-work budget.

Results

Completed outcomes and exact command exits are in the append-only queue (local evidence omitted from stage). The coordinator's per-job receipts distinguish command exit from scientific acceptance.

The five completed C1 cells have command exit 0, every applicable ordinary guard true, and independent audit exit 0. All historical and live log bodies, Batch hashes and sequences matched; no retries occurred.

Cell	Condition	Exact recovered logs	Server peak HWM, MiB	Timed server CPU, s
C1-1	Fresh, 20 nodes, 1,000/3,000/1,000 logs/s, queries off	300,000	68.50	10.50
C1-2	H256, same offer, Scan	592,390	489.59	95.98
C1-3	Fresh, same offer, Scan	300,000	481.40	63.10
C1-4	H256, same offer, queries off	592,390	69.22	10.59
C1-5	H256, 20 nodes, 100/300/100 logs/s, Scan	322,390	248.06	52.85

H256 contributes 292,390 of the recovered logs. This table uses process HWM and whole sampled CPU for accessible context; the registered nomination uses balanced phase CPU and sampled phase peak RSS, reported in the consolidated results.

C1-6 was not run. After C1-5, about 242 seconds of C1's allocation remained, below the observed seeded-cell estimate of 273 seconds. The memory lab also had less than its private runner's 10 MiB evidence reserve. See the admission record (local evidence omitted from stage). No shortened trial or failed workload is substituted for that missing cell.

Registered hypothesis decisions

The consolidation (local evidence omitted from stage) derives phase RSS from raw samples and retains audited phase CPU, individual query shapes, Spool rates, IO, clocks, cgroup counters, queues and transition sample brackets. The table below applies the preregistered 15% diagnostic nomination rule. It is not a confidence interval or a statistical rejection of a null hypothesis.

Contrast	Balanced phase CPU	Sampled phase peak RSS	Screen decision
Queries on/off, fresh	5.71x	7.76x	Nominate query work
Queries on/off, H256	8.70x	7.82x	Nominate query work
H256/fresh, queries off	+0.72%	+1.81%	History alone below the practical threshold
H256/fresh, queries on	+53.37%	+2.64%	Nominate historical query CPU; no memory nomination
Lower offer with H256/queries	-46.15%	-51.65%	Nominate rate/accumulated-live-volume sensitivity

The source offers and ordinary query populations were intact: 20 requests per query kind per phase. Visibility polling counts vary, so this is a comparison of the registered consumer behavior, not a matched count of visibility requests. All H256 logical prefix hashes match. Lower offer over a fixed duration also means fewer live rows and a smaller active journal; it does not isolate instantaneous rate from accumulated volume. No three-pair confirmation ran.

Availability and first publication

Cell	Live / historical logs	Source-write-to-visible p99/max, ms	ACK-observed-to-visible p99/max, ms	Final Segments
M0, aligned probes	3,000 / 0	273.34	254.46	0
Q1 fresh, jittered probes	3,000 / 0	1,055.03	266.99	0
Q1 near rotation, jittered probes	3,000 / 72,954	1,281.97	385.98	1

All three commands and audits exited 0, all ordinary guards were true, and each cell returned all 36 targets. With 36 samples the reported empirical p99 is the maximum; these are not population-tail estimates. The difference between aligned M0 and jittered Q1 demonstrates the importance of source/collection timer alignment in interpreting this small probe set. It is not a repeatable speed comparison.

Near-rotation input left a 66,986,960-byte active journal below 64 MiB. A sealed file first appeared in the sample ending 11.03 seconds after offers began; the first published Segment appeared in the sample ending 13.04 seconds. Two requests intersect the roughly one-second publication bracket. Those observations do not identify exact build boundaries or prove a request overlapped a running builder. Live answer fidelity, selected visibility and quiescent independent exact answers passed. Query access did not wait for all data to become Segments.

The near-rotation server reached 338.04 MiB HWM (337.89 MiB sampled phase peak) despite the development offer of 10/30/10 logs/s. That nominates pending-journal occupancy as a useful axis independently of current offer rate. It is not an isolated proof: seeded content, query work, startup and the first build differ from the fresh condition.

CPU, acceptance and resource observations

At the high small offer, successful Spool cycles were approximately 20 batches/s through all phases; payloads grew with the burst. Every observed send attempt was ACKed, with no retries. This is not a Spool append acceptance percentage: the append-attempt denominator is unavailable. Collection-to-ACK-stdout-observation p99 was 48.25–49.00 ms across C1-1..4. Source-write latency includes the collection cycle and must not be replaced with that number.

Query-off phase CPU returned to approximately 0.037 core in recovery after about 0.103 core during the burst. Fresh Scan rose from 0.186 to 0.384 to 0.437 core across normal/burst/recovery; H256 Scan used 0.357, 0.551 and 0.636 core. Growing query work is a stronger explanation for that pattern than instantaneous ingestion rate alone. Server IO user/system counters and physical/logical bytes are preserved per phase rather than inferred from event counts.

The largest completed native job cgroup peak was 4,014,977,024 bytes (3.74 GiB), including producers, observers, file cache and replay/verification. It occurred in C1-4, whose server HWM was only 69.22 MiB. This illustrates why whole-job memory must not be attributed to the server or sealer. Native node peaks stayed below 6.48 MiB. Exact per-job cgroup counters, swap, pressure and disk peaks are retained in coordinator receipts and the consolidation. All eight native cgroups recorded zero memory-high/max/OOM events and zero swap. The largest sampled owned scratch footprint was 1,553,890,824 bytes (1.45 GiB), below the 4 GiB trial bound.

Interpretation and next experiments

The strongest diagnostic lead is Scan query materialization, with a separate CPU effect from accumulated history. Source inspection supports the mechanism: History::sources gathers uncovered journal Groups and the Scan path extracts query rows before filtering; concurrent consumers can repeat that work. Full- history searches also read historical row groups. This is a source-supported hypothesis, not a profiler-derived allocation attribution.

Prioritize the already planned Q2a Scan/Walk comparison with identical inputs, exact answers and explicit pending-journal occupancy. Then confirm the nominated effect with separately budgeted alternating pairs and attribute allocations, query concurrency, cache/allocator retention and row projection. Keep builder heap acceptance C2 separate: these RSS results neither repeat nor replace the earlier 97% incremental-builder-heap result.

General development and small log/host-metric use passed these finite local screens. Before recommending an application-pipeline release, the campaign still needs mixed-signal R2a, deployed memory limits C4, recovery R1, retention/query correctness Q2b, full builder C2 and lifecycle C3. C1-6 remains an explicit gap. Installed-profile and remote forwarding evidence were not produced here.

Verification and cleanup

cargo xtask checks --profile fast, run through the resource launcher and lab coordinator, exited 0: all seventeen checks passed, including formatting, Clippy, workspace tests, layer/purity checks, independent oracles, properties and network simulation. It consumed 147.62 seconds. After measurements, the private seed helper's parity expression was changed to is_multiple_of(2) for the lint convention; measured release binaries remained frozen. The measured helper source is retained in each cell's source archive.

Consolidation and retained-archive verification each exited 0. Every retained cell manifest and shared corpus hash matched; a representative changed hash was rejected. Lossless source/clock reconstruction had already run before each cell's successful scratch deletion. See the archive verification receipt (local evidence omitted from stage).

Successful native and verification scratch was removed by the coordinator and launcher. Final cleanup archives launcher/fast-check receipts and removes only the reviewed empty preparation-failure directory; the compiler diagnostic, original source and repaired outcome remain. Shared build caches are preserved. Documentation and final cleanup command outcomes are recorded in the queue and the cleanup review (local evidence omitted from stage).

The manual Bun documentation check and git diff --check exited 0. Automatic Bun documentation CI/hooks remain disabled. Final cleanup exited 0: the reviewed compiler-failure directory contained zero file bytes, its empty directory tree was removed, and all completed transient units were inactive. The data drive had 473,179,123,712 bytes available at the final inspection.

Machine work consumed about 39.7 minutes of the 60-minute aggregate allocation: M0 206.92 seconds, C1 1,558.47 seconds, Q1 426.90 seconds, plus about 3.2 minutes of builds, controls, checks and cleanup. C1-6 was deferred under its stage allocation and evidence reserve even though aggregate budget remained; allocations were not silently transferred. Compact evidence retained about 44.0 MiB in memory, 8.0 MiB in query and 0.02 MiB in recovery; coordination evidence stayed below 25 MiB. Each lab stayed below 50 MiB. Delegated task allowances totaled an estimated 21,500 tokens; actual agent/coordinator token usage was unavailable, so no measured efficiency claim is made.

Interpretation limits

One cell per condition can nominate a mechanism for confirmation; it cannot establish statistical significance or a sustainable capacity ceiling. Query-on includes serial visibility polling as well as the ordinary HTTP consumer. Server RSS/HWM, allocator heap and enclosing cgroup memory are different metrics; allocator heap is not instrumented in these native binaries. The cgroup includes producers, observers, file cache and post-run verification.

Visibility is a sampled polling upper bound. Ordinary probes are phase-aligned; Q1 uses source-target jitter. ACK timestamps are stdout observations, not an instrumented server durability timestamp. Quiescent queries have an independent complete-answer oracle; live returned rows are checked for fidelity, without claiming complete concurrent snapshots from that check alone. There is no complete Spool append-attempt denominator, so a Spool acceptance percentage is unavailable.

No remote host, installed deployment, trace-load, destructive fault, retention eviction, full builder acceptance, confirmation or long-soak result follows from this screen. Those remain separately budgeted campaign work.