Historical report imported 2026-10-10 from docs/experiments/benchmarks/ingestion-memory-run-01.md. Original SHA-256: 813ab2f881458e0359b4ddf0937aa46e274c0e0e851b03ce2e7e8078590cb98f. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Ingestion memory and pending-journal query run 01

Executed locally on 2026-10-04 on milestone/streaming-segment-output, with candidate source captured in each population's source-snapshot.tar.gz and working-tree.diff. The memory protocol (local checkout reference docs/experiments/benchmarks/ingestion-memory-protocol.md; not present at origin/main) and pending-query protocol (local checkout reference docs/experiments/benchmarks/pending-journal-query-protocol.md; not present at origin/main) were registered before their respective measurements. This is a finite implementation screen, not completion of the bounded-sealer milestone or target-host qualification.

Mechanism and measurement

The production worker calls segment::build_sealed (local checkout reference crates/fabric-server/src/segment/bounded.rs; not present at origin/main) instead of retaining the complete journal and all projected rows. One frame feeds byte-capped sorted runs for logs, metrics and spans. Merges have at most sixteen inputs and multiple passes. Arrow chunks are flushed explicitly; raw Batches remain exact. Publication, sync and checkpoint/reclaim order are unchanged. A failed build removes its private directory and keeps the journal.

The reference is the retained whole-file read_sealed plus build. Trials use fresh processes and three alternating pairs at each size: seed 42, one node, shuffled timestamps, 128 logs per Batch, 1,024-byte bodies and empty attributes. Nominal sizes refer to body volume; actual journal sizes below include encoding. Fixture creation and answer grading are outside build timing. Incremental Rust heap is the peak above the retained-fixture baseline. Process RSS includes fixture generation and allocator retention and is a separate metric. The probe's global allocator overlaps queries in the pending population, so those allocation values are not reported as builder-only memory.

Memory results

All eighteen measured trials exited 0, preserved raw Batch custody and exact source rows in order, and reported zero spill leftovers. Values are medians of three repetitions; heap was identical across repetitions of each cell.

Actual journal	Reference heap	Candidate heap	Heap reduction	Reference wall	Candidate wall	Wall increase
16.29 MiB	80.79 MiB	29.49 MiB	63.5%	339.53 ms	377.13 ms	11.1%
65.17 MiB	300.82 MiB	30.79 MiB	89.8%	1,347.58 ms	1,450.97 ms	7.7%
260.69 MiB	1,185.29 MiB	32.06 MiB	97.3%	5,306.62 ms	6,633.34 ms	25.0%

Candidate heap increased 4.15% from the middle to largest fixture, below this screen's 80 MiB ceiling. The registered 50% reduction at the middle size was met. This does not establish BS-3 on its different four-shape acceptance suite. Median CPU time was 302.29/335.98, 1,253.98/1,368.37 and 5,039.96/6,319.64 ms (reference/candidate). Whole-process median peak RSS was 153.83/113.27, 558.38/354.50 and 2,183.34/1,145.50 MiB.

The candidate writes more row groups when byte caps close a chunk before 8,192 rows. File and filter hashes therefore differ on these fixtures; exact rows, raw bytes and schemas are preserved. The differential unit fixture separately compares every table/filter byte where byte caps do not change boundaries. Additional disk traffic and CPU are the cost of reducing retained heap; raw per-process I/O counters are in each measurement, and are not physical-device traffic measurements. Spill-only byte attribution is not separately measured.

Query access while sealing

Six measured trials exited 0 on a 65,536-log fixture with 511 Batches in a sealed journal and the final Batch still active. Both Scan and Walk were queried at a fixed snapshot before, during and after building the sealed Segment. Broad 20-row pages and a selective one-row marker in the active journal were graded against the unchanged Python oracle's expected rows, shape and envelope checks.

All 171 sampled first pages matched exact content, order, continuation presence and completeness. Each trial had 20 or 21 during-build samples. Changed, missing and duplicated row controls were rejected. This is first-page grading; full pagination remains the responsibility of the unchanged HTTP history integration tests.

Phase	Plan	Samples	Median engine wall	Maximum engine wall
Before	Scan	12	90.01 ms	122.86 ms
Before	Walk	12	245.65 ms	453.49 ms
During	Scan	63	71.28 ms	121.07 ms
During	Walk	60	56.00 ms	80.00 ms
After	Scan	12	60.39 ms	68.09 ms
After	Walk	12	26.50 ms	58.39 ms

These aggregate timing cells mix reference/candidate and query shapes; they show observed response costs, not an isolated acceleration claim. Before samples include cold initialization. They are engine timings, not HTTP p99 or Spool-ACK-to-first-queryable latency. The owner's 30-second headroom means data may take time to become queryable; it does not request an intentional wait. No timer, delay or product freshness guarantee was added. Existing queries can already read committed active and uncovered sealed journals.

Reproduction and retained evidence

Commands ran through python3 tools/resource_group.py --, with verified cgroup memory.max=21474836480, memory.swap.max=0, data-drive builds and scratch.

python3 tools/resource_group.py -- cargo build --offline --locked -p fabric-server --example responsibility_probe --features responsibility-alloc-probe --release
python3 tools/resource_group.py -- python3 -B tools/bench/run_ingestion_memory.py --phase preflight --destination docs/experiments/benchmarks/data/ingestion-memory-run-01/preflight-corrected
python3 tools/resource_group.py -- python3 -B tools/bench/run_ingestion_memory.py --phase memory --destination docs/experiments/benchmarks/data/ingestion-memory-run-01/memory
python3 tools/resource_group.py -- python3 -B tools/bench/run_ingestion_memory.py --phase pending --destination docs/experiments/benchmarks/data/ingestion-memory-run-01/pending

Each command above exited 0. Memory service runtime was 4 min 34.8 s with 3.5 GiB cgroup peak; pending runtime was 1 min 3.0 s with 1.2 GiB peak; swap was zero. The cgroup includes fixture generation and Python grading. This resource cap is an experiment safeguard, not a proof of a 20 GiB production requirement.

Memory summary (local evidence omitted from stage) and pending summary (local evidence omitted from stage) retain raw measurements, query verdicts, binary SHA-256, commands, source snapshots and working-tree diffs. Per-trial cleanup receipts record deleted logical scratch bytes. Source fixtures are gzip-archived by hash on the mounted data drive at $FABRIC_DATA/evidence/ingestion-memory-fixtures; paths/hashes are in trial summaries. They are intentionally retained evidence, not temporary files. Build caches remain on that drive.

Thirty completed preflight/measured trial cleanup receipts all reported owned scratch removed: 12,512,145,815 logical bytes (11.65 GiB cumulative). This is cumulative cleanup, not peak disk usage. Failed preflight scratch was retained separately as evidence.

The first preflight (local evidence omitted from stage) failed because the full-page-chain oracle was incorrectly applied to a sampled first page. Its exit/failure and owned scratch were retained, then the grading correction was registered before measured trials. The corrected preflight (local evidence omitted from stage) ran four trials with negative controls and exited 0. The initial Arrow writer test also found double-finalization; a deterministic empty-table regression now preserves that failure. A corrupt-frame regression checks cleanup after valid rows have already spilled, and a 16 KiB-row test checks byte admission.

Limits and next evidence

The four-shape bounded-sealer suite, injected write/full-disk failures, every crash stage, registered sealer mutants, unchanged soak and remote forwarding were not run here. No installed defaults changed. Per-node freshness, filter and Parquet metadata grow; frame decoding can expand; multiple workers multiply buffer ownership. Scan's decoded journal tail and Walk's tail index remain separate backlog memory risks. The next consumer measurement needs native Spool ACK timestamps and first exact query visibility under sustained arrivals, with the proposed 30-second allowance measured explicitly.

The earlier workstation OOM occurred in query_spec during semantic-mutant verification, according to the kernel log; it was not a measured ingestion trial. Source inspection identifies non-progressing pagination as a plausible cause, but the exact mutant allocation was not reproduced in this screen. That continuation remains interrupted. Whole-workload cgroup containment now covers verification as well as benchmark subprocesses.

Candidate verification

The final python3 tools/resource_group.py -- cargo xtask checks --profile fast exited 0: formatting, compilation, workspace tests with all features, Clippy, layers, core purity, harness/oracle controls, properties, fuzz corpus and simulation. Runtime was 2 min 33.8 s; cgroup peak 705.1 MiB, swap zero. Preserved receipts and checked sources (local evidence omitted from stage) record this working-tree candidate. Earlier attempts exited 1 for a regression test import and lint findings; they were corrected without weakening checks.

Manual documentation check python3 tools/resource_group.py -- /tmp/fabric-bun-runtime/bun-linux-x64/bun tools/docs/check.mjs exited 0. Containment tests (python3 -B tools/test_resource_group.py) and hook contract tests (python3 -B tools/docs/test_hooks.py, retained Bun on PATH), each through the cgroup launcher, exited 0 with six and ten tests. The automatic Bun documentation CI and Stop hook remain disabled as requested.