Historical report imported 2026-10-10 from docs/experiments/benchmarks/sealer-speed-run-01.md. Original SHA-256: 3a0d49857cd51b354e16987c96dd8cc21e2bf7ae5578d67e25e5d9530869c5dd. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Bounded sealer speed run 01

Local investigation on 2026-10-04, registered in the speed protocol (local checkout reference docs/experiments/benchmarks/sealer-speed-protocol.md; not present at origin/main) before measurement. The question is whether the memory screen's 97% heap reduction can coexist with much faster complete Segment builds. This is a finite optimization screen; full bounded-sealer acceptance and qualification remain unfinished.

What the different representations exposed

Ordering: sorting is a small cost on this fixture. The globally ordered Parquet output must still tolerate arbitrary late timestamps and stable ties. Sorting each chunk independently would transfer work to queries and changes row-group pruning; the earlier study measured that trade-off. It is outside this optimization.

Representation: the previous bounded builder used JSON for disposable run rows. That repeatedly escaped/copied payloads and parsed numbers and strings. The candidate uses fixed-width little-endian scalars and length-prefixed UTF-8, including floating-point bits. It adds no dependency and changes only private scratch bytes; the source journal remains durable custody.

Data movement: the largest fixture makes eighteen runs. The former algorithm rewrote all eighteen into two, even though merging three into one leaves sixteen runs and meets the same fan-in limit. The candidate rewrites only enough contiguous runs to reach the limit, then renames untouched run files into the next level. Larger inputs still use full passes as needed. Equal keys retain original journal order across both merged and untouched runs.

Time and system boundaries: preparing sorted runs while a journal fills could move some work earlier. Arbitrarily late rows still prevent final global ordering until the input closes; merging, compression and durable publication remain. Publishing independently sorted small Segments and compacting later could shorten publication latency, but changes query fan-out, pruning and recovery responsibilities. That is a future design branch, not an implemented result. Committed journals are already query sources; query availability can precede final Segment materialization.

Attribution

One opt-in native profile at each size, with nested spans reduced to exclusive wall time on the build thread. Fixture work and grading are excluded. On the 260.69 MiB journal the total was 6.436 seconds:

Work	Exclusive wall
Parquet encoding/compression/output, including sink hashing	2.400 s
Raw Batch Arrow construction, including Batch SHA-256	1.138 s
Intermediate merge of all eighteen runs	0.937 s
Final merge row decoding/selection, excluding table children	0.429 s
Spill encoding/writing, excluding sorting	0.415 s
Trigram filter construction	0.453 s
Table finalization/sync, excluding nested flushes	0.208 s
Sorted log Arrow construction	0.156 s
Run sorting	0.033 s

The remaining time is journal/frame decoding, projection, bookkeeping and directory sync. These are observed stage costs, not immutable hardware lower bounds. Removing all scratch/merge overhead alone would still leave several seconds in the measured materialization path.

The 65,536-row observer-off/control build took 1.344 s; the observer-on build took 1.370 s (about 2% apart). One pair cannot separate observer cost from run noise. Phase ledgers contain 5,740 and 22,921 entries, below their fixed cap; the preallocated ledger belongs to the pre-build baseline. Final comparisons use binaries compiled without the phase-probe feature. The host is an Intel i7-10750H with twelve logical CPUs and no advertised SHA instruction extension; storage is the mounted ext4 data drive. This differs from older VM measurements.

Raw profiles and exclusive summaries are under profile evidence (local evidence omitted from stage).

The final candidate profile (local evidence omitted from stage) reduced intermediate merge wall time from 937 to 26 ms, spill encoding/write from 415 to 138 ms, and final merge overhead from 429 to 130 ms. Residual costs were 2.308 s in Parquet encoding/output, 1.099 s in raw Batch Arrow construction including hashing, and 0.462 s in filters. This diagnostic build took 5.298 s: finalization/sync took 0.700 s versus 0.208 s in the baseline profile. Keep that variability visible; the observer-off paired medians below are the speed decision. The candidate profile exited 0 with a 2.4 GiB cgroup peak and zero swap.

Comparison and correctness

Comparisons use fresh processes, fixed seed 42, shuffled 1,024-byte logs, empty attributes, one node, 128 logs per Batch and one build worker. Each size uses three alternating pairs. Frozen binaries/source snapshots distinguish baseline JSON/full-pass, binary-only/full-pass and binary/partial-pass. Exact per-file hash equality against the bounded baseline is required, in addition to source rows, raw Batch custody and no spill leftovers. This preserves row-group boundaries and optional filters on the measured fixtures.

The independent source/oracle limits remain those of the prior screen: large mixed-signal performance populations and sustained consumer load are separate tests. Incremental heap excludes retained input fixtures; RSS includes fixture creation and allocator retention. Logical IO counters are not physical-device traffic, particularly when deleted scratch remains in the page cache.

All eighteen build trials exited 0 and matched every output file's hash. The twelve combined-candidate trials produced these medians of three runs:

Actual journal	Bounded JSON baseline	Binary/partial candidate	Faster	Baseline / candidate heap
65.17 MiB	1,347.32 ms	1,209.65 ms	10.2%	30.785 / 30.785 MiB
260.69 MiB	6,017.00 ms	4,771.02 ms	20.7%	32.061 / 32.061 MiB

CPU medians were 1,264.37 → 1,127.70 ms and 5,736.27 → 4,522.77 ms. Process RSS medians were 356.18 → 356.23 MiB and 1,145.84 → 1,152.61 MiB; there is no claimed whole-process RSS improvement over the bounded baseline. The unchanged largest-fixture heap remains 97.3% below run 01's historical whole-file-builder heap. This comparison of heap is distinct from comparing build times against an earlier revision or thermal state.

Both finite speed gates (at least 10% lower median), both heap-growth gates (no more than 10%) and the largest heap ceiling (at most 3% of the prior whole-file baseline) were met. The smaller speed gain is close to its threshold; three samples are not a broad capacity or latency claim. The largest baseline had a 6.494 s sample; it was retained. Other baseline samples were 6.017 and 6.003 s, and candidate samples 4.771, 4.788 and 4.770 s.

The separate binary-only/full-pass ablation had medians 6.039 → 4.838 s (19.9% faster), with identical heap and output hashes. Representation accounts for most of the measured speed improvement. The combined candidate's additional wall-time benefit over binary-only is small; these are separate paired populations and do not establish a precise incremental speedup for scheduling.

Scheduling does remove substantial data movement. At the largest fixture, logical reads/writes were 860.61 / 822.01 MiB for baseline, 806.19 / 767.60 MiB for binary-only, and 578.94 / 540.34 MiB for the combined candidate: about 33% fewer logical bytes than baseline. These counters include scratch operations, not just final output, and do not establish the same reduction in physical-device IO.

Six pending-journal trials exited 0. All 168 first-page query samples matched the independent Python expected answers across Scan/Walk and before/during/after sealing. Each trial had 19 or 21 during-build samples; eighteen changed/missing/duplicate-row controls were rejected. This checks existing journal visibility; it does not measure native ACK-to-queryable latency. The build timings in this population overlap concurrent queries and are not the isolated speed result above.

See combined comparison (local evidence omitted from stage), binary-only ablation (local evidence omitted from stage), query results (local evidence omitted from stage), and summary (local evidence omitted from stage).

Reproduction and resources

Every workload runs under python3 tools/resource_group.py --, with the existing 20 GiB memory max, 16 GiB high, no swap and 30-minute deadline. Build caches, frozen comparison binaries and trial scratch use the mounted data drive. The runner preserves command/binary/source evidence and cleanup receipts.

python3 tools/resource_group.py -- cargo build --offline --locked -p fabric-server --example responsibility_probe --features responsibility-alloc-probe,phase-probe --release
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py profile
# Build each registered source variant with responsibility-alloc-probe only,
# then freeze it before the next source change:
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py freeze --variant baseline
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py freeze --variant binary-only
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py freeze --variant candidate
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py binary
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py compare
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py pending
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py profile --variant candidate
python3 tools/resource_group.py -- python3 -B tools/bench/run_sealer_speed.py report

These commands describe the sequence; a frozen binary must be built from its matching archived source, not three copies of the final tree. Source fixture archives from the prior screen are reused by content hash. Near-instant is not a measured result or a changed freshness promise.

The initial profile, binary-only ablation, combined comparison and pending screen exited 0. Their cgroup peaks were respectively 2.5 GiB, 2.7 GiB, 2.7 GiB and 1,002.8 MiB; swap was zero. The comparison durations were 3 min 3.7 s (binary-only), 3 min 49.8 s (combined) and 1 min 3.6 s (pending), including fixture creation, exactness grading and cleanup.

All 28 profile/comparison/query trial cleanup receipts report owned scratch removed: 23,140,960,472 logical bytes, about 21.55 GiB cumulatively. This is not peak disk use. Content-addressed fixture evidence and build caches remain on the data drive. Temporary frozen comparison binaries are removed after the completed comparisons; their hashes and source snapshots remain in the record.

Verification and remaining limits

python3 tools/resource_group.py -- cargo test --offline --locked -p fabric-server segment::bounded --lib exited 0: thirteen tests cover every projected field, fixed scalar byte fixtures, float bits, malformed/truncated records, Unicode/control characters, stable ties, full/partial merge boundaries up to 513 actual runs, byte admission, raw custody, exact tables/filters and cleanup after corrupted input.

python3 tools/resource_group.py -- cargo xtask checks --profile fast exited 0, all seventeen current checks. Runtime was 2 min 47.0 s; cgroup peak 5.5 GiB, swap zero. The preserved receipts (local evidence omitted from stage) belong to the actual working-tree candidate, not a clean tagged revision. The arithmetic report rejected injected no-speed-gain and excess-heap controls and exited 0 for the registered finite gates. The manual documentation command python3 tools/resource_group.py -- /tmp/fabric-bun-runtime/bun-linux-x64/bun tools/docs/check.mjs also exited 0; automatic Bun CI/hooks remain disabled.

The full four-shape suite, sustained arrivals with concurrent queries and workers, native ACK-to-queryable latency, destructive recovery faults, registered sealer mutants, extended checks, unchanged soak and remote edge forwarding were not run here. Existing first-page grading is not full-chain query qualification. Source fixtures are synthetic, one signal and empty attributes; mixed-signal correctness is covered by tests, not these timing populations. Metadata growth and query-tail memory remain separate constraints.

The measured result supports retaining the compact representation and reduced rewrite schedule. For substantially shorter completion latency, the next design decision is how much work to perform before rotation and whether query availability or final sorted materialization is the deadline. Merely moving work outside the build timer would not establish a throughput or CPU saving.