Experiment benchmarks soak run 02 - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/benchmarks/soak-run-02.md. Original SHA-256:c70f25627a065d8fb252787e904706f955ca18bd87487edfe96bb98ffa741456. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Companion-compatible native soak, run 02
Status: the native protocol-R2 (local checkout reference docs/experiments/benchmarks/soak-protocol-r2.md; not present at origin/main) trial passed all ten
registered gates. Evidence archival and owned-scratch cleanup completed.
The trial used the frozen opt-in build; default adoption has separate
unflagged build verification. Failed run 01
retains its original result and thresholds.
Build and freeze
The native release build exited 0, launcher unit
cf0df87d63c1452fa847d390b7c22145, with
FABRIC_ROW_GROUP_CHUNKS_EXPERIMENT=1 and
FABRIC_PAGE_STORE_EXPERIMENT=1, without allocation instrumentation:
cargo build --release --offline --locked \
-p fabric-server --bin fabric-server --example server_dump \
-p fabric_o11y --bin fabric-node --example spindle_sim --example spool_dump
$FABRIC_DATA/results/readiness-continuation-soak-r2-01/build.json
records the exact environment, unchanged before/after source hashes and all
five binary hashes. The fresh frozen mini-tree is
$FABRIC_DATA/scratch/soak-r2-01/frozen; preparation
exited 0, unit f1773532692d49f2acced78c7addeb3d.
Pre/post smoke freeze verification both passed, units
8412ed05003e4b3fb6ac32ab1f2640db and
f15671f2fb4e44aa9e64ae8f3350df6d, with matching manifest SHA-256
97ac7cd29170c8c86526735812d440519f5a614c556b3c7bb2c5de11c9831dcd.
Their receipts are smoke-pre.json and smoke-post.json beside the build record.
Startup counterexample
The registered smoke launcher exited 1, unit
5159eb12dd9d471388d6a1eb60975b3b. Its runner result and traceback were
written under frozen/target/alpha-soak-r2-smoke in the mini-tree above.
The runner recorded passed=false, command exit 1, no truncated output and
successful process-group cleanup.
The full freeze, source snapshot and failed output were archived as
$FABRIC_DATA/results/readiness-continuation-soak-r2-01/failed-smoke-01.tar.gz,
SHA-256 9d77131fcd5a11dd7f7189322a60488def1d8f4ce9c6b6178c050ebb5a535313,
11,739,769 bytes and 314 verified members. Preservation exited 0, unit
aa839e300a324c84a3981304405a915e. After archive and stopped-group checks,
owned scratch was removed, exit 0, unit 0eeac0c83ff1483f8bc87641b63f8f2f.
failed-smoke-preservation.json retains the cleanup receipt. The five native
binaries can be reused unchanged in a fresh freeze after the helper correction.
The helper required exact decimal readback of memory.max=4000000000 and
memory.high=3000000000. Linux returned their page-aligned floors,
3999997952 and 2999996416; zero swap, 512 tasks and the two-CPU limit
matched. The mismatch raised before certificates, server or simulator startup.
This is a containment-helper defect, not evidence for any of the ten soak gates.
Corrected smoke and admission
Helper correction 8d4578b requires exact page-floor enforcement within the
requested ceilings. Five regressions passed, unit
056de2ac73f4434f980b6415fb34eee7, exit 0. A real delegated empty-group
control verified those floors and cleanup, unit
0a02ff6346004e2d853c40cf3a07e30e, exit 0.
Fresh freeze soak-r2-02 completed, unit
56678d2b08bc4c8ca06c6e28d4813573, exit 0; its precheck passed, unit
d0f06f0353404f508e749c028bf32946, exit 0. Smoke unit
7691fee255774a55a41ba37a838f8d29 exited 1 after 121.287 seconds, with only
the expected smoke sealing gate false: the offered data stayed below 64 MiB
and no Segment was created. All other original gates passed. All 7,000 simulator
batches were ACKed; all sixteen independently inspected companion sources were
ACKed and recovered exactly. Both production exits and group cleanup succeeded.
Server peak RSS was 34,208 KiB; combined cgroup peak was 38,928,384 bytes.
ACK p99 was 42.959/42.800 ms across the smoke windows; query p99 was 0.034 s.
Postcheck e58cc36018f8430eac8e42fd9cadafdb exited 0.
Admission receipt
$FABRIC_DATA/results/readiness-continuation-soak-r2-01/smoke-02-admission.json
records exit 0, unit d73dea89997447ceadef9930fa831cd3, and frozen manifest
SHA-256 18e6ee2d397999758819076098f4ea9e20f044e30a8382202adf053806412662.
The complete source frames occupied 52,498 bytes. Conservative 5,880-second
projections were 4,409,832 bytes by offered-duration scaling and 5,322,300
bytes using two largest observed frames per 15 seconds plus startup, below the
8 MiB rotation threshold. This assumes stable metric cardinality and no
exceptional diagnostic burst; final independent complete-prefix checking remains
decisive. An extra, unregistered 50% rotation-margin assertion failed in unit
e84d39b739eb44698c756bac5083e3d1, exit 1. That observer rejection is retained
separately and is not a registered protocol-gate failure.
Full trial results
Full precheck 778f3dcb1c8742458acaf2d09ebbb2fd exited 0. The full trial
ran on freeze 02, unit 5f9ee6b610a1418bb7ff96d172385516: 5,460 seconds
offered workload, 7,200-second runner deadline and 7,500-second outer deadline.
The runner exited 0 after 5,509.351 seconds; the summary has passed=true,
all ten gates true and no violations. Simulator and production server exited 0;
the combined group was empty with no forced kill, swap or OOM events.
Postcheck 69176dcad59e40309c70a35a4dc1f512 exited 0 with the same
18e6ee2d397999758819076098f4ea9e20f044e30a8382202adf053806412662
manifest hash.
| Measurement | Observed result |
|---|---|
| Simulator custody | 546,000 created, ACKed and recovered exact batches across 100 identities |
| Companion custody | Complete sequences 1โ825, all ACKed and recovered exactly; no retained unACKed batch |
| Sealing | Ten Segments; zero sealed journal files left |
| Backlog | First, last and maximum all zero |
| Queries | 546; zero errors or incomplete answers; p50 0.093644724 s, p99 0.215541813 s |
| Management | 91 successes, zero failures |
| Server RSS drift | First/last window medians 136,952/271,352 KiB; within the unchanged rule |
| Server RSS readings | VmHWM 271,360 KiB; sampled window maximum 271,364 KiB, retained as separately reported readings |
| Combined containment | Peak 945,795,072 bytes, ten tasks; zero high/max/OOM events and zero swap |
| Combined CPU/I/O | 197,903,013 ยตs CPU; reported wbytes=1896140800 on each stacked device, not additive physical I/O |
| Live bytes at end | 1,358,813,066 |
Each measured window held 60,000 ACKs:
| Window start (s) | ACK p99 (ms) | RSS median (KiB) | RSS sampled maximum (KiB) |
|---|---|---|---|
| 60 | 43.064032 | 136952 | 232868 |
| 660 | 43.111727 | 238892 | 238968 |
| 1260 | 42.994775 | 250148 | 269760 |
| 1860 | 42.954342 | 270664 | 270676 |
| 2460 | 43.218010 | 270712 | 270736 |
| 3060 | 42.903293 | 270756 | 270756 |
| 3660 | 42.995071 | 270776 | 271280 |
| 4260 | 42.951109 | 271284 | 271348 |
| 4860 | 42.849510 | 271352 | 271364 |
The raw summary is
$FABRIC_DATA/scratch/soak-r2-02/frozen/target/alpha-soak-r2-full/soak-summary.json.
The verified archive below preserves this summary. This native trial includes the dedicated companion;
comparison with historical standalone-server runs is not a controlled
single-variable attribution. Companion wire attempts remain unobserved; its
committed-source prefix, durable ACK cursor and recovered bytes were independently
checked. Default-scale retention, arbitrary inputs and physical power loss remain
outside this finite trial. No protocol, oracle or threshold changed.
Supplemental retry observations
$FABRIC_DATA/results/readiness-continuation-soak-r2-01/supplemental-observations.json
records 1,342,953 send attempts and 546,000 distinct created batches with an
ACK-kind event. Window attempt outcomes were 39.83โ41.44% ACK and
58.56โ60.17% HTTP 503 retries; each retry counts as another attempt. Every
creation cohort reached 100% ACK-kind completion by the event log's end.
Creation-to-first-ACK p99, including retry waiting, peaked at 144.672 ms;
the registered single successful-attempt ACK p99 peaked at 43.218 ms.
The event file lacks committed_through, so these supplemental completion
counts do not replace custody grading by the unchanged delivery oracle.
The reducer exited 0, unit dfbb027d224a4a5e95581ee26ff8ea9f; its eight
controls passed, unit eedad4bc7a6a4d77b13d10ebee53be1b, exit 0.
Six archive-checker controls passed, unit
0425a8895c0d4710b06b5924b45a2184, exit 0. These observations introduce
no acceptance gate or threshold.
Preservation
Archive unit 26051252e33e4895bf110a8d92e5e549 exited 0 after 37.567 seconds,
673.8 MiB cgroup peak, with 680 verified members. The full freeze and trial
evidence are retained at
$FABRIC_DATA/results/readiness-continuation-soak-r2-01/soak-r2-02-full-freeze.tar.gz,
SHA-256 3b5b31ee03bd7de43fe8607eb20a9d4065f9409dd7459525e17aa73f99fc3fd7.
After the unflagged binary comparison, cleanup unit
39397bc72cc54e8090642f456ff3311b exited 0 (8.831 seconds, 26.2 MiB peak).
It reverified every archive member and the unchanged live tree, checked stopped
processes, and removed exactly scratch/soak-r2-02.
soak-r2-02-full-freeze-cleanup.json records owned_freeze_removed.
A preceding cleanup invocation used the wrong launcher-receipt filename;
unit cebbd7d3f85440c983ef6e1c3649d9c1 exited 2 before deleting anything.
The first standalone report copy had literal backslash-n manifest separators;
its observer unit 5e33248e7dcc4de4a7365733e54a1dd2 exited 0, which did not
establish copy validity. That artifact remains preserved. The corrected
full-reports-verified-02 copy independently verified five files, exit 0, unit
d3071def51d84b7cb31ff8bce231be53, manifest SHA-256
745e3d2729e036d4a53e5d14370dde5fcab4c31baebfb0cb4504538d84b0a1ad.