Experiment benchmarks soak run 02 - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/soak-run-02.md. Original SHA-256: c70f25627a065d8fb252787e904706f955ca18bd87487edfe96bb98ffa741456. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Companion-compatible native soak, run 02

Status: the native protocol-R2 (local checkout reference docs/experiments/benchmarks/soak-protocol-r2.md; not present at origin/main) trial passed all ten registered gates. Evidence archival and owned-scratch cleanup completed. The trial used the frozen opt-in build; default adoption has separate unflagged build verification. Failed run 01 retains its original result and thresholds.

Build and freeze

The native release build exited 0, launcher unit cf0df87d63c1452fa847d390b7c22145, with FABRIC_ROW_GROUP_CHUNKS_EXPERIMENT=1 and FABRIC_PAGE_STORE_EXPERIMENT=1, without allocation instrumentation:

cargo build --release --offline --locked \
  -p fabric-server --bin fabric-server --example server_dump \
  -p fabric_o11y --bin fabric-node --example spindle_sim --example spool_dump

$FABRIC_DATA/results/readiness-continuation-soak-r2-01/build.json records the exact environment, unchanged before/after source hashes and all five binary hashes. The fresh frozen mini-tree is $FABRIC_DATA/scratch/soak-r2-01/frozen; preparation exited 0, unit f1773532692d49f2acced78c7addeb3d.

Pre/post smoke freeze verification both passed, units 8412ed05003e4b3fb6ac32ab1f2640db and f15671f2fb4e44aa9e64ae8f3350df6d, with matching manifest SHA-256 97ac7cd29170c8c86526735812d440519f5a614c556b3c7bb2c5de11c9831dcd. Their receipts are smoke-pre.json and smoke-post.json beside the build record.

Startup counterexample

The registered smoke launcher exited 1, unit 5159eb12dd9d471388d6a1eb60975b3b. Its runner result and traceback were written under frozen/target/alpha-soak-r2-smoke in the mini-tree above. The runner recorded passed=false, command exit 1, no truncated output and successful process-group cleanup.

The full freeze, source snapshot and failed output were archived as $FABRIC_DATA/results/readiness-continuation-soak-r2-01/failed-smoke-01.tar.gz, SHA-256 9d77131fcd5a11dd7f7189322a60488def1d8f4ce9c6b6178c050ebb5a535313, 11,739,769 bytes and 314 verified members. Preservation exited 0, unit aa839e300a324c84a3981304405a915e. After archive and stopped-group checks, owned scratch was removed, exit 0, unit 0eeac0c83ff1483f8bc87641b63f8f2f. failed-smoke-preservation.json retains the cleanup receipt. The five native binaries can be reused unchanged in a fresh freeze after the helper correction.

The helper required exact decimal readback of memory.max=4000000000 and memory.high=3000000000. Linux returned their page-aligned floors, 3999997952 and 2999996416; zero swap, 512 tasks and the two-CPU limit matched. The mismatch raised before certificates, server or simulator startup. This is a containment-helper defect, not evidence for any of the ten soak gates.

Corrected smoke and admission

Helper correction 8d4578b requires exact page-floor enforcement within the requested ceilings. Five regressions passed, unit 056de2ac73f4434f980b6415fb34eee7, exit 0. A real delegated empty-group control verified those floors and cleanup, unit 0a02ff6346004e2d853c40cf3a07e30e, exit 0.

Fresh freeze soak-r2-02 completed, unit 56678d2b08bc4c8ca06c6e28d4813573, exit 0; its precheck passed, unit d0f06f0353404f508e749c028bf32946, exit 0. Smoke unit 7691fee255774a55a41ba37a838f8d29 exited 1 after 121.287 seconds, with only the expected smoke sealing gate false: the offered data stayed below 64 MiB and no Segment was created. All other original gates passed. All 7,000 simulator batches were ACKed; all sixteen independently inspected companion sources were ACKed and recovered exactly. Both production exits and group cleanup succeeded. Server peak RSS was 34,208 KiB; combined cgroup peak was 38,928,384 bytes. ACK p99 was 42.959/42.800 ms across the smoke windows; query p99 was 0.034 s. Postcheck e58cc36018f8430eac8e42fd9cadafdb exited 0.

Admission receipt $FABRIC_DATA/results/readiness-continuation-soak-r2-01/smoke-02-admission.json records exit 0, unit d73dea89997447ceadef9930fa831cd3, and frozen manifest SHA-256 18e6ee2d397999758819076098f4ea9e20f044e30a8382202adf053806412662. The complete source frames occupied 52,498 bytes. Conservative 5,880-second projections were 4,409,832 bytes by offered-duration scaling and 5,322,300 bytes using two largest observed frames per 15 seconds plus startup, below the 8 MiB rotation threshold. This assumes stable metric cardinality and no exceptional diagnostic burst; final independent complete-prefix checking remains decisive. An extra, unregistered 50% rotation-margin assertion failed in unit e84d39b739eb44698c756bac5083e3d1, exit 1. That observer rejection is retained separately and is not a registered protocol-gate failure.

Full trial results

Full precheck 778f3dcb1c8742458acaf2d09ebbb2fd exited 0. The full trial ran on freeze 02, unit 5f9ee6b610a1418bb7ff96d172385516: 5,460 seconds offered workload, 7,200-second runner deadline and 7,500-second outer deadline. The runner exited 0 after 5,509.351 seconds; the summary has passed=true, all ten gates true and no violations. Simulator and production server exited 0; the combined group was empty with no forced kill, swap or OOM events. Postcheck 69176dcad59e40309c70a35a4dc1f512 exited 0 with the same 18e6ee2d397999758819076098f4ea9e20f044e30a8382202adf053806412662 manifest hash.

Measurement Observed result
Simulator custody 546,000 created, ACKed and recovered exact batches across 100 identities
Companion custody Complete sequences 1โ€“825, all ACKed and recovered exactly; no retained unACKed batch
Sealing Ten Segments; zero sealed journal files left
Backlog First, last and maximum all zero
Queries 546; zero errors or incomplete answers; p50 0.093644724 s, p99 0.215541813 s
Management 91 successes, zero failures
Server RSS drift First/last window medians 136,952/271,352 KiB; within the unchanged rule
Server RSS readings VmHWM 271,360 KiB; sampled window maximum 271,364 KiB, retained as separately reported readings
Combined containment Peak 945,795,072 bytes, ten tasks; zero high/max/OOM events and zero swap
Combined CPU/I/O 197,903,013 ยตs CPU; reported wbytes=1896140800 on each stacked device, not additive physical I/O
Live bytes at end 1,358,813,066

Each measured window held 60,000 ACKs:

Window start (s) ACK p99 (ms) RSS median (KiB) RSS sampled maximum (KiB)
60 43.064032 136952 232868
660 43.111727 238892 238968
1260 42.994775 250148 269760
1860 42.954342 270664 270676
2460 43.218010 270712 270736
3060 42.903293 270756 270756
3660 42.995071 270776 271280
4260 42.951109 271284 271348
4860 42.849510 271352 271364

The raw summary is $FABRIC_DATA/scratch/soak-r2-02/frozen/target/alpha-soak-r2-full/soak-summary.json. The verified archive below preserves this summary. This native trial includes the dedicated companion; comparison with historical standalone-server runs is not a controlled single-variable attribution. Companion wire attempts remain unobserved; its committed-source prefix, durable ACK cursor and recovered bytes were independently checked. Default-scale retention, arbitrary inputs and physical power loss remain outside this finite trial. No protocol, oracle or threshold changed.

Supplemental retry observations

$FABRIC_DATA/results/readiness-continuation-soak-r2-01/supplemental-observations.json records 1,342,953 send attempts and 546,000 distinct created batches with an ACK-kind event. Window attempt outcomes were 39.83โ€“41.44% ACK and 58.56โ€“60.17% HTTP 503 retries; each retry counts as another attempt. Every creation cohort reached 100% ACK-kind completion by the event log's end. Creation-to-first-ACK p99, including retry waiting, peaked at 144.672 ms; the registered single successful-attempt ACK p99 peaked at 43.218 ms. The event file lacks committed_through, so these supplemental completion counts do not replace custody grading by the unchanged delivery oracle.

The reducer exited 0, unit dfbb027d224a4a5e95581ee26ff8ea9f; its eight controls passed, unit eedad4bc7a6a4d77b13d10ebee53be1b, exit 0. Six archive-checker controls passed, unit 0425a8895c0d4710b06b5924b45a2184, exit 0. These observations introduce no acceptance gate or threshold.

Preservation

Archive unit 26051252e33e4895bf110a8d92e5e549 exited 0 after 37.567 seconds, 673.8 MiB cgroup peak, with 680 verified members. The full freeze and trial evidence are retained at $FABRIC_DATA/results/readiness-continuation-soak-r2-01/soak-r2-02-full-freeze.tar.gz, SHA-256 3b5b31ee03bd7de43fe8607eb20a9d4065f9409dd7459525e17aa73f99fc3fd7. After the unflagged binary comparison, cleanup unit 39397bc72cc54e8090642f456ff3311b exited 0 (8.831 seconds, 26.2 MiB peak). It reverified every archive member and the unchanged live tree, checked stopped processes, and removed exactly scratch/soak-r2-02. soak-r2-02-full-freeze-cleanup.json records owned_freeze_removed. A preceding cleanup invocation used the wrong launcher-receipt filename; unit cebbd7d3f85440c983ef6e1c3649d9c1 exited 2 before deleting anything.

The first standalone report copy had literal backslash-n manifest separators; its observer unit 5e33248e7dcc4de4a7365733e54a1dd2 exited 0, which did not establish copy validity. That artifact remains preserved. The corrected full-reports-verified-02 copy independently verified five files, exit 0, unit d3071def51d84b7cb31ff8bce231be53, manifest SHA-256 745e3d2729e036d4a53e5d14370dde5fcab4c31baebfb0cb4504538d84b0a1ad.