Experiment benchmark soak r3 amendment proposal 01 - kmosoti/FabricO11y GitHub Wiki
Prospective two-hour soak amendment
Source draft SHA256 ba53f17fb8163b5597cc033318b5d4f7df57323565382fe7fa8e0edfaff2cdaa. Published for protocol-scope and pre-flight review; no amendment or execution is implied by publication.
Prospective R3 soak amendment
PRIVATE PROPOSAL; NOT REGISTERED, ADMITTED OR RUN. Olo publishes this draft and requests protocol scope before any trust or implementation change. R2 remains historical: its 60 + 9×600 = 5460 offered seconds cannot establish two hours; its 7200-second runner allowance is a timeout.
R3 offers the same seeded workload for 7200 seconds, including the unchanged 60-second warmup. Retain 100 identities/workers, seed 0xA11FA001, rates/body shapes, packaged daemons/source-matched helpers, query/management/UI/sampling cadence, storage settings and CPU placement. Retain nine nonempty measured cohorts, all ten original gate names, numerical thresholds and first/last RSS rule. This is one finite trial, not deployment qualification.
The literal cohort boundaries are 60, 853, 1646, 2440, 3233, 4026, 4820, 5613, 6406, 7200. Warmup is [0,60); each adjacent boundary pair is one half-open measured cohort. Bucket ACKs by attempt-start integer nanoseconds against the simulator's recorded start; retain end-minus-start latency. Align RSS cohorts to that same start. Keep backlog sampling every60 seconds, with comparison endpoints60 and7195. Preserve raw source/attempt/answer data and separately report creation latency, retries and censoring.
The existing approved outer deadline remains 7500 seconds, from the admitted launcher start. Proposed phase ceilings:
| Phase | Ceiling |
|---|---|
| Setup, including enrollment and browser readiness | 180s |
| Actual offered exposure, with server/probes/sampling active throughout | 7200s |
| Shared post-offer simulator drain and sealing | 60s |
| Orderly production shutdown | 20s |
| Raw measurement/resource checkpoint | 20s |
| Process/temporary-fixture closeout | 20s |
These ceilings total7500 with no overhead slack; they are failure boundaries, not a static guarantee of completion. Use the smaller of each phase's remaining allowance and the global remaining deadline; no phase borrows another allowance. Any overrun, early exposure end, forced shutdown or unconfirmed stop fails and preserves evidence. Retain the original300-second seal upper bound, but exhaustion of the shared60-second tail fails rather than granting more time. The simulator can exit after the final7199 tick, and its60-second watchdog only sets a stop flag before joining workers: neither proves7200-second exposure or a hard drain bound. Replace unbounded/oversized waits and sleeping probe joins with bounded phase handling; do not count setup/drain as offered exposure.
Move stopped Spool inspection (existing60s/128MiB boundary), server dump and unchanged oracle grading into separate default1800-second contained cells. Checkpoint complete stopped server/Spool and resource evidence outside automatically cleaned launcher scratch before the measurement cell exits. Measurement and closed-runtime receipts remain immutable and explicitly pending acceptance. A final grading receipt binds their hashes, source/package/helper/freeze identities, exact stopped units and complete raw inputs; do not rewrite a pending receipt into a pass. Preserve the same ten gate decision rules and custody requirements. Failure or budget exhaustion grants no cleanup authority.
Require actual whole-parent kernel evidence for memory.high5GiB, memory.max6GiB, memory.swap.max0 and pids.max1536, plus existing subgroup limits, peaks/events/swap/population and confirmed stop. Requested outer16/20GiB values cannot substitute for these tightened actual limits. Keep5GiB live-data,1MiB command/evidence limits and fresh strict namespace census/reservations;7200 seconds increases raw volume, so capacity is not assumed. Archive reservation remains actual complete stopped bytes plus1GiB. Reject same-device descendant bind mounts using decoded mountinfo before inventory and again before deletion; st_dev equality alone is insufficient. Reuse the existing guarded mount refusal; preserve all failures until verified archival.
Register independent literal controls before implementation: old5460 mislabeled7200; missing/overlapping cohorts and exact edge±1ns buckets; old600-second arithmetic; early final exposure; every phase overrun and exhausted remaining budget; grading before stop or with changed raw bindings; missing/forged parent high/max/swap/tasks, OOM or populated final group; same-device nested mount before inventory and before first unlink, with zero deletion. Keep existing adverse gate, custody, freeze and archive controls. Shortened synthetic timers verify handling only, never elapsed two-hour acceptance.
Affected files: docs/experiments/benchmarks/soak-protocol-r2.md (linked prospective successor; preserve R2 meaning), tools/qualification/{soak_tier.py,runner.py,soak_observations.py,release_runtime.py,production_access.py,prepare_soak.py}, existing soak/runner/runtime tests and tools/bench/labs/completion/cgroups.py; private soak-command-plan.{md,json}, soak-closeout-candidate.py, soak-preserve-candidate.py, soak-preservation-controls.py and control plan. Runner currently caps the whole setup/workload/grading command at7200: any successor allowance is an explicit prospective verification-policy amendment within the already-approved7500 outer, never a bypass. Freeze all changed loaded/archive dependencies and pins before new smoke/full execution. Publish results only in the wiki.