Experiment benchmarks cross system operations sweep findings - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/cross-system-operations-sweep-findings.md. Original SHA-256: 83241007dcba41182d73791cfeeb0202b30be3e131e0c3e324fbe6299189131b. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Native operations sweep findings

The registered 54-cell cadence sweep (local checkout reference docs/experiments/benchmarks/cross-system-cadence-sweep-proposal.md; not present at origin/main) and fresh 12-cell boundary confirmation (local checkout reference docs/experiments/benchmarks/cross-system-cadence-boundary-proposal.md; not present at origin/main) completed. The original sweep's authenticated reduction (local evidence omitted from stage) completed; its coordinator receipt (local evidence omitted from stage) records exit 0/child exit 0 and cleanup in 4.369 seconds. The reduction matched the original telemetry hashes and receipt metrics, rechecked the existing telemetry validator, counted the actual independent query verdicts and checked paired append sync coverage. No production change or quiet-timer causal win is nominated.

Coordinator receipt (local evidence omitted from stage) records exit 0/child exit 0, 331.056 seconds and cleanup. Native receipt (local evidence omitted from stage) records 54 cells, 326.602 internal seconds and 5819973 logical retained bytes. The contained job peaked at 333701120 bytes, about 318.2 MiB, with no high/max/OOM events and zero swap. Per-cell process memory and that cgroup peak describe different domains. The frozen phase-probe binary SHA256 is e53193430ef923b914d6f3290b6caa16d43d64f18e550c72352c1c0a1fb8b92f.

Actual counts are 27648 offered Batches and 27648 accepted ACKs, 432 positive complete pre/post query chains, 108 rejected missing/duplicate query controls, and 108 rejected telemetry controls. Every cell records exact Batch custody and reopened retry/conflict checks, and all scratch cleanup flags are true. The reduction binds original receipt SHA256 be18bd52ab6f562b73841e610696c288c0cda8ac211324c47625a017bc961416; its helper SHA256 is bfc20fb1b5c7cf20590b2b2572ff975d9e4ce07d367c049151f367f565b3ca02. One complete exact-readback representative fixture is retained; other successes retain whole-file hashes, exact ACK/hook/poll telemetry and oracle verdicts. These are finite native Store/Intake checks, not HTTP/TLS, sustained capacity, power-loss testing or deployment qualification.

The strongest reproducible screening observation is observer contamination. For offers made after sealer completion, polled versus event-only median ACK latency rises in every one of the 12 paced matched comparisons. At 2 ms target cadence the paired increase is 5.34–9.64%, with 20–50 ms more process CPU. At 5 ms it is 17.05–38.24%, with 70–100 ms more CPU. The CPU resolution is 10 ms. Event-only versus quiet 2 ms medians differ by −1.26% to +1.08%; the 5 ms comparisons include a quiet worker 1 tail outlier, so event neutrality is not established universally.

Target cadence Actual event-arm append groups for 512 ACKs Event-arm median ACK after sealing
Burst 0 ms 4 No after-sealing offers
2 ms 165–170 6.801–6.911 ms
5 ms 506–508 4.738–4.763 ms

Existing append-intent hooks count actual append calls, and data/marker sync hooks equal twice that count in every observed cell. The burst fills four 1 MiB Batch-payload groups. At 2 ms, observed average group size is about 3.01–3.10 Batches; at 5 ms it is about 1.01. The 5 ms cells have lower typical after-sealing latency but roughly three times as many append/sync groups and more process CPU per 512 ACKs. The lower offered rate, longer measurement duration and different actual timer intervals are part of these workloads; this is not a throughput comparison at equal offered load or a causal alteration of the 2 ms quiet rule.

After-sealing actual 2 ms event-arm offer intervals have medians 2.072–2.074 ms, with 26.6–28.8% at or below 2 ms. Polling changes those ranges to 2.075–2.078 ms and 19.1–20.7%. Actual 5 ms intervals remain above 2 ms. These are producer offer observations, not timestamps of commit-thread receives. They support treating observer work, producer timers and thread scheduling as mechanisms to test.

Tail behavior is mixed. After-sealing 2 ms p 99 ranges 10.758–13.006 ms in event-only cells and 11.192–26.494 ms with polling. At 5 ms the ranges are 6.058–7.352 ms and 8.295–25.005 ms. Quiet 5 ms includes a 25.235 ms worker 1 outlier; whole-cell skewed 3 quiet p 99 reaches 31.659 ms, mostly associated with offers made during sealing. Separating those populations reduces that cell's p 99 to 7.167 ms. Neither filtering nor 512 observations removes correlated replies: many ACKs share a commit group, and two body markers are not independent scheduler seeds or a tail guarantee.

Initial sealed files match exactly across row shapes and observer/cadence arms: 1056531, 1056658 and 1056658 bytes. The dense-middle row count differs 64-fold while encoded Batch size stays 8192 bytes, separating row work from journal byte size. Builder-return hooks lack labels; checkpoint service duration remains unmeasured. Polling and event-free debt intervals have different widths, so they cannot be ranked as equally precise reclamation speed measurements.

The fresh discriminator uses worker 1 at 1/ 2/ 3 ms, quiet/events and two new markers. Its full registered outcomes follow. The original 54-cell observations above remain part of the evidence; the confirmation does not replace their outliers.

Fresh boundary confirmation

The boundary reduction (local evidence omitted from stage) records 12 cells, 6144/ 6144 offers/ACKs, 96 positive complete query chains, 24 rejected query controls and 24 rejected telemetry controls. Initial file byte equality holds within both markers, with the same three sizes recorded above. The native coordinator (local evidence omitted from stage) records exit 0/child exit 0, 84.011 seconds and cleanup; internal execution took 79.734 seconds and retained 3018752 logical bytes. Cgroup peak 319262720 bytes (about 304.5 MiB) had no high/max/OOM events and zero swap. The reducer coordinator (local evidence omitted from stage) records exit 0/child exit 0 and cleanup in 4.424 seconds. Binary SHA256: 31333c9c51992ecd1bba8474b8a08859a0fd0bc521b8196cb6c3881296e0ffa6.

Both registered association screens pass:1 ms/ 3 ms event append-group ratio is 0.0331 in each marker (threshold ≤0.75); after-sealing median ACK ratios are 5.695 and 5.617 (threshold ≥1.10). All six event/quiet median comparisons meet the separate ±10% observer screen:

Marker suffix 1 ms 2 ms 3 ms
4363 −1.02% −0.05% −1.43%
4364 +1.16% +1.94% −0.94%

These are the registered median screens, not a claim of observer-neutral tails. Both replications' event-arm outcomes are retained here:

Marker Target After-sealing ACKs ACK p 50/p 95/p 99, ms Append groups / data-marker syncs Process CPU, ms
4363 1 ms 326 35.391 / 62.833 / 65.532 9 / 18 120
4364 1 ms 331 35.579 / 61.803 / 65.041 9 / 18 130
4363 2 ms 465 6.904 / 9.189 / 10.409 169 / 338 160
4364 2 ms 465 6.950 / 11.252 / 37.665 159 / 318 150
4363 3 ms 479 6.215 / 8.013 / 8.427 272 / 544 180
4364 3 ms 480 6.334 / 8.138 / 8.710 272 / 544 190

Append/sync counts cover all 512 accepted Batches, while latency columns use only offers made after sealing. Mean accepted Batches per append are 56.89 at 1 ms, 3.03/ 3.22 at 2 ms, and 1.88 at 3 ms. Quiet-arm after-sealing p 99 is 10.940/ 11.176 ms at 2 ms: the 37.665 ms event-arm outlier survives the successful median screen. Quiet-arm 1 ms p 99 is 65.585/ 64.430 ms and 3 ms is 8.829/ 9.563 ms. Event-minus-quiet CPU differences are 0/ 20 ms at 1 ms, 10/ 0 ms at 2 ms and 0/ 10 ms at 3 ms; these differences are coarse because the tick resolution is 10 ms.

Event-arm after-sealing actual interarrival medians are 1.080/ 1.083 ms, 2.075/ 2.073 ms and 3.069/ 3.068 ms at the respective targets. Coefficients of variation are 0.306/ 0.305, 0.181/ 0.139 and 0.115/ 0.113. All observed 1 ms intervals, 28.45%/ 25.86% of 2 ms intervals and none of the 3 ms intervals are at or below 2 ms. Maximum event-arm offer lateness ranges 1.173–1.302 ms. Producer observation is distinct from commit-thread receive timing; 3 ms still groups nearly two Batches per append despite every observed interval exceeding the 2 ms quiet setting.

The confirmation supports the registered finite arrival-cadence association: much larger groups accompany fewer append/sync calls and lower measured process CPU, alongside higher ACK latency. It does not establish a quiet-timer causal win or an optimal production setting. Changing cadence also changes offered rate and duration, and IO/thread scheduling can change receive gaps. Remaining discriminators are a separately registered fixture-only quiet/window parameter ablation at fixed actual offered cadence, and investigation of the 2 ms tail outlier using existing frame-sync spans plus fresh replication. Checkpoint service identity remains unavailable. Neither experiment is executed or authorized by this findings record.