Experiment benchmarks production access campaign run 01 - kmosoti/FabricO11y GitHub Wiki

Production access campaign prerequisites and retained failure

Finite evidence recorded on 2026-10-10. The registered production-access adapter uses real browser passkeys, production console routes, scoped workload credentials and the exact linked Debian/RPM receipts. Historical campaign gates and independent delivery/custody oracles are unchanged. These finite results do not qualify a long campaign or physical authenticator.

Finite adapter results

production-soak-smoke-04 exited 1 in 145.391 seconds, with 831.6 MiB cgroup peak under 6 GiB. Its sole failed gate required a sealed Segment; the short workload stayed below the unchanged 64 MiB seal threshold. All nine other original gates were true: 7,000 offered records were acknowledged, final backlog was zero, the delivery oracle found zero violations, all nineteen companion sources passed exact custody, and seven queries had p99 0.0335 seconds. Both server and simulator exited 0. This remains a failed smoke, not a qualified soak. Earlier missing-helper, browser socket-path and enrollment-rate failures remain preserved.

Interrupted full trial

production-soak-full-01 used the original final-01 package pair and immutable freeze manifest bc899204771b2af62995b5002002a55b0f48f6205acf23f1b31f4d5b09161563. After about eighteen minutes, an actual read-only browser witness found the visible console locked, disconnected and no longer polling. A privileged owner operation refreshed user verification and replaced the cookie session, while the native UI retained its former session state. Successful owner API calls could not establish that the native UI remained alive.

The trial was interrupted at 1,100.243 seconds; the outer launcher exited 1, with 1 GiB peak under the admitted 6 GiB limit. Its original ledgers and failure state remain private. The stopped process groups were empty, browser scratch was removed, and one hundred generated simulator credential mirrors were hashed and removed without deleting oracle ledgers. Both pre/post freeze verifications exited 0. ResourceTiming had filled its bounded buffer; the last retained query timestamp does not identify the exact time the console locked. The observed locked DOM and absent active polling establish the failed prerequisite.

Repair and executable witnesses

Bridge source now pauses and drains requested UI polling before a real passkey refresh, reloads the native UI session, resumes polling, and keeps cumulative failure counters. A bounded observer counts only actual native UI Request objects for /v1/console/query; it observes response-body completion without parsing or retaining telemetry. Owner API requests do not contribute to those counters. Readiness requires a new native body completion within twenty seconds. Status, transport, visibility and progress controls fail closed.

console-bridge-refresh-smoke-01 exited 0 in 332.448 seconds, with 493 MiB peak under 4 GB: one genuine user-verification refresh, sixty-five actual UI body completions, zero status/transport failures, and no pending request at the final observation. Visible polling continued after refresh, then the same retained virtual authenticator signed in after a real server restart. Owned browser, TLS and server fixtures were cleaned. This tested bridge SHA-256 d05496b8f8d97e3e076d07a5f3f4ad7e7d0567ae80acaf53dbe82007ba5aa0f2 before the explicit readiness helper was added.

The final helper source is committed as c1a5542. Twenty-nine focused boundary, cleanup, source-freeze and original oracle controls exited 0. console-bridge-readiness-smoke-01 then exited 0 in 19.279 seconds with 401.8 MiB peak: its first actual body completion took 5.081 seconds, three bodies completed, no transport/status failure occurred, restart sign-in succeeded, and cleanup completed. Bridge SHA-256 was d117e59ab9ad7560043422724edad458d6a9e73ce03a288f70c8508c45e9bacf.

These bridge smokes used immutable binaries from the previous package pair; they verify the harness repair, not the successor package's release gates. A fresh R2 attempt and outage cells must use the cache-repaired successor pair and a new exact freeze. Their results were not available when this checkpoint was written.

Controlled evidence

Receipts and original failure ledgers are private under $FABRIC_DATA/results/production-soak-full-control-01, production-access-controls-11, console-bridge-refresh-smoke-01 and console-bridge-readiness-smoke-01. No raw telemetry, credentials or private keys are uploaded. Local controlled evidence is unavailable in a fresh checkout.

Successor trial stopped for candidate replacement

The next exact pair was Debian bcbf90976a16daf65186c89b37e32b6f14dff29122ad40670c9fc6447e1a1c42 / RPM 692984872fb6734773122d5db6b170868a71e7b993eb98400477f8a3e0ace030, with cache-repaired native source and the existing UI08 assets. production-soak-full-02 used fresh immutable freeze bab4e84e62d8e2ac189f58babf2d94267a5ede1d99ad1efe226de7b24d761534.

Its independent read-only browser witness crossed a real passkey refresh: at 273–286 seconds the console was visible and connected, cumulative body completions increased from 51 to 53, and status/transport failures remained zero. This supports the early harness repair, not full soak acceptance.

Hosted CI run 38071282547 then confirmed a distinct UI defect: the standalone Next snapshot page button overflowed a 390-pixel viewport at 200% zoom. The owner required corrected assets and a new package. Accordingly the soak was interrupted orderly after 562.893 seconds, before a measured ten-minute window completed. The harness exited −2, outer invocation exited 1, and peak memory was 1 GiB under 6 GiB. Both freeze verifications exited 0. Cleanup confirmed empty owned processes, removed browser scratch and one hundred hashed credential mirrors; original raw ledgers and private failed state remain preserved. No gate was relaxed and no outage cell was started on that superseded asset set.

The exact interruption and read-only witnesses are private in $FABRIC_DATA/results/production-soak-full-control-02/. The corrected candidate still requires a fresh full R2 trial and three outage cells.

Active UI09 checkpoint — 2026-10-10

production-soak-full-03 remains active on Debian build 04 / RPM build 03 with console build 09. A read-only post-refresh witness observed visible, connected polling, body completions 72→74 and zero transport/status failures. No full-soak outcome is available. Resident enrollment strengthening passed two native tests, while its actual browser acceptance and successor packages are pending. See current release progress for query and CI scope; no historical result is inherited by the new candidate.

UI09 resource-prerequisite failure

The previously active production-soak-full-03 ended autonomously with harness and outer exit 1 after 1,587.618 seconds (26 minutes 28 seconds), under the original 6 GiB bound. Its aggregate du probe could not read two rootless Podman apt partial directories created by a concurrent Debian05 build. The sampler retained the exception and the resource prerequisite failed closed; no disk bound or gate was bypassed. The actual UI prerequisite passed, immutable input/harness checks held, and both freeze verifications exited 0. Owned groups were empty, browser scratch was removed, and one hundred hashed credential mirrors were removed without deleting oracle ledgers. The later read-only browser observer found no driver because cleanup had already completed. Original raw evidence and private failed state remain preserved in the controlled production-soak-full-control-03/failure.json inventory. This failure does not complete the full-soak gate. A fresh final-candidate run remains queued.