Experiment benchmarks performance frontier run 01 - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/benchmarks/performance-frontier-run-01.md. Original SHA-256:4d193d1c6292cb708cdc3f8428e8863bcca2a171e8e3c5fe270a93ae087505cb. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Performance frontier, run 01
Status: execution started on 2026-10-05 after data-drive access was restored.
Plan (local checkout reference docs/experiments/benchmarks/performance-frontier-lab-plan.md; not present at origin/main); existing registered completion protocols
govern the prerequisite jobs. New comparisons remain in preparation.
Three PIs are active in capacity, query and operations preparation. Root alone
admits workloads, serially through tools/resource_group.py, on data-drive
scratch with 16 GiB memory high / 20 GiB max / no swap. The frontier screen has
a four-hour aggregate allowance; prerequisite jobs below count against it even
though their receipts use the existing completion runner.
Operations: storage-transition prerequisite
Command (exit 1 from coordinator; Cargo exit 101):
python3 -B tools/resource_group.py -- python3 -B tools/bench/labs/completion/run_job.py --id frontier-storage-01 --lab recovery --stage recovery --seconds 600 -- cargo test --offline --locked -p fabric-server --test completion_storage -- --test-threads=1
Seven tests passed, one failed and the intentional subprocess helper was ignored. The actual pre-publication pause/full-query-chain case and span table/filter write/sync-failure case passed. The missing-query-table case failed the unchanged oracle's retained-window, freshness and gaps checks: it expected empty metadata after excluding all records; the application retained manifest/gap metadata. This is an unresolved contract/fixture/oracle-model mismatch, not permission to weaken the oracle or claim the storage suite passed. Operations PI is tracing it.
PI source review localized the mismatch: --unavailable excludes whole records
from all expected metadata, whereas this fixture deletes only logs.parquet.
The manifest, raw Batches and gaps survive. Failure occurred on the first Scan
case, before Walk or deletion of metrics/spans tables. Those later cases remain
unestablished. A table-specific availability checker requires a separate
trust-boundary design and negative controls; do not normalize the actual answer
or erase the three violations. No production fix is justified by this result.
Elapsed job time 14.024 seconds; systemd reported 14.824 CPU seconds and 1.3 GiB
peak memory including compilation. Swap was zero. Successful fixture scratch
was removed by the tests; failed state was retained by the outer launcher under
$FABRIC_DATA/evidence/fabric-work-15a7415d71db483481bb0c2c924eb63a.
No deletion of this failure is authorized by a successful independent job.
Receipt and output (local evidence omitted from stage). The existing test fixture overwrites some per-call oracle artifacts; its appended verdict ledger is not a full replay archive. Improve capture before a rerun.
Capacity: supplemental spill accounting
The PI added fail-closed comparison of the selected binary against all eight historical provenance records before collection. Current and historical hashes match. This enforces the already registered same-binary requirement; no protocol or acceptance outcome changed. Collection uses the existing spill protocol and does not rebuild the shared benchmark executable.
python3 -B tools/resource_group.py -- python3 -B tools/bench/labs/completion/run_job.py --id frontier-spill-01 --lab memory --stage capacity --seconds 900 -- python3 -B tools/bench/labs/completion/spill_measure.py --id frontier-spill-01
Outcome: passed, exit 0, all eight registered cells collected with identical input hashes and final manifests. The observer's exact 9-byte/two-call control passed. Job time 306.322 seconds; systemd reported 53.454 CPU seconds and 824.2 MiB peak cgroup memory including generation/readback, with zero swap. Owned success scratch was removed. This does not change the historical C2 large-row failure. Coordinator receipts (local evidence omitted from stage) and measurements (local evidence omitted from stage). Observed logical spill bytes are not physical disk writes or timing comparisons.
| Cell | Journal bytes | Logical spill bytes | Spill / journal |
|---|---|---|---|
| steady-16 | 16778133 | 19295590 | 1.1500 |
| steady-32 | 33558873 | 38380800 | 1.1437 |
| steady-64 | 67110273 | 76539600 | 1.1405 |
| steady-128 | 134222207 | 153061770 | 1.1404 |
| steady-256 | 268440962 | 336302946 | 1.2528 |
| outage-64 | 67110273 | 76539600 | 1.1405 |
| adversarial-64 | 67110273 | 76539600 | 1.1405 |
| bigrows-64 | 67183343 | 67640110 | 1.0068 |
The rise at 256 MiB is consistent with an additional merge rewrite, but these receipts do not count runs or passes: causal attribution remains a hypothesis. This supports C5's focused boundary investigation, not immediately increasing buffer sizes. Final Segment writes are excluded from the measured quantity.
Query: prospective comparison preparation
The PI is specifying source-order/page-limit attribution against the existing probe. Identified harness gaps include hardcoded page size and continuation guards, fixed plan order, and shuffled timestamp assignment changing semantic ledgers. These must be addressed explicitly before a matched comparison is registered. No Q3 performance result exists yet.
The PI produced the Q3 draft (local checkout reference docs/experiments/benchmarks/query-frontier-protocol.md; not present at origin/main). Root retains it as
unregistered: selector-dependent transcript checks, separate plain/counting
binary handling and source-work/construction instrumentation remain necessary.
The 100-row page cell requires 656 pages for a broad full chain, so budget
feasibility must be checked without weakening complete-chain correctness.
Additional admission controls
All commands below used the same outer launcher and serialized run_job.py:
| Job / child command | Exit | Elapsed job seconds | Interpretation |
|---|---|---|---|
frontier-provenance-controls-01: python3 -B tools/bench/labs/completion/spill_provenance_control.py |
0 | 0.503 | Matched provenance reaches the controlled admission boundary; a mismatched historical hash is rejected before scratch/compiler/workload |
frontier-query-controls-01: python3 -B tools/bench/labs/completion/profile.py --controls |
0 | 0.503 | Existing profiling ledger controls pass; not Q3 timing evidence |
frontier-cgroup-01: python3 -B tools/bench/labs/completion/prerequisites.py cgroup |
0 | 0.504 | Refreshed delegated cgroup preflight; not application load evidence |
Exact commands/source snapshots and outputs are under corresponding
data/lab-completion-run-01/coordinator/ job directories. All three success
scratch directories were removed. Approximate aggregate job time through this
checkpoint is 321.856 seconds of the 14400-second frontier screen allowance.
No workload remains active at this checkpoint. R3's unresolved checker-model
boundary prevents claiming its prerequisite passed; new Q3 comparisons and
dependent capped-load cells remain queued, not silently admitted.
Repository verification checkpoint
frontier-docs-01 ran completion/checks.py --profile documentation --id frontier-docs-01 through the same contained coordinator, exit 0, 16.526
seconds, owned scratch removed. The spill table was appended afterward; this
receipt applies to the source snapshot captured for that check.
frontier-fast-01 ran completion/checks.py --profile fast --id frontier-fast-01,
exit 1, 90.784 seconds. Formatting, workspace tests and Clippy failed.
Workspace tests encounter the same missing-table oracle mismatch; Clippy reports
five errors in the prepared completion_builder example, including type
complexity. These failures remain open; this campaign has no clean fast-profile
claim. Check/layer/purity and independent delivery/query-oracle checks passed.
Exact individual receipts and retained failures are under the corresponding
coordinator/recovery directories and the outer launcher's owned evidence folder.
Whole-job peak was 8.7 GiB including verification processes, zero swap.
Aggregate recorded job time including both verification jobs is approximately 429.166 seconds of the four-hour allowance. All launched jobs have terminated; success scratch was cleaned, failure scratch preserved. No release/default or oracle change was made. The next admission checkpoint must address the verifier coverage design and query harness adaptations, rather than claim readiness from the successful spill collection alone.