Experiment benchmarks native key first findings - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/benchmarks/native-key-first-findings.md. Original SHA-256:c466ecb2f1aa734056b7ac1d54704c635b06974ccdbe15ac8cac5f6128c8f61d. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Native key-first query admission screen
The registered proposal (local checkout reference docs/experiments/benchmarks/native-key-first-proposal.md; not present at origin/main) is rejected at screen: no held-out confirmation or nomination. Screen and independent reduction both completed with exit 0. The reduction report (local evidence omitted from stage) recomputed every median from native outputs and found 57 CPU failures, 56 wall failures and 58 populations failing either 1.05 guard. Its admission receipt (local evidence omitted from stage) records decision: reject against the exact screen-result SHA256. The primary benefit passes; the unconditional adverse-population guards do not. No threshold or population was changed after measurement.
The 192 plain paired populations are 4 fixture cells × 2 layouts × 2 plans × 6 shapes × 2 first/reuse boundaries: 96 first-call pairs and 96 reuse-median pairs, derived from 768 actual plain timed calls across eight fresh children. They are not 192 independent repeated fixtures. Counted phase-on/off contribute another 384 paired populations; all three regimes together contain 2,304 timed calls. All boundaries remain separate.
Later workspace verification found Clippy-only redundant initializers in the
new corruption test and benchmark example. The test repair (local checkout reference docs/experiments/benchmarks/native-verification-retry-proposal.md; not present at origin/main)
and example repair (local checkout reference docs/experiments/benchmarks/native-verification-example-retry-proposal.md; not present at origin/main) remove those
initializers; assertions, query implementation and frozen benchmark binaries are
unchanged. Native results remain bound to their retained pre-repair source
snapshots, and both failed verification receipts are preserved.
| Population | CPU ratio | Wall ratio | Registered consequence |
|---|---|---|---|
| Wide/eight attributes, tail Scan/common reuse | 0.887289 | 0.886114 | Primary benefit passes |
| Same fixture and query, first call | 1.062809 | 1.065875 | First-call guards fail |
| Wide/eight attributes, sealed Scan/common reuse | 1.179948 | 1.179716 | Diagnostic population guards fail |
| Narrow/no attributes, tail Scan/broad first | 0.538127 | 0.539866 | Minimum observed plain ratios |
| Narrow/no attributes, tail Scan/absent reuse | 2.041953 | 2.026387 | Maximum observed plain ratios |
The registered primary, 1,024-byte bodies with eight attributes, tail Scan/common reused first page, has baseline/candidate CPU 7.169/6.361 ms and wall 7.208/6.387 ms: ratios 0.887289 and 0.886114. This meets the primary CPU ≤0.90 and wall ≤1.05 thresholds. Both counted regimes save 2,558,531 requested allocation bytes, with incremental peak unchanged; requested ratios are 0.927521 with phases enabled and 0.927463 with phases disabled. This supports avoided cloning/matching in that finite population, while the overall cost guard still rejects the candidate.
Adverse populations include 16-byte/no-attribute tail Scan/absent reuse, CPU/wall ratios 2.041953/2.026387, and wide/eight-attribute sealed Scan/common reuse, 1.179948/1.179716. The primary population's first call also regresses, 1.062809 CPU and 1.065875 wall, violating the separate first-call guard. An isolated good reuse median cannot override these registered failures. This screen has one seed and no fresh held-out repetition; adverse ratios are observed comparisons, not proof that every change is caused by the extra key branch. No samples or populations are pooled.
The executed screen retained 24 fresh children, 2,304 first-page calls, 576 full canonical continuation chains comprising 11,040 pages, and 1,440 rejected oracle output controls. Independent reduction regraded all 576 chains and all 1,440 actual bad-answer vectors with the unchanged Python oracle, and authenticated 4,840 decoded content objects. Its positive synthetic admission control succeeded and seven injected defects were rejected: primary CPU, another CPU population, another wall population, allocation, semantic failure, missing population and nonfinite ratio. Every child exercised six shapes, two layouts and Scan/Walk, with one first call and three subsequent calls. Full drains were outside query clocks; these measurements establish no complete-chain CPU gain. Native sealing sorts the complete key and these fixtures occupy one row group, so this is not a new pruning/locality experiment.
Native controls key-first-controls-01 ran eight test commands under flag 0/1 and borrowed flag 0, exit 0. Scan/Walk retained the exact incomplete answer when a later heap-rejected row contained malformed attributes JSON inside a valid size/schema/digest envelope: expected ident at line 1 column 2, one unavailable segment. Existing missing-table, missing-raw and raw-integrity controls remained executable. Independent reduction authenticated their receipts and retained outcomes; these checks are finite counterexamples, not exhaustive corruption coverage. Broader candidate checks key-first-regressions-01 completed with exit 0: 64 library tests passed, two ignored; ten history tests and one snapshot-transition test passed, totaling 75 passed. Correctness evidence does not override the failed performance guards.
The screen coordinator reports 545.645 seconds including builds and the whole Python/native batch, cgroup peak 507,408,384 bytes, zero swap and zero high/max/OOM events, with owned scratch removed. The harness receipt reports 109,162,496 allocated retained bytes and both exact native-state readbacks before cleanup. Source archives, exact input/answer/control objects and binary/build hashes bind the working-tree source rather than HEAD alone. Compiler: rustc 1.99.0 (b940084d7), Cargo 1.99.0, Python 3.14.7. Plain/count builds used a shared Cargo cache; their build durations are not independent clean-build comparisons.
| Contained job | Exit | Wall seconds | Cgroup peak bytes | Owned scratch |
|---|---|---|---|---|
| Native controls receipt (local evidence omitted from stage) | 0 | 17.891 | 1,893,449,728 | Removed |
| Screen receipt (local evidence omitted from stage) | 0 | 545.645 | 507,408,384 | Removed |
| Independent reduction receipt (local evidence omitted from stage) | 0 | 155.306 | 151,212,032 | Removed |
| Candidate regression receipt (local evidence omitted from stage) | 0 | 200.892 | 1,838,968,832 | Removed |
All four receipts report zero swap and zero high/max/OOM events. Cgroup peaks cover whole jobs, including compilation or Python grading where applicable, and are not native-query peak heap. Root disposes frozen binary copies after final authenticated readback; executable hashes and source/build evidence remain.
Observer sensitivity is material even on the primary: counted phase-on CPU ratio 0.959867 versus phase-off 0.864902, while plain gives 0.887289. Allocation evidence and plain timing remain separate; no sum of nested phase intervals establishes a cost breakdown. Owned sealed decoding/validation remains unchanged. Earlier borrowed-log CR2 non-nomination also remains unchanged. The supported next research question is whether avoiding validated payload construction can help more broadly, with its own integrity-preserving design and prospective guards; this rejected screen authorizes no further experiment or production default change.
Several causal explanations remain open. Absent queries never fill the heap, so the new threshold/key request rejects no rows and cannot avoid matching or cloning; the baseline requests offer keys after a successful predicate. Compiler optimization can eliminate some source-level operations, so this is a hypothesis about overhead, not a measured instruction count. Owned sealed readers construct and validate strings/attributes before callback admission, limiting avoidable work without explaining the exact adverse ratio. First/reuse boundaries can also differ in allocation, file/code cache and Walk catalog state; Scan repeats discovery and raw decoding. Separate compile gates can change inlining/layout, and one fresh pair cannot exclude host variation.
A future discriminating test would use fresh balanced ABBA repetitions of one fixed fixture, limits 1/64/2,048, and absent/matching literals at identical byte positions, with independent full-chain grading and counts of threshold readiness, key rejection, matching and cloning. Absent and unfilled heaps isolate added checks with no avoidance; filled small heaps isolate saved work. This is a research question requiring new prospective admission, not an authorized extension of this finite round.