Experiment benchmarks native key first findings - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/native-key-first-findings.md. Original SHA-256: c466ecb2f1aa734056b7ac1d54704c635b06974ccdbe15ac8cac5f6128c8f61d. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Native key-first query admission screen

The registered proposal (local checkout reference docs/experiments/benchmarks/native-key-first-proposal.md; not present at origin/main) is rejected at screen: no held-out confirmation or nomination. Screen and independent reduction both completed with exit 0. The reduction report (local evidence omitted from stage) recomputed every median from native outputs and found 57 CPU failures, 56 wall failures and 58 populations failing either 1.05 guard. Its admission receipt (local evidence omitted from stage) records decision: reject against the exact screen-result SHA256. The primary benefit passes; the unconditional adverse-population guards do not. No threshold or population was changed after measurement.

The 192 plain paired populations are 4 fixture cells × 2 layouts × 2 plans × 6 shapes × 2 first/reuse boundaries: 96 first-call pairs and 96 reuse-median pairs, derived from 768 actual plain timed calls across eight fresh children. They are not 192 independent repeated fixtures. Counted phase-on/off contribute another 384 paired populations; all three regimes together contain 2,304 timed calls. All boundaries remain separate.

Later workspace verification found Clippy-only redundant initializers in the new corruption test and benchmark example. The test repair (local checkout reference docs/experiments/benchmarks/native-verification-retry-proposal.md; not present at origin/main) and example repair (local checkout reference docs/experiments/benchmarks/native-verification-example-retry-proposal.md; not present at origin/main) remove those initializers; assertions, query implementation and frozen benchmark binaries are unchanged. Native results remain bound to their retained pre-repair source snapshots, and both failed verification receipts are preserved.

Population CPU ratio Wall ratio Registered consequence
Wide/eight attributes, tail Scan/common reuse 0.887289 0.886114 Primary benefit passes
Same fixture and query, first call 1.062809 1.065875 First-call guards fail
Wide/eight attributes, sealed Scan/common reuse 1.179948 1.179716 Diagnostic population guards fail
Narrow/no attributes, tail Scan/broad first 0.538127 0.539866 Minimum observed plain ratios
Narrow/no attributes, tail Scan/absent reuse 2.041953 2.026387 Maximum observed plain ratios

The registered primary, 1,024-byte bodies with eight attributes, tail Scan/common reused first page, has baseline/candidate CPU 7.169/6.361 ms and wall 7.208/6.387 ms: ratios 0.887289 and 0.886114. This meets the primary CPU ≤0.90 and wall ≤1.05 thresholds. Both counted regimes save 2,558,531 requested allocation bytes, with incremental peak unchanged; requested ratios are 0.927521 with phases enabled and 0.927463 with phases disabled. This supports avoided cloning/matching in that finite population, while the overall cost guard still rejects the candidate.

Adverse populations include 16-byte/no-attribute tail Scan/absent reuse, CPU/wall ratios 2.041953/2.026387, and wide/eight-attribute sealed Scan/common reuse, 1.179948/1.179716. The primary population's first call also regresses, 1.062809 CPU and 1.065875 wall, violating the separate first-call guard. An isolated good reuse median cannot override these registered failures. This screen has one seed and no fresh held-out repetition; adverse ratios are observed comparisons, not proof that every change is caused by the extra key branch. No samples or populations are pooled.

The executed screen retained 24 fresh children, 2,304 first-page calls, 576 full canonical continuation chains comprising 11,040 pages, and 1,440 rejected oracle output controls. Independent reduction regraded all 576 chains and all 1,440 actual bad-answer vectors with the unchanged Python oracle, and authenticated 4,840 decoded content objects. Its positive synthetic admission control succeeded and seven injected defects were rejected: primary CPU, another CPU population, another wall population, allocation, semantic failure, missing population and nonfinite ratio. Every child exercised six shapes, two layouts and Scan/Walk, with one first call and three subsequent calls. Full drains were outside query clocks; these measurements establish no complete-chain CPU gain. Native sealing sorts the complete key and these fixtures occupy one row group, so this is not a new pruning/locality experiment.

Native controls key-first-controls-01 ran eight test commands under flag 0/1 and borrowed flag 0, exit 0. Scan/Walk retained the exact incomplete answer when a later heap-rejected row contained malformed attributes JSON inside a valid size/schema/digest envelope: expected ident at line 1 column 2, one unavailable segment. Existing missing-table, missing-raw and raw-integrity controls remained executable. Independent reduction authenticated their receipts and retained outcomes; these checks are finite counterexamples, not exhaustive corruption coverage. Broader candidate checks key-first-regressions-01 completed with exit 0: 64 library tests passed, two ignored; ten history tests and one snapshot-transition test passed, totaling 75 passed. Correctness evidence does not override the failed performance guards.

The screen coordinator reports 545.645 seconds including builds and the whole Python/native batch, cgroup peak 507,408,384 bytes, zero swap and zero high/max/OOM events, with owned scratch removed. The harness receipt reports 109,162,496 allocated retained bytes and both exact native-state readbacks before cleanup. Source archives, exact input/answer/control objects and binary/build hashes bind the working-tree source rather than HEAD alone. Compiler: rustc 1.99.0 (b940084d7), Cargo 1.99.0, Python 3.14.7. Plain/count builds used a shared Cargo cache; their build durations are not independent clean-build comparisons.

Contained job Exit Wall seconds Cgroup peak bytes Owned scratch
Native controls receipt (local evidence omitted from stage) 0 17.891 1,893,449,728 Removed
Screen receipt (local evidence omitted from stage) 0 545.645 507,408,384 Removed
Independent reduction receipt (local evidence omitted from stage) 0 155.306 151,212,032 Removed
Candidate regression receipt (local evidence omitted from stage) 0 200.892 1,838,968,832 Removed

All four receipts report zero swap and zero high/max/OOM events. Cgroup peaks cover whole jobs, including compilation or Python grading where applicable, and are not native-query peak heap. Root disposes frozen binary copies after final authenticated readback; executable hashes and source/build evidence remain.

Observer sensitivity is material even on the primary: counted phase-on CPU ratio 0.959867 versus phase-off 0.864902, while plain gives 0.887289. Allocation evidence and plain timing remain separate; no sum of nested phase intervals establishes a cost breakdown. Owned sealed decoding/validation remains unchanged. Earlier borrowed-log CR2 non-nomination also remains unchanged. The supported next research question is whether avoiding validated payload construction can help more broadly, with its own integrity-preserving design and prospective guards; this rejected screen authorizes no further experiment or production default change.

Several causal explanations remain open. Absent queries never fill the heap, so the new threshold/key request rejects no rows and cannot avoid matching or cloning; the baseline requests offer keys after a successful predicate. Compiler optimization can eliminate some source-level operations, so this is a hypothesis about overhead, not a measured instruction count. Owned sealed readers construct and validate strings/attributes before callback admission, limiting avoidable work without explaining the exact adverse ratio. First/reuse boundaries can also differ in allocation, file/code cache and Walk catalog state; Scan repeats discovery and raw decoding. Separate compile gates can change inlining/layout, and one fresh pair cannot exclude host variation.

A future discriminating test would use fresh balanced ABBA repetitions of one fixed fixture, limits 1/64/2,048, and absent/matching literals at identical byte positions, with independent full-chain grading and counts of threshold readiness, key rejection, matching and cloning. Absent and unfilled heaps isolate added checks with no avoidance; filled small heaps isolate saved work. This is a research question requiring new prospective admission, not an authorized extension of this finite round.