Experiment benchmarks native ownership findings - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/benchmarks/native-ownership-findings.md. Original SHA-256:e6210719cbea17c39570006251beedc7359dc7d989cb79e877de9e537819f602. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Native early source-row release findings
The ownership prototype reduced requested build heap with exact output identity, but failed full held-out confirmation: adversarial seed 2703204360 exceeded the prospective whole-native RSS guard. This failed cost result remains part of the evidence; it is not a semantic defect or a confirmed production nomination. Defaults remain unchanged.
The registered proposal (local checkout reference docs/experiments/benchmarks/native-ownership-proposal.md; not present at origin/main) compared baseline and compile-opt-in early release at the same8 MiB signal run cap, target64 MiB, three shapes and four seeds. All twelve paired semantic checks (24 native cells) succeeded, including complete regular-member path/size/SHA identity, manifest bytes, raw custody, logical ordered ledgers, independent physical FTF1 reconstruction and no leftover scratch. Baseline/candidate unit jobs also exercised both lifetime policies, conversion/write errors and existing byte/custody/cleanup controls. Logical ledgers remain differential Rust evidence; the physical checker is independent. The pre-screen conditional model remains unchanged.
Observed benefits and adverse confirmation
| Stage / seed | Shape | Candidate/baseline heap | Throughput | Whole CPU | Whole RSS | Cost screen |
|---|---|---|---|---|---|---|
| Screen4357 | steady | 0.878265 | 0.978688 | 1.019741 | 1.011410 | true |
| Screen4357 | adversarial | 0.878343 | 0.960778 | 1.022738 | 1.004856 | true |
| Screen4357 | bigrows | 0.827502 | 1.009510 | 0.986128 | 1.003824 | true |
| Screen4358 | steady | 0.878266 | 0.986557 | 1.003621 | 1.010659 | true |
| Screen4358 | adversarial | 0.878345 | 0.954453 | 1.030060 | 0.944376 | true |
| Screen4358 | bigrows | 0.827512 | 1.030878 | 0.979175 | 0.996505 | true |
| Confirm4359 | steady | 0.878268 | 0.950568 | 1.024927 | 0.935102 | true |
| Confirm4359 | adversarial | 0.878343 | 0.983832 | 1.006376 | 0.987542 | true |
| Confirm4359 | bigrows | 0.827508 | 0.972922 | 0.998130 | 0.970793 | true |
| Confirm4360 | steady | 0.878266 | 0.958714 | 1.002029 | 0.994234 | true |
| Confirm4360 | adversarial | 0.878345 | 1.012951 | 1.000723 | 1.151341 | false: RSS |
| Confirm4360 | bigrows | 0.827505 | 0.961074 | 1.012222 | 0.973261 | true |
Full seeds are2703204357 through2703204360. Source summaries: screen0 (local evidence omitted from stage), screen1 (local evidence omitted from stage), confirmation0 (local evidence omitted from stage), confirmation1 (local evidence omitted from stage). Every build-write ratio was exactly1.0 and every regular output byte agreed; there is no IO benefit claim. All throughput and CPU guards held. Steady/adversarial heap fell about12.17%; bigrows about17.25%. Each steady/adversarial whole-heap difference was exactly4,171,181 bytes; requested retained heap after build differed by only8 bytes. The effect is consistent with transient ownership lifetime rather than retained-state reduction. Whole RSS did not reliably improve and cannot be inferred from requested heap.
Preserved RSS counterexample
The failed cost pair (local evidence omitted from stage) executed baseline then candidate. Exact whole-native wait4 HWM was87,828→101,120 KiB (+15.1341%), above the registered110% limit. Requested build peak was34,286,878→30,115,697 bytes; throughput ratio1.012951, CPU1.000723, writes1.0. Its deterministic input SHA is c509bb4b79c262c931cb551e3e6912bd35fe94392c8e6323c3b9680322c6c210. Both arms' complete member maps, exact comparison gates and raw rows/process receipts are retained. Passing semantic fixtures were removed under the prospective cleanup rule; recreating the same seed must authenticate that input and member map before a new diagnostic can be compared. Historical failed confirmation is never rewritten.
The baseline timeline (local evidence omitted from stage) and candidate timeline (local evidence omitted from stage) constrain the explanation:
- Parent pre-spawn HWM was30,908 KiB in both arms and PyArrow was absent. Worker HWM substantially exceeded it. This discounts inherited-parent residency as the cause of the ratio.
- During generation, sampled RSS stayed about4,368/4,132 KiB, respectively, through approximately5.1 seconds. Candidate generation residency was not higher.
- Build-end
/proc/self/statcaptured 7,768/11,072 resident pages and virtual sizes 37,408,768/54,804,480 bytes. Read-onlygetconf PAGESIZEreports 4,096-byte pages, giving 31,072/44,288 KiB resident: a roughly 13 MiB gap already exists at build return, despite nearly identical tiny requested retained heap. These are native probe endpoints; they are not the parent-clock timeline samples or wait4 peaks. - After publication, baseline sample6.910 seconds reported57,168 KiB RSS; candidate6.827 seconds reported70,428 KiB. During later verification/scanning, baseline7.840 seconds reported88,212 KiB; candidate7.760 seconds101,420 KiB. The approximate gap persists as post-build work increases both residents.
These source-labeled boundaries support allocation-layout/allocator or foreign-runtime residency as a mechanism family, amplified by post-build reads. They do not identify glibc internals, a particular owner or a leak. The observer covers Rust requested allocations and excludes foreign allocation ownership. Sampled VmHWM sometimes decreases slightly or exceeds final wait4 HWM; samples must not be treated as a perfectly monotonic/exact peak ledger. Final wait4 remains the registered cost metric. Phase times are not exclusive ownership.
Meaning and follow-up
The candidate delivered a consistent finite requested-heap benefit with identical files, yet failed the complete prospective cost criterion on one held-out pair. The separately registered diagnostic below further constrains the RSS mechanism while retaining that nonconfirmation. No threshold widening, selective seed replacement, source change, production promotion or shared-budget inference follows from the current results.
Executed allocator-policy contrast
The prospective RSS diagnostic (local checkout reference docs/experiments/benchmarks/native-ownership-rss-diagnostic-proposal.md; not present at origin/main), registered separately before execution, completed four new pairs / eight native cells on the exact frozen binaries, adversarial shape, target64 MiB and the preserved seed2703204360. Each pair had a fresh Python parent. Two default-policy pairs preceded two fixed-policy pairs; order was balanced within each policy. Only native children received MALLOC_MMAP_THRESHOLD_=131072; decoder and parents retained their ordinary environment. Recorded inherited allocator/loader overrides were empty, libc was2.43 and page size4096 bytes.
All four pairs retained the original input SHA and exact original regular-member maps, with unchanged semantic/physical gates. The diagnostic summary (local evidence omitted from stage) reports:
| Native allocator policy | Arm order | Build-return resident gap, candidate−baseline bytes | Whole-native RSS ratio | Requested heap ratio | Whole CPU ratio | Build-write ratio |
|---|---|---|---|---|---|---|
| Default | baseline → candidate | +13,750,272 | 1.152811 | 0.878345 | 0.999789 | 1.0 |
| Default | candidate → baseline | +13,631,488 | 1.150141 | 0.878345 | 0.997304 | 1.0 |
| Fixed128 KiB mmap threshold | baseline → candidate | −393,216 | 1.013461 | 0.878345 | 1.011131 | 1.0 |
| Fixed128 KiB mmap threshold | candidate → baseline | −884,736 | 1.005532 | 0.878345 | 1.007052 | 1.0 |
Both new default-policy pairs repeat the >110% whole-RSS excess, in opposite orders. Both default build-return gaps are positive, while each fixed-policy absolute gap is less than half the smaller default gap, satisfying the prospective primary contrast rule. The early-release requested-heap ratio is unchanged across policies; every write ratio and every output byte remains identical. Throughput ratios range0.991243–1.022172. These are descriptive diagnostic costs, not a replacement confirmation gate.
The default native build-end virtual sizes repeat37,408,768/54,804,480 bytes (baseline/candidate) in both orders. With fixed threshold they are31,059,968/30,445,568 bytes. Native stat-after resident pages, sampled parent-clock VmRSS/VmHWM and final wait4 HWM remain different measurement domains; the primary gap uses the native stat-after endpoints and actual4096-byte pages. The policy contrast supports allocator-policy-sensitive build-end residency, with a subsequent whole-worker RSS consequence. It discounts simple arm-order noise as the sole explanation in these repetitions.
The GNU libc manual documents that explicitly setting the mmap threshold disables its dynamic adjustment. A next hypothesis is allocation-migration feedback: changed source-release timing changes which allocations use normal heap versus mmap, leaving different resident allocator state for later verification. The contrast does not establish that specific trajectory, identify an allocation owner, measure glibc's actual threshold, or exclude related allocator/trim adjustments. Both arms' allocation placement changes under the fixed setting, so this is not a production tuning nomination or a source-only remedy. The original default-environment ownership confirmation remains failed.
The negative-control receipt (local evidence omitted from stage) records rejection of changed/missing/extra member maps and fixture/row/custody/manifest/filter defects, plus counter-reset rejection, missing-phase handling, equality ambiguity and truncated/space-containing process-stat controls. The coordinator receipt (local evidence omitted from stage) records the exact command, child/outer exit0,278.928 seconds,615,301,120-byte cgroup peak (586.80 MiB), zero OOM events, zero final swap usage and owned scratch removal under enforced no-swap containment. Each paired state was removed only after original-map authentication and gates. The decoder cleanup receipt (local evidence omitted from stage) records PyArrow22 install exit0, package metadata hashes and removal. No original failed receipt or frozen source was rewritten.
Deferred trajectory: consume source rows during conversion
A future consuming conversion could drain source rows, append each owned payload into Arrow and release that row immediately, preserving source Vec capacity. This targets conversion overlap C in the pre-screen model, while the implemented candidate targets the later write/flush overlap E. Raw batches are the smallest candidate pilot; typed attributes and error paths require broader invariants. Filters must complete before consuming rows. A conversion failure after partial drain must abort the entire journal-preserving build, without publishing a manifest or silently returning partial rows.
Preallocation is an adversary to this mechanism. Reserving the entire destination payload capacity N while all N source bytes still exist can establish approximately 2N requested live bytes immediately, before draining has released anything. Pre-sizing row offsets/scalars differs from preallocating payload bytes; incremental payload growth can avoid that particular initial overlap but adds capacity and reallocation costs. Therefore drain plus pre-sized payload builders is not a deduced heap improvement. Construction/growth strategy, exact output bytes and whole-native RSS need a separately registered comparison. No consuming conversion, additional allocation policy or third prototype is implemented or nominated in this round.