Research native ownership model - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/research/native-ownership-model.md. Original SHA-256:a3d775425293c89112a639bfb31b3d2286ba69cc3ebc0296a2b361e862f426d1. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Conditional native ownership lifetime model
This research prediction was written before the new native ownership screening jobs. It uses existing historical traces; no new screen outcomes were inspected. It supplements the ownership proposal (local checkout reference docs/experiments/benchmarks/native-ownership-proposal.md; not present at origin/main) without changing its whole-build heap threshold, cost guards, protocol or oracle. The new screen uses ordinary unobserved allocation counters, baseline/candidate local8 and fresh seeds; historical phase observations use a different seed and installed observer. Predictions across those boundaries require explicit assumptions.
Competing maxima
For identical input and allocation paths, let C be the maximum requested live heap during conversion before source release; E_j the baseline maximum during write/flush while source owner j survives; D_j the actual requested bytes released from that owner; and U the maximum in other unaffected work. Use one consistent absolute or baseline-relative coordinate system throughout. The conditional model is:
P_baseline = max(C, max_j(E_j), U)
P_early = max(C, max_j(E_j - D_j), U)
The candidate cannot lower a conversion maximum reached before it clears source rows. At a baseline peak-setting write, gain cannot exceed released source bytes or the distance to a winning unaffected/conversion maximum. Retained Vec spare capacity is excluded from D_j; deep string/raw-batch payloads released by clearing are included. The nominal 4 MiB batches chunk limit is not an observation of D_j: it excludes some metadata/capacity differences and permits a single oversized entry.
Assumptions are fixed input/order, row-group boundaries and writer behavior; unchanged later allocation paths and retained state; identical counter scope; and ownership release affecting only the intended lifetime. This is a causal model of requested heap, not a reservation bound or RSS model. Exact output identity helps verify fixed work but does not prove identical allocation paths. Actual per-owner released bytes are not instrumented.
Historical observations and exact trace locations
The historical seed is 2703204353, steady shape, target 64 MiB. Each JSONL record has sample slots [unsupported CPU, LIVE, global PEAK, unsupported requested]. The locations below identify genuine new global records inside arrow_batches followed by an enclosing write/flush record. Line numbers are one-based decompressed JSONL lines; zero-based record indices are also given.
| Cap | Trace | Conversion record | Enclosing write record |
|---|---|---|---|
| 8 MiB | observed phase trace (local evidence omitted from stage) | line5023 / index5022, arrow_batches, start239218945 ns |
line5024 / index5023, bounded_parquet_encode_write, start239218783 ns |
| 16 MiB | observed phase trace (local evidence omitted from stage) | line5019 / index5018, arrow_batches, start231442538 ns |
line5020 / index5019, bounded_parquet_encode_write, start231442336 ns |
At cap8, conversion after is [0,56480404,56480404,0] and write after is [0,51631342,63625389,0]. At cap16, conversion after is [0,68349227,68349227,0] and write after is [0,63500165,75494212,0]. The writer interval encloses conversion and writer.flush; these are not exclusive phase allocations or additive timings.
The historical paired summary (local evidence omitted from stage) records:
| Cap | Observer-inclusive heap start | Observed whole-build peak | Observed incremental peak | Witnessed conversion floor, incremental |
|---|---|---|---|---|
| 8 MiB | 29,362,009 | 63,625,389 | 34,263,380 | 27,118,395 |
| 16 MiB | 29,362,017 | 75,494,212 | 46,132,195 | 38,987,210 |
The observer's installed requested live baseline is 29,360,128 bytes (28 MiB). The witnessed conversion floors are conversion.after.PEAK - observed.heap_start. Both final encoding/flush peaks lie exactly 7,144,985 bytes above those floors. These are witnessed floors, not identified maxima of all conversions: an earlier global record can mask a later local conversion maximum.
Percentage denominators and prospective threshold
The maximum possible reduction conditional on those unchanged witnessed floors is 20.8531% of incremental heap at cap8 and 15.4881% of incremental heap at cap16. Those figures must not be labeled observer-inclusive whole-heap reductions. Adding the recorded heap start back to the denominator gives maximum reductions of 11.2298% and 9.4643%, respectively, of the actual observer-inclusive whole-build requested peaks. The cap16 observed case therefore cannot meet a 10% observer-inclusive whole-heap threshold under this model. That is not the new experiment's cap or measurement configuration.
The new registered threshold is ≥10% of ordinary whole-build peak_live_heap_bytes, not incremental heap or an observer-adjusted RSS. Historical plain heap starts are1,857/1,865 bytes and whole peaks34,265,213/46,134,036 bytes. Their increments are24 bytes below the observed increments in both cases; transfer of the observed conversion floor to the plain probe is an assumption, not an independently measured phase attribution. If that transfer and the other fixed-path assumptions hold, the ordinary whole-heap ceiling is approximately20.85%/15.49%; adding the small plain heap baseline changes the ratio slightly. No precise prediction of the fresh-seed whole peak is asserted from the observer-subtracted data.
For the actual historical plain cap8 whole peak, a 10% gain would require at least 3,426,521.3 bytes of reduction and a candidate whole peak ≤30,838,691.7 bytes. The release must remove at least that many relevant peak-setting owner bytes, and every conversion/unaffected peak must remain below that candidate bound. For each new pair these conditions must instead use its own measured baseline whole peak; no historical denominator may substitute. No observer baseline is subtracted from RSS.
Observable counterexamples and identifiability
A near-zero whole-heap gain is consistent with conversion setting the maximum before release, a small releasable owner at the relevant peak, or another unaffected phase winning. A gain near a witnessed conversion floor is consistent with a switch to conversion dominance, but does not prove that switch. A candidate remaining above the floor is consistent with residual writer/staging state, without identifying which table owns it. Different hidden phase/owner assignments can produce the same scalar P_early.
Therefore the existing whole/live/incremental/after-build metrics and byte/IO checks can falsify the prospective useful-gain claim, but cannot uniquely distinguish conversion dominance from residual concurrent staging. The previous phase trace locates a family of peak-setting intervals; it does not identify every local high-water or per-table occupancy. Shared-budget superiority cannot be inferred from an ownership null. A later attribution question would need separately scoped evidence, not a retrospective threshold change or a post-hoc refit of this conditional model.