Research native ownership model - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/research/native-ownership-model.md. Original SHA-256: a3d775425293c89112a639bfb31b3d2286ba69cc3ebc0296a2b361e862f426d1. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Conditional native ownership lifetime model

This research prediction was written before the new native ownership screening jobs. It uses existing historical traces; no new screen outcomes were inspected. It supplements the ownership proposal (local checkout reference docs/experiments/benchmarks/native-ownership-proposal.md; not present at origin/main) without changing its whole-build heap threshold, cost guards, protocol or oracle. The new screen uses ordinary unobserved allocation counters, baseline/candidate local8 and fresh seeds; historical phase observations use a different seed and installed observer. Predictions across those boundaries require explicit assumptions.

Competing maxima

For identical input and allocation paths, let C be the maximum requested live heap during conversion before source release; E_j the baseline maximum during write/flush while source owner j survives; D_j the actual requested bytes released from that owner; and U the maximum in other unaffected work. Use one consistent absolute or baseline-relative coordinate system throughout. The conditional model is:

P_baseline = max(C, max_j(E_j), U)

P_early = max(C, max_j(E_j - D_j), U)

The candidate cannot lower a conversion maximum reached before it clears source rows. At a baseline peak-setting write, gain cannot exceed released source bytes or the distance to a winning unaffected/conversion maximum. Retained Vec spare capacity is excluded from D_j; deep string/raw-batch payloads released by clearing are included. The nominal 4 MiB batches chunk limit is not an observation of D_j: it excludes some metadata/capacity differences and permits a single oversized entry.

Assumptions are fixed input/order, row-group boundaries and writer behavior; unchanged later allocation paths and retained state; identical counter scope; and ownership release affecting only the intended lifetime. This is a causal model of requested heap, not a reservation bound or RSS model. Exact output identity helps verify fixed work but does not prove identical allocation paths. Actual per-owner released bytes are not instrumented.

Historical observations and exact trace locations

The historical seed is 2703204353, steady shape, target 64 MiB. Each JSONL record has sample slots [unsupported CPU, LIVE, global PEAK, unsupported requested]. The locations below identify genuine new global records inside arrow_batches followed by an enclosing write/flush record. Line numbers are one-based decompressed JSONL lines; zero-based record indices are also given.

Cap Trace Conversion record Enclosing write record
8 MiB observed phase trace (local evidence omitted from stage) line5023 / index5022, arrow_batches, start239218945 ns line5024 / index5023, bounded_parquet_encode_write, start239218783 ns
16 MiB observed phase trace (local evidence omitted from stage) line5019 / index5018, arrow_batches, start231442538 ns line5020 / index5019, bounded_parquet_encode_write, start231442336 ns

At cap8, conversion after is [0,56480404,56480404,0] and write after is [0,51631342,63625389,0]. At cap16, conversion after is [0,68349227,68349227,0] and write after is [0,63500165,75494212,0]. The writer interval encloses conversion and writer.flush; these are not exclusive phase allocations or additive timings.

The historical paired summary (local evidence omitted from stage) records:

Cap Observer-inclusive heap start Observed whole-build peak Observed incremental peak Witnessed conversion floor, incremental
8 MiB 29,362,009 63,625,389 34,263,380 27,118,395
16 MiB 29,362,017 75,494,212 46,132,195 38,987,210

The observer's installed requested live baseline is 29,360,128 bytes (28 MiB). The witnessed conversion floors are conversion.after.PEAK - observed.heap_start. Both final encoding/flush peaks lie exactly 7,144,985 bytes above those floors. These are witnessed floors, not identified maxima of all conversions: an earlier global record can mask a later local conversion maximum.

Percentage denominators and prospective threshold

The maximum possible reduction conditional on those unchanged witnessed floors is 20.8531% of incremental heap at cap8 and 15.4881% of incremental heap at cap16. Those figures must not be labeled observer-inclusive whole-heap reductions. Adding the recorded heap start back to the denominator gives maximum reductions of 11.2298% and 9.4643%, respectively, of the actual observer-inclusive whole-build requested peaks. The cap16 observed case therefore cannot meet a 10% observer-inclusive whole-heap threshold under this model. That is not the new experiment's cap or measurement configuration.

The new registered threshold is ≥10% of ordinary whole-build peak_live_heap_bytes, not incremental heap or an observer-adjusted RSS. Historical plain heap starts are1,857/1,865 bytes and whole peaks34,265,213/46,134,036 bytes. Their increments are24 bytes below the observed increments in both cases; transfer of the observed conversion floor to the plain probe is an assumption, not an independently measured phase attribution. If that transfer and the other fixed-path assumptions hold, the ordinary whole-heap ceiling is approximately20.85%/15.49%; adding the small plain heap baseline changes the ratio slightly. No precise prediction of the fresh-seed whole peak is asserted from the observer-subtracted data.

For the actual historical plain cap8 whole peak, a 10% gain would require at least 3,426,521.3 bytes of reduction and a candidate whole peak ≤30,838,691.7 bytes. The release must remove at least that many relevant peak-setting owner bytes, and every conversion/unaffected peak must remain below that candidate bound. For each new pair these conditions must instead use its own measured baseline whole peak; no historical denominator may substitute. No observer baseline is subtracted from RSS.

Observable counterexamples and identifiability

A near-zero whole-heap gain is consistent with conversion setting the maximum before release, a small releasable owner at the relevant peak, or another unaffected phase winning. A gain near a witnessed conversion floor is consistent with a switch to conversion dominance, but does not prove that switch. A candidate remaining above the floor is consistent with residual writer/staging state, without identifying which table owns it. Different hidden phase/owner assignments can produce the same scalar P_early.

Therefore the existing whole/live/incremental/after-build metrics and byte/IO checks can falsify the prospective useful-gain claim, but cannot uniquely distinguish conversion dominance from residual concurrent staging. The previous phase trace locates a family of peak-setting intervals; it does not identify every local high-water or per-table occupancy. Shared-budget superiority cannot be inferred from an ownership null. A later attribution question would need separately scoped evidence, not a retrospective threshold change or a post-hoc refit of this conditional model.