Historical report imported 2026-10-10 from docs/research/cross-system-sweep-vectors.md. Original SHA-256: b55fa07c952787d661f6f29b29030a7d322d217d571ec6710cfc1d65b90d75a4. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Fabric research vectors from matched workloads

These are research models and candidate experiments, not production algorithms or novelty claims. The sweep protocol (local checkout reference docs/experiments/benchmarks/cross-system-sweep-protocol.md; not present at origin/main) keeps correctness, custody, exact query semantics and resource bounds as hard constraints. Source mechanisms become useful only after their costs and failure regimes survive a matched experiment. Do not multiply gains from unrelated cells.

The subsequent native mechanism round tested earlier source release and key-first query admission. Both showed their predicted local benefit but failed full cost nomination. The storage diagnostic adds a previously hidden dimension: allocator state depends on the allocation/free sequence, so equal retained live bytes need not imply equal resident memory. Default-policy RSS excess repeated on the exact failed input; fixing the mmap threshold sharply reduced that excess in both arm orders. This supports policy sensitivity without establishing a production allocator setting or a unique owner.

Memory is a simultaneous lifetime problem

The frozen small-input models predict peak from a constant per-cap value, or base + gamma * run_cap_bytes. Their larger-input holdout tests that strong assumption without refitting. A more expressive candidate is

M_peak = max_t [M_log_run(t) + M_metric_run(t) + M_span_run(t) + M_batch_writer(t) + M_gap_writer(t) + M_decode(t) + M_sort_workspace(t) + M_output(t) + M_metadata(t)].

Each term is a hypothesized owner or stage, not a measured decomposition. The maximum of a sum is not the sum of independently observed maxima. The existing source closes all resident sort runs before merging them into output tables; an attribution experiment must respect that impossible-overlap constraint. Requested live heap, allocator residency, RSS and charged cgroup memory remain different domains.

Both frozen predictions were falsified: every 64 MiB holdout exceeded the 5% error threshold, with maximum underprediction about 23%. In the separately instrumented steady-input cells, the new peak occurred during ingestion's batch write, before final merges. Additional live background explained nearly all the larger-input peak shift. The observer reserved 28 MiB; incremental peaks differed from plain by 24 bytes, while RSS remains unadjusted. These observations narrow the mechanism to co-resident ingestion state without identifying each owner.

One remaining candidate is a shared staging budget across signal types, with byte and row bounds and a deterministic spill choice. It competes with independent per-table limits and simple smaller run caps. A policy must charge transient sort/decode/write work and the largest admitted operation, not merely the bytes retained between operations. Its null is extra spill/merge work or ACK interference that erases the memory benefit. This is motivated by Comet's ownership/accounting lessons and Feldera's consumed-work accounting; neither source supplies a ready-made bound for Fabric.

The smaller ownership-only prototype was tested first. A future consuming converter targets the overlap during conversion itself, but destination payload preallocation can recreate the full source-plus-destination peak before any rows are drained. Compare growth strategies, requested allocation and RSS together. Keep the pre-screen conditional prediction separate from this post-measurement explanation; its assumptions were never a process-RSS bound. New resident-memory predictions need allocator-history inputs and held-out tests rather than fitting a constant overhead to one failed pair.

Query choice depends on lifetime and physical locality

The query vector matrix records the measured pruning, representation and ingestion tradeoffs. Model one candidate as

C_lifetime = C_build + C_publication + C_maintenance + sum(C_lookup + C_candidate_decode + C_exact_verify + C_output).

An optional accelerator repays construction only if repeated query savings are positive and occur before its data expires. A selective row predicate need not be selective at block level. Match density, clustering, group size, payload width and query reuse are separate dimensions. The exact-set Python index in the screen explains this mechanism; its construction cost is not a production index estimate.

For candidate row sets, a first memory model is M_sorted ≈ 4k bytes for k u32 identifiers and M_dense ≈ N/8 for a universe of N rows. Their idealized break-even is k/N ≈ 1/32, before capacity, headers and output temporaries. Roaring adds container/run structure, making clustering and range count relevant as well as cardinality. The native screen must include construction, combination, serialization, Top64 and conversion; an already-built intersection's Top64 time is not a fused query. A production RowSet must remain bound to its immutable source generation, canonical ordering and exact validation obligations.

A storage-owned candidate plan could choose a scan, group summary, sorted IDs, dense bitmap or compressed containers using those statistics. That is a research interface concept, not an authorization to add a generic storage abstraction. Its null is that conversion, ownership and validation dominate the saved lookup work. A new native Fabric experiment must test full continuations and corruption controls before any replacement is considered.

The held-density followup keeps logical A/B sizes and intersection fixed while changing the bijection from logical to physical row IDs. It removes the original screen's density confound. At 262,144 rows and density 1/16, shuffled/contiguous Roaring AND time is 400–667 times larger and retained input-pair storage about 179–183 times larger across two seeds. These are instrumented repeated kernel ratios; they are not query-service speedups. Dense combinations remain comparatively stable.

A better candidate state descriptor is (N, k, run_count, occupied_containers, operations_remaining, source_generation). Predict each representation's cost as C_construct + sum(C_operation + C_conversion), subject to peak memory and source-identity constraints. Cardinality-only selection is the competing simpler model. A fresh cross-validation grid must include intermediate locality and conversion before deciding whether the larger descriptor earns its overhead. Do not infer a switching threshold from these two extreme layouts alone.

Admission and maintenance consume the same service capacity

The cadence screen measures actual append groups and exact accepted Batches. Use sync_cost_per_batch = observed_sync_calls / accepted_batches alongside ACK quantiles, actual arrival intervals, lateness and outstanding bytes. Offered cadence does not directly equal arrival timing at the commit thread. Grouped ACKs are correlated; 512 acknowledgments are not 512 independent disk failures.

A future controller could choose a bounded waiting policy using observed queue age and arrival variation, while keeping custody and durability ordering fixed. The present adjacent-cadence experiment does not change the quiet timer, so it cannot identify the timer's causal contribution. The competing explanation is producer scheduling, IO or observer work. Compare journal byte-time relief and ACK tails together before combining such a controller with oldest-prefix maintenance. Vector's backpressure and Fjall's oldest-obstruction mechanisms motivate the question; neither implies that more concurrency is beneficial.

The fresh boundary experiment records nine append groups for 512 Batches at 1 ms offers, versus 272 at 3 ms, in both markers. After-sealing median ACK latency moves from roughly 35.5 to 6.3 ms. This supports explicitly modeling C_sync * groups/accepted_batch together with ACK age, without treating target cadence as commit-thread arrival timing. Event/quiet median agreement does not erase the retained event-arm p99 outlier. Resource observation has its own cost: paced polling raised median latency in all twelve original matched pairs.

Candidate ledger

Vector	Measured discriminator	Candidate and null	Next independent boundary
Smaller sort runs	8 MiB caps save 20–27% peak requested heap against 16 MiB, confirmed on two fresh seeds with unchanged cost guards; 32 MiB loses its screen	Preserve smaller buffers if service guards hold; null is extra spilling or service contention	Concurrent service pressure before a default change
Earlier source release	12–17% requested-heap reduction; held-out RSS guard fails and excess repeats under default policy	Alter ownership overlap; null/adversary is allocator residency or an unaffected peak	Consume rows during conversion with growth-policy contrasts and journal-preserving errors
Key-first admission	Primary CPU improves 11.3% with 2.44 MiB fewer requested allocations; 58 of 192 plain comparisons fail a timing guard	Avoid matching/cloning when a row cannot enter the page; null is unfilled heaps or mandatory decoding	Balanced repeated churn/limit contrasts, key-rejection work counts and unchanged corruption checks
Shared signal staging	Flat-input peak models fail; added background is present at ingestion batch writes	Share a byte/row budget and reserve writer workspace; null is premature spills and accounting overhead	Tag live allocation owners; generate skewed metric/span/log occupancy and maximum admitted rows
Optional group pruning	Clustered selective groups can avoid 31/32 groups; mixed large groups avoid none	Pay only for useful candidate avoidance; null is saturated summaries or mandatory validation work	Native full query continuations, false-negative controls, exact corruption behavior and build cost
Selected payload work	Vortex predicates move cost from downstream conversion/filtering into scan	Delay costly representation movement until candidates are known; null is integrity checking or ownership requiring the same work	Native borrowed/owned paths with identical typed validation and complete outputs
Representation choice	Equal-density physical layouts strongly change Roaring, much less dense bitmaps	Select using locality and operation mix; null is conversion/dispatch cost exceeding savings	Intermediate locality, query-shaped operation chains and immutable-generation binding
Prepared plan reuse	Preparation 0.020–0.201 ms; savings reverse across some repetitions	Reuse only if measured lifetime savings are positive; null is shared-state overhead or unchanged execution cost	Isolate planning, bound retained plans, test changed parameters and cancellation
Admission grouping	Sync groups and ACK age trade off across fresh cadence markers	Bounded age-aware grouping with unchanged durability order; null is scheduling noise or worse tails	Real arrival timestamps, equal offered demand, seeded scheduling and overload/recovery
Observation cost	Polling perturbs paced ACK medians and CPU	Prefer bounded event recording and explicit observer ablations; null is recorder overhead or missing coverage	Compare quiet, event and sampled modes at equal workload and retain tail counterexamples

These research vectors do not authorize production mechanisms. The existing two-prototype ceiling still applies. Alternative format experiments currently provide adverse full-scan and footprint evidence, so a wholesale format replacement is lower priority than candidate avoidance and lifetime accounting. Fresh confirmations and full service constraints decide nomination.

Selection and confirmation

Keep a candidate when it has a repeatable mechanism benefit, explicit adverse regimes, an independently checked semantic boundary and a useful deployment constraint it might relieve. Prefer the next experiment that separates competing explanations, rather than adding another knob to a fitted curve. New model parameters need held-out workloads; two descriptive repetitions do not supply a calibrated probabilistic capacity guarantee. No engine migration, automatic controller or production default is selected by this record.