Historical report imported 2026-10-10 from docs/experiments/benchmarks/cross-system-rowset-sweep-findings.md. Original SHA-256: 189865e9b68026a32bf37d7d74bc2a37a75b375f041ba24b4685f9882abb8c33. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Finite RowSet representation findings

The initial representation screen completed 48 native cells: sorted Vec, dense bitset and actual upstream Rust Roaring at universes 32,768/262,144, four generated shapes and two seeds. Exact Python expected AND/OR membership and descending Top64 checks succeeded. The representation report (local evidence omitted from stage) separates construction, operations, serialization, canonical conversion, allocation counters and native HWM. A separate 48-cell held-density locality screen (local evidence omitted from stage) and its paired report (local evidence omitted from stage) now compare contiguous and shuffled bijections of the same logical sets. Both screens use the pinned upstream proposal (local checkout reference docs/experiments/benchmarks/cross-system-rowset-sweep-proposal.md; not present at origin/main), revision 0439d576fb3d36acba4444f5a5bc0d4173ecd932, and exact dependencies. These are standalone mechanism screens, not Fabric production qualification.

Representation and lifetime interact

Dense set operations favored common/high-density, clustered and spread inputs. Across both seeds, 262,144 common AND/OR batches took approximately 0.0015–0.0017 of Vec time, clustered about 0.019–0.035, and spread about 0.0066–0.0104. Construction cost more than copying an already sorted Vec. Dense sparse input paid for the entire universe: its two inputs occupied 65,536 requested bytes versus roughly 2 KiB of Vec, and its native serialization was 32–34 times Vec size. Sparse timings varied markedly between repetitions. Dense is not uniformly favorable.

Roaring's distinctive advantage was clustered/run structure. At universe 262,144, both seeds retained only 832 requested bytes for the constructed pair and serialized it in 602 bytes, against Vec's 262,144 bytes. Batched AND/OR ratios were about 0.016–0.031, but construction was 7.7–12.0 times Vec. The synthetic construction + AND + Top64 sum was 2.90–3.79 times Vec for one operation, versus 0.091–0.114 for 128 operations. These sums combine separately measured phases; they are not measured continuous pipelines.

Spread input counters global Roaring superiority. At universe 32,768, Roaring AND batches cost 1.57–1.71 times Vec and OR 2.32–2.98 times, despite roughly half the serialized size. At 262,144, batched operations improved to about half Vec time; construction still cost 13.1–37.4 times Vec. The synthetic one-operation sum remained 1.42–2.20 times Vec, while 128-operation sums were 0.508–0.557. Cardinality, locality and reuse matter jointly.

Sparse Vec remains a candidate for short-lived sets. At universe 262,144, Roaring's single-operation sum was 1.69–1.93 times Vec. Even its 128-operation sum ranged from 0.913 to 1.668 across seeds, giving no consistent sparse nomination. Dense sums were worse in both repetitions. Preserve this counterexample in any representation-dispatch study.

Held-density arrangement control

The earlier clustered-versus-spread screen varied density as well as arrangement: clustered inputs contained 1/8 of the universe, while spread inputs contained about 1/16. That contrast cannot attribute their differences to layout alone. The follow-up fixed the logical pair at each universe/density/seed: A=[0,k), B=[k/2,3k/2), so both contain k, AND contains k/2, and OR contains 3k/2. It mapped both sets through the same cyclic shift or deterministic shuffle, retaining and checking exact inputs and mapping hashes. The successful reducer authenticated all 48 cells and their archived outputs.

The table gives shuffled/contiguous ratios. AND and OR columns are ratios of the 128-operation batch wall times, so each value is dimensionless (×), not a duration in nanoseconds. Byte columns compare the constructed input-pair retained bytes and native serialized input-pair bytes. Each range spans both seeds, 2,703,204,355 and 2,703,204,356; exact cardinalities and overlap match within every pair.

Universe	Density	Representation	AND batch ratio	OR batch ratio	Input-pair bytes ratio	Serialized-pair bytes ratio
32,768	1/16	Vec	3.52–5.62×	2.03–3.86×	1.00×	1.00×
32,768	1/16	Dense	0.52–1.00×	0.998–1.001×	1.00×	1.00×
32,768	1/16	Roaring	57.62–104.32×	206.70–217.36×	24.18×	274.13×
32,768	1/2	Vec	8.82–8.83×	6.45–6.49×	1.00×	1.00×
32,768	1/2	Dense	0.57–0.69×	0.79–1.00×	1.00×	1.00×
32,768	1/2	Roaring	4.90–6.90×	9.23–10.16×	47.45×	482.82×
262,144	1/16	Vec	8.40–8.78×	5.40–5.82×	1.00×	1.00×
262,144	1/16	Dense	1.00–1.09×	0.76–1.06×	1.00×	1.00×
262,144	1/16	Roaring	399.58–667.22×	474.39–977.64×	178.96–182.93×	1,632.30–1,636.40×
262,144	1/2	Vec	6.18–7.03×	4.57–4.60×	1.00×	1.00×
262,144	1/2	Dense	0.84–0.87×	0.79–0.97×	1.00×	1.00×
262,144	1/2	Roaring	17.43–19.21×	18.99–19.05×	158.31×	937.37×

For Roaring, shuffled/contiguous AND/OR batch cost rises in both seeds at both tested densities and universe sizes, alongside much larger retained and serialized pairs. The same exact cardinality is therefore insufficient to predict Roaring's representation cost in this synthetic index domain. Vec batch times also rise under the shuffled mapping, while dense bitwise batch ratios stay near 1×; those operation sensitivities differ by representation. These are measured effects of the registered cyclic-versus-shuffled ID mappings, not evidence that real Fabric records have either layout or that a production dispatch should use an inferred locality label.

This control has only two seeds, two densities, two universes, and the specified logical overlap; four paired cases per universe/density/representation. It uses generated IDs, not Fabric payloads, row order, storage, or query planning. Operation batches reuse already constructed inputs and include the registered allocator instrumentation. Top64 still measures an already materialized intersection, not fused AND/top64. The report imposes no performance threshold and makes no global winner claim.

Top64 and conversion are separate costs

The locality followup orders Top64 by physical ID. Its permutation can therefore change which logical identities are returned in that subset, while exact full AND/OR membership remains equivalent. It is not an identical logical Top-k query under two layouts; canonical query-order costs need a separate native test.

Top64 operates on an already materialized intersection. Dense and Roaring were slower than Vec in all observed shapes/seeds. At universe 262,144 common input, Roaring batched Top64 cost 26.2–41.4 times Vec; dense cost 2.52–2.64 times. A cheap set operation does not establish a cheap complete query. A fused intersection/early-stop Top64 iterator was not tested.

Serialization bytes, time and conversion also disagree. In the first 262,144 clustered repetition, Roaring serialized in about 0.025 of Vec time, yet canonical AND/OR conversion cost 19.25/17.44 times Vec. For common input, wire size was about 4.18% of Vec, while full conversion cost about 23–25 times. Sparse Roaring produced roughly half Vec wire bytes but took about twice Vec serialization time in that repetition. Native wire roundtrip and independent canonical membership checks ran; no independent Roaring-wire decoder is claimed.

Requested retained bytes describe the representation more directly than whole-process residency. Native post-exec HWM is recorded; the first 262,144 common repetition used 7,376 KiB for Roaring versus 6,924 KiB for dense, including runtime, source vectors, outputs and validation. Wait4 RSS remains uninterpreted because parent residency can contaminate it. Allocation instrumentation is present in all arms without an uninstrumented control.

Source failure and research candidate

The locality report's later preservation receipt (local evidence omitted from stage) resolves its historical cleanup flag: the launcher had relocated the empty tree, which was archived with two exact readbacks before removal. The failed report itself and the initially rejected archive reservation remain recorded.

Run 01 failed before compilation when the conservative archive checker rejected two upstream license symlinks. Its archive was retained/authenticated and scratch removed; this was not an algorithm failure. The separately registered retry (local checkout reference docs/experiments/benchmarks/cross-system-rowset-license-retry-proposal.md; not present at origin/main) permits only the two exact aliases, records their regular root-license targets/hashes, and never extracts links. Wrong path/target, absolute target, wrong package and absent-target controls reject; other special entries remain forbidden. Retry evidence does not rewrite run 01.

The held-density native run completed all 48 cells with exit 0 in 16.063 seconds; its receipt records scratch removed, a 211,951,616-byte cgroup peak, zero memory high/max/OOM events, and zero swap under the configured 16 GiB high / 20 GiB maximum / no-swap limits. The first locality report attempt, report-01 receipt (local evidence omitted from stage), exited 1 because the reducer rejected the runner's legitimate output/ archive directory and its two retained .native serialization files. This was a reducer archive-shape defect after the native cells, not a cell/oracle failure. That receipt records scratch_removed: false; its recorded scratch path was absent on later read-only inspection. The failed report attempt is preserved. The bounded reducer correction admitted only that one root directory and the five exact direct-child files, kept traversal/link/duplicate controls, and checked all archive payloads. The fresh report-02 receipt (local evidence omitted from stage) exited 0 in 14.347 seconds with scratch removed; the resulting report (local evidence omitted from stage) authenticated 48 cells and produced 24 per-seed layout pairs. No native cells were rerun for this reducer retry.

A research candidate is dispatch based on cardinality, locality/run structure and expected reuse, counting conversion and Top64 in the objective. A future prospective H1 would require an improvement in the complete specified query lifetime with exact membership; H0 is no gain once construction/conversion are counted or regression on sparse short-lived sets. This screen does not identify an optimal threshold.

The Vec baseline uses two-pointer merge and Vec::new() output growth, without reservation or galloping. It starts from sorted source vectors while Roaring includes container construction/optimization. Gains compare these implementations, not the best attainable Vec algorithm. Two seeds and tiny operation times support screening rather than stable qualification. No Fabric default or product code changed.