Research cross system query vectors - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/research/cross-system-query-vectors.md. Original SHA-256: 49da4169c5b5a562ea1e46619eb6e08d98cf8a97b88c3d0bbf57f43c5a294635. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Query trajectory vectors after the matched mechanism screen

This is research, subordinate to the product contract and registered evidence. Measured findings support optional candidate avoidance plus physical locality as a more promising selective-cluster vector than unconditional format conversion. They do not demonstrate a native Fabric speedup or prescribe a production index/format.

Vector Observed mechanism evidence Falsifiable next question Costs and constraints
Optional group pruning Clustered wide selective 1,024-row groups retain 1/32 and query 2.94–3.00 ms; mixed 8,192-row groups retain 4/4 and offer little saving. Does a bounded native group-summary pruner avoid row decoding while preserving malformed-row/identity validation and exact page continuations? H0: metadata/selectivity or validation obligations erase savings. Python explanatory index costs 4.51–4.66 s; do not import that builder as a production design. Disable/skip when all groups remain; maintain exact verification, no false negatives.
Physical locality and granularity Same logical matches distribute over 1/32 versus 22/32 small groups; large mixed groups retain everything. Does a permissible sealed-segment ordering/grouping choice reduce candidate bytes under actual mixed time/body queries? H0: ingestion cost, cross-query conflicts or ordering constraints dominate. Charge layout construction and ingestion; preserve canonical result order, custody/durability semantics and immutable source identity. Locality optimized for one predicate may hurt another.
Selected representation conversion Wide Vortex predicate selective conversion 0.53–1.09 ms versus full 35–36 ms, with exact operation also avoided; scan itself becomes more expensive. Can Fabric defer payload decoding/materialization until safe candidate validation, without accepting corrupt rejected rows? H0: decoder validation or output ownership still requires equivalent work. Separate scan/filter, ownership/conversion, exact ordering and final 64-row projection. Historical copy-only CR2 CPU guard failure remains.
Prepared plan reuse Validated ablation (local evidence omitted from stage): 384 exact answers; preparation 0.020–0.201 ms. Wide clustered selective prepared/new wall 0.914/0.986 and CPU 0.969/0.978 across the two repetitions; wide mixed wall 1.180/0.957. Planning dominance is unsupported. Optional reuse can repay cheaply for positive fixed-query populations, but adverse/null cases remain; narrow mixed unrestricted wall 1.223/1.018 is slower in both repetitions. Tiny preparation cannot alone explain millisecond execution changes; shared reusable state/cache and variation remain. Record both repetitions, CPU and first calls. No blanket prepared-plan policy follows.
Ingestion/format replacement DuckDB table penalizes wide shapes except clustered narrow-time; wide Vortex files 22.1–22.4 MB versus Parquet 1.49–1.72 MB and full scans lose. Is there any workload where persistent conversion repays ingestion/compression plus query CPU and footprint? H0: cheap Parquet/locality is sufficient. Measure full construction/checkpoint/refill and storage, not just query latency. No FSST dispatch attribution from encoding-tree text; no unconditional format migration.

The optional-pruner trajectory preserves a scan fallback and separates candidate selection from exact verification. Its strongest current support is clustered selectivity, not every query. A native experiment must admit the same logical fixture/query contract and expose validation obligations before this mechanism screen becomes evidence for a Fabric change. Index build/update distribution, nullable/rich attributes, long unpredictable bodies, concurrent queries and realistic cold storage are unresolved dimensions.