Experiment benchmarks coupled query completion readiness - kmosoti/FabricO11y GitHub Wiki
Historical repository record migrated on 2026-10-10. Original repository path:
docs/experiments/benchmarks/coupled-query-completion-readiness.mdOriginal SHA-256:f2e2ed360080a252575ce15913e171eaffce30ec07ac267c7b23c141583f2e5cStated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.
Query completion readiness
Status: read-only assessment; no new workload or permission-derived result. Elevated filesystem access does not resolve correctness, evidence attribution or remaining time admission. Root still serializes contained workloads.
Close Q4 without nomination
Yes: close the admitted Q4 screen as completed without nomination. Its findings retain six children, 48 complete chains, optional-filter controls and five final-source lifecycle tests. The single 64-Segment plain pair reduced schedule CPU 3.16% and wall 0.91%, below the 10% primary threshold and without three fresh pairs. Counted query allocations fell only 1.06%; tiny preflight total CPU increased. Initial snapshot preparation was outside timing. Nothing supports promoting reuse or extending this same comparison merely to obtain more favorable samples. Defaults remain unchanged; metadata handles grant no file lease. Controls01's overlapping accessor edit remains a limitation; controls02 supplies the final-source control.
Highest-value next query hypothesis
C4 observed eight repeated string attributes adding about 3.6–4.0 ms first-page CPU and 15.9 MB requested allocation, while borrowing saved essentially the same absolute allocation at both attribute settings. Its observer failed admission, so attribute parsing dominance is unproved; page decoding and other work remain alternative explanations.
The narrow hypothesis is that validating every in-window attribute map while
materializing owned map strings only for predicate/top-k survivors can reduce
selective-query cost. Today attrs_of
deserializes an owned BTreeMap<String,String> before the borrowed callback.
This is a next investigation, not an authorized implementation in this note.
H0: validation remains the dominant cost, a second pass cancels savings, or
equivalent validation cannot preserve behavior. A syntax-only JSON check is
insufficient: keys/values, escapes, duplicates, node identity, column types,
malformed rejected rows and error behavior must remain equivalent. Start with
validation/materialization discrimination and observer calibration, with empty
attributes and broad queries as controls. Do not skip validation or claim C4
already identified the causal stage. Full chains remain independent-oracle gates.
Exact gates before C5/O8
O6/O7 remain unresolved. O6 agrees on zero rows but disagrees on retained window, freshness and gaps; O7's hypothetical footer counts do not establish pruning soundness. Preserve their failed gates. Any availability model or physical acceptance change needs its separately scoped decision; permission escalation is not that decision. Relevant existing commands are:
python3 -B tools/resource_group.py -- cargo test --offline --locked -p fabric-server --test completion_storage --all-features missing_query_tables_are_incomplete_with_independently_declared_unavailability -- --nocapture
python3 -B tools/resource_group.py -- cargo xtask checks --profile fast
C5 must retain exact producer Batch custody and complete logs/metrics/spans/rate chains for each worker/run configuration under fixed concurrent query demand. Compare logical rows, total ordering/ties, continuation, retained window, freshness, gaps and availability across active, pending, published and reclaimed sources. Actual row-group identity/bounds and visits must establish conservative pruning; changed layouts cannot silently relax the historical C2 criterion. Optional index failures require exact fallback; source movement requires explicit bounded retry or the existing error semantics, never a silently complete omission.
O8 must distinguish prepared, durably Spool-accepted, sent, server-committed and ACKed custody. Unsent prepared data is not query-visible server evidence. Retry the same Strand/sequence/encoded bytes; grade one logical commit despite lost ACK or retries, and preserve cursor/Spool ownership on cancellation. Server Group frontiers are not Strand sequence or ACK-count arithmetic. Use actual committed frontier and producer ledgers to bind expected query snapshots.
Measure observation→Spool, Spool→ACK and ACK→queryable separately. Queries can read active/uncovered journals; Segment publication is not required for visibility. Polling gives interval-censored visibility latency, not an instantaneous or new 30-second guarantee. Correctness, finite lifecycle controls, explicit clocks, resource/evidence attribution and fresh admission precede either service trial.