Research coupled performance review - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/research/coupled-performance-review.md. Original SHA-256:4958eee9b859d578addf5b3cdc3e5e91feed2ca867c119caa5bf9d972136d4e3. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Reviewed 2026-10-07 against the active working tree and retained lab receipts.
Input: the owner's supplied FabricO11y Development Direction: Systems Research
and Performance Strategy (main snapshot through 5f0c52f1, 2026-10-03), plus
the accompanying 36-part coupled optimization model. This is a research review,
not a new registered experiment, product contract, configuration or solver.
The model's strongest idea is to optimize collection, storage and query together under hard semantic and resource constraints. The completed catalog campaign supplies direct counterexamples to choosing a mechanism by isolated allocation savings.
| Direction in the supplied material | Current evidence | Implication |
|---|---|---|
| Implement bounded sealing first | Production worker calls the bounded builder; largest finite screen reduced incremental builder heap from 1,185.29 to 32.06 MiB | Implementation exists; aggregate deployment boundedness still needs evidence |
| Remove representation transitions | Binary private spill rows and partial merge passes reduced paired build time 20.7% at 260.69 MiB, with unchanged 32.061 MiB incremental heap | Strong local result; final materialization still takes seconds |
| Avoid owning rows until needed | CR2 saved 24.61–24.68% of selective allocation bytes, but one plain pair missed CPU/wall preservation | Borrowing removed allocations without establishing an overall speed win |
| Share immutable metadata | CR3 reduced absent-query allocation peak about 64–67%, but total requested bytes only 5.3%; two small-case p95 guards failed | Peak memory and cumulative allocation answer different questions |
| Proactively maintain the catalog | CQ2 eager refresh failed its total-CPU improvement rule at every measured query rate | Moving work earlier can duplicate work; background maintenance remains untested |
| Storage owns source discovery | Narrow Walk boundary preserved performance; both Walk and Scan had a reproducible publication/reclaim omission window, now corrected | Coverage correctness belongs in the storage/query boundary as much as lookup speed |
The memory screen and speed screen distinguish heap from process RSS. Their percentages are not server-wide RSS reductions. The current sealer still builds files in worker-sized concurrent chunks. ADR-0022 now explicitly records its historical single-worker assumption and the remaining aggregate-bound uncertainty.
The attached recommendation to make tail work asynchronous is not disproved by CQ2: that experiment refreshed synchronously after ACK, with no background worker or service contention. It does disprove assuming that precomputation automatically reduces total work. The measured path discarded acquired source descriptors and repeated discovery on queries. A future candidate must identify exactly which work disappears and which state it retains.
- Preserve units and measurement boundaries. The historical collection profile labels 9.7 as MB/s, not MiB/s. Thus 9.7 × 0.027 is approximately 0.2619 MB per cycle, and the idealized overlap result is 17.46 MB/s. Historical ingest numbers are also MB/s, from a 30-second offered-load test with backpressure and later draining, not a direct steady-state sealing service curve. The 19–27 estimate is a scenario, not a measured result or statistical prediction interval.
- Balance is diagnostic, not a universal optimum. Full overlap is optimal only in the stated model with no contention, synchronization or extra-memory penalty. More encoding effort is not justified merely to make stage times equal. Measure whether the saved transport cost exceeds the added preparation and shared-CPU cost. A bounded prepared Batch must preserve cursor, Spool, retry, shutdown and one-sent-Batch-per-Strand invariants.
- Average waiting time does not bound p99. B/(2λ) approximates mean formation delay under restrictive arrival assumptions. Bursts, quiet sources, flush timeouts and queueing determine tails. Specify both maximum bytes and maximum age, then measure observation-to-Spool, Spool-to-server-ACK and ACK-to-queryable separately. The owner's 30-second breathing room is not an intentional delay or a demonstrated service guarantee.
- Three points fit; they do not validate a curve. The worker throughput, latency and RSS curves interpolate historical observations from another revision/host. Two workers met the illustrative sampled 1 GiB/200 ms limits; that does not certify them for a deployment. Workers are discrete. Continuous KKT conditions require regularity and do not establish global optima here. Compare feasible measured configurations, uncertainty and boundary cases.
- Shadow prices need units. CPU and memory multipliers have different units; their raw magnitudes cannot meaningfully be compared. Compare predicted objective changes for specified increments of each resource. Do not add independently measured component peaks as if simultaneous.
- Model queue stability and disk headroom. An admitted rate must be serviced over time, not merely fit instantaneous CPU estimates. Track journal/Spool backlog slopes, refusal/retry rates, temporary spill bytes, retained bytes, read/write contention and cleanup. Retention capacity also includes raw custody, metadata and reserve; a Segment-only ratio does not automatically multiply whole-system retention by the same factor.
- Pruning factors are conditional. Multiplication is valid when each pass fraction is measured conditional on surviving earlier stages. Independent marginal selectivities cannot generally be multiplied. Node/token clustering can make indexes useful at cardinalities where the uniform independent model predicts little benefit. Fleet cardinality alone is not an activation rule.
-
Clarify block-cost units. The proposed per-row objective charges a block
read cost
bwithout dividing by rows/block. Ifbis truly cost per block, useb(R) × h(R,u) / R; otherwise define it as a per-row cost. Compression, false-positive behavior, finite filter sizes and observed clustering affect the derivatives. A stationary point still needs boundary/global comparison. -
Respect exact query order and decoding costs. The current Walk uses a
strict timestamp bound (
min > threshold.timestamp) because ties need the complete key. Replacing it with timestamp>=could skip eligible tied rows. Payload materialization approaching k does not mean only k rows are decoded: compressed pages, text predicates and required validation still cost work. An Ω lower bound does not assert that an implementation achieves that cost; source discovery and heap operations also remain.
The compaction and run-size equations are useful hypotheses, not established laws. Three run sizes fitting three parameters leave no independent fit check. Reserve a held-out size and test mixed/adversarial shapes before extrapolation. Compaction also changes physical coverage, pagination and reclaim obligations; it cannot be admitted solely by predicted source-count savings.
| Lab | Next discriminating question | Hypothesis and null | Evidence needed |
|---|---|---|---|
| Operations | Can missing-table availability be represented consistently by runtime and independent specification? | Explicit source availability reconciles envelopes; null: counterexamples remain | Separate contract/oracle decision first, then unchanged expected semantics, deterministic fixtures and full fast checks |
| Capacity | Where does borrowed-row time go after allocations fall? | Attribute validation or decode dominates; null: those stages do not explain residual CPU | Plain stage CPU, allocated bytes, decoded/materialized rows and exact complete chains; include large attributes and mixed signals |
| Query | Can a storage snapshot eliminate repeated discovery without retaining unbounded generations? | Reusing validated descriptors removes work; null: invalidation/lifetime costs erase savings | Cold/warm and append/seal/reclaim schedules, clone/shared baseline, active-reader lifetimes, full answer envelopes and total maintenance-plus-query CPU |
| Capacity + Operations | Which worker/run-size combinations remain bounded under real ingestion and query demand? | A feasible region survives contention; null: backlog, ACK or resource guards fail | Small finite grid with held-out cell; simultaneous cgroup/RSS/heap, offered/accepted/refused/retried rates, backlog slope, CPU and disk amplification |
| Operations + Query | Does depth-one preparation overlap help a dev/small service? | Waiting can hide preparation at fixed custody; null: contention or latency cancels gain | Same durable bytes/order, one sent Batch, bounded prepared Batch, restart/cancel controls, actual Spool acceptance and ACK-to-queryability |
Start with the known availability-model failure and C2 physical-pruning mismatch; they currently block stronger claims. Register each new experiment and its negative controls before workloads. The completed campaign does not authorize a new unlimited budget or qualification/release runs. Keep three lab ownership roles, serial workload admission and task-scoped assistance.
FOB1-only storage remains a separate high-potential research branch. The storage direction must be evaluated across metrics, rates, traces, text, pagination and recovery, not just text density. Canonical semantic equivalence is insufficient to replace today's exact received-byte custody requirement. Wire/storage migration and any custody change need explicit scope. No adaptive solver, new index, compaction layer or format is implemented by this review.
Fabric has a substantially smaller builder working set, a faster bounded build, useful source pruning, and stronger evidence for a narrow storage-owned query boundary. The labs also found and corrected a real omission race. That is meaningful progress toward a usable dev/small service.
Full fast verification still fails the known partial-table metadata/oracle case; C2 pruning equivalence remains unresolved. No experimental catalog optimization was promoted, and no end-to-end capacity or release qualification follows from these component trials. The model helps choose the next measurements; it cannot fill those evidence gaps with calculated throughput.