Research cross system candidate screen - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/research/cross-system-candidate-screen.md. Original SHA-256:6d680053dc04c075bc04941d904185b24a178ca4542818c5514fbe180c57b113. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Status: source evidence and analytical deductions, 2026-10-08. No executable
experiment, implementation change or performance result. This follows the
generative stage
and narrows the existing queue (local checkout reference docs/research/cross-system-lab-queue.json; not present at origin/main).
| Candidate | Disposition | Decisive evidence |
|---|---|---|
| D1: reclaim each completed prefix | Retain for a small correctness screen, then mixed-load measurement | Fabric waits for all workers in a group before reclamation. Reclamation itself blocks on the commit thread and saves its checkpoint; eliminating the group barrier does not eliminate that cost. |
| D2: choose copy versus borrow for log results | Revise; do not build a new generic policy for the inspected log paths | Winners already become owned rows and decoded-block borrows do not escape their callback. Remaining allocation and cache lifetimes are different mechanisms. |
| D3: share derived-memory effort between hot reads and sealing | Keep diagnostic until M1/Q1 measure actual owners and reuse | Query source snapshots hold compressed blocks by Arc; eviction from the catalog alone need not free their backing allocations. |
| D4: feedback-scheduled merge work | Keep behind its isolated components | No new evidence distinguishes network RTT from durable-sync contention or establishes a useful control law. |
| D5: match literal log substrings in compressed bodies | Add as an M3 research branch; no additional native campaign | Concrete Vortex FSST kernels preserve compressed filtering and evaluate a supported subset of LIKE patterns over codes. Conversion, copies and predicate fallbacks remain material costs. |
Observed in sealer.rs:
build_and_reclaim joins all workers in a chunk and collects their results before
calling reclaim_prefix. It records successful labels, retains the earliest
build error in label order, reclaims the contiguous published prefix, and stops
before starting a new chunk if a build failed. Existing tests include progress
before a later group finishes, failures/panics and checkpoint failure/retry.
The historical result
does not test early reclamation within a still-running worker group.
Observed in store.rs:
Intake::reclaim sends a command through the commit queue and waits for its reply.
Store::reclaim requires the oldest sealed journal, saves stream state, then
removes that journal. Reclaims encountered while forming an ingestion group wait
until that group commits. Consequently a completion notification is neither an
immediate disk-space release nor permission to delete a journal directly.
The smallest proposed distinction is bounded completion notification for one existing worker group, with reclamation of each newly eligible oldest prefix. Keep the worker count, build function, publication protocol, checkpoint operation and journal ordering unchanged. Finish and join every started worker before returning, including after a build, panic or reclaim error. Preserve deterministic error choice instead of returning whichever thread fails first.
Adversarial deduction: a bounded completion channel can deadlock if a coordinator stops draining it and joins workers blocked on send. One result per worker and capacity for the whole admitted group, or explicit draining during shutdown, must be part of the ownership argument. That is not a tested implementation.
The discriminating schedule is an early successful oldest label with a blocked later sibling in the same group. Reverse which label is blocked as a null case; also test holes, panics, checkpoint failure, retry and a busy commit queue. Measure published-but-unreclaimed journal byte-time and capacity refusal duration before claiming ingestion benefit. Query visibility can already use the published Segment; neither freshness nor RSS improvement follows from earlier deletion.
Observed in query.rs:
- The Walk log-block path checks predicate, pagination and selection threshold
before
tail::log_rowcreates an owned result. - The borrowed Parquet log path checks those conditions before
into_owned.Smallest<LogRow>stores owned payloads, not references to decoded blocks. - Metrics and spans differ: the block path calls
metric_roworspan_rowbefore checking the row predicate, and rejected threshold candidates can reachSmallest::offerafter their owned fields have been constructed.
Observed in tail.rs: visit_block
keeps decompressed bytes local, decodes a borrowed view and invokes a callback.
Metric construction owns node/name/unit/attributes; span construction also
formats identifiers. The explicit TailReader cache stores up to 64 decoded
Groups independently of whether a row wins selection.
Observed in segment.rs:
LogView borrows node/body, but owns attributes. read_logs_borrowed parses and
validates attributes for each in-window row before the query callback. Deferring
all of that work until selection would change error behavior. An allocation
optimization must preserve validation, including rejected-row controls.
Conclusion: retain copy-versus-borrow as an adversarial design check, but retire
the premise that current selected log rows need a new policy to release decoded
frame borrows. Q2 should first measure rejected metric/span ownership and
attribute construction. D3/M1 separately measure cache generations: the
source view (local checkout reference crates/fabric-server/src/read_catalog.rs; not present at origin/main) retains an Arc to
each selected compressed TailBlock until that view is dropped. Shared objects
must be charged once and remain live while any actual owner holds them.
Another M1 boundary is the existing Rate query. Unlike the paginated row queries,
it collects matching points into a Vec before evaluating the rate. A limit + 1
selection bound therefore does not describe every query. Include a finite wider
rate window in the census; a count limit on simultaneous requests alone does not
establish a byte limit on their results or intermediate state.
These observations reject duplication of an existing mechanism. They do not establish the magnitude of any remaining allocation, RSS or service-rate cost.
All files below were retrieved at commit
881b867d736e8955a0381655f80d6d2d630fc590. The initial assistant inspection of
registrations/tests could not establish the returned representation. Following
those registrations into the implementations resolved part of that uncertainty:
| Pinned implementation | Observed behavior | Consequence for the experiment |
|---|---|---|
| FSST filter | Filters the compressed codes using VarBin, then constructs another FSST array with its symbol table and filtered lengths | The representation is preserved at this boundary. That does not prove zero copying or whole-path peak memory savings. |
| VarBin filter | Nontrivial selection builds output offsets/validity and appends selected encoded bytes to a builder | Count compact-code output and input/output overlap; preserved encoding is different from shared backing storage. |
| FSST LIKE kernel | Supported constant, case-sensitive patterns use a matcher over codes and produce a Boolean mask; unsupported cases decline the kernel | This supplies an actual compressed predicate path, not merely a registration or a canonicalized test result. |
| Matcher construction | Builds a symbol-aware matcher for supported prefix/contains shapes; enforces byte-length/state limits and otherwise returns None | Construction cost depends on the pattern and dictionary. Contains pushdown is limited to 254 literal bytes at this pin; a longer valid predicate still needs exact fallback. |
| Scalar access | Decompresses the requested scalar into a new byte buffer | Returning payloads still costs decoding and allocation even after compressed selection. |
| Filter/take fusion | Can translate selection indices into child operations; Boolean children and specified large/fanout cases decline fusion | Include specialized and fallback paths; source presence does not guarantee a particular query uses the specialized path. |
Combine the existing Fabric key/threshold walk with an experimental compressed log-body predicate, then decode selected output rows. This uses a known upstream mechanism in a proposed Fabric path; no novelty claim or production format change is implied. It is an M3 candidate under the existing two-prototype nomination cap.
Fabric uses literal str::contains. SQL LIKE syntax has different semantics.
An adapter must encode percent, underscore and backslash as literals, or route
those predicates to the unchanged exact path. Never silently treat a literal as
a wildcard. Test empty needles, escaped characters, multibyte UTF-8, and lengths
on either side of the matcher boundary. Byte length is not character count.
Unsupported patterns require exact fallback, not an empty candidate set.
Measure three boundaries separately: building the representation and matcher; executing a predicate/selection without canonicalizing prematurely; and decoding final output with the required evidence/integrity checks. Then compare the whole path against the current bounded-Parquet and journal-backed behavior. Include new dictionaries and new needles as adverse cells, plus repeated predicates for the reuse hypothesis. Record buffer overlap, code copying, dictionary residency, input/output bytes and accepted/complete query costs.
For a fixed corpus and CPU-only diagnostic, let E be extra construction cost,
B baseline query cost, C candidate query cost including matcher work, and Q the
number of queries before eviction/publication. Reuse only repays construction
when Q * (B - C) > E; if B <= C, more reuse cannot pay this CPU bill under the
model. Memory, I/O and semantic constraints remain separate. The values of these
coefficients are unmeasured. No format comparison may omit original Batch custody,
conversion or validation from its claimed end-to-end result.
The queue (local checkout reference docs/research/cross-system-lab-queue.json; not present at origin/main) records source/blob identities. Local
source excerpts, hashes and the mount observation were read with commands that
exited 0. The preceding turn made progress by identifying the worker-group
barrier from source; this continuation refined that mechanism, rejected an
already-addressed log-ownership premise and identified concrete compressed
kernels. These are source findings, not successful executable experiments.
At the source-screen stopping point, the data mount reported ext4 read-only with errors=remount-ro.
The cause is unestablished. The same unavailable execution volume has persisted
across the goal's initial research turn, generative-direction turn and this
continuation. Git metadata also remains outside this session's write permission.
No native workload, checkout, compiler, validator or scratch artifact was created.
There is no live workload handle to wait on. No resource allowance was reset.
The next necessary evidence was executable: register and run the G0-admitted M1/Q1/O1 cells, then nominate candidate prototypes. More source predictions cannot establish their CPU, RSS, latency, custody or recovery outcomes. Execution requires the mounted data drive to be writable and the session to permit registration commits. The full goal is not achieved; all native cells and documentation validators remain unrun, and the historical C5 preservation constraints remain.
After the owner's renewed permission, the mount reported writable and the
launcher verified the existing 20 GiB/no-swap cgroup. The separately registered
readiness slice (local checkout reference docs/experiments/benchmarks/catalog-cross-system-readiness-protocol.md; not present at origin/main)
passed the six existing sealer controls and downloaded the pinned Turso archive.
The D1 investigation (local checkout reference docs/experiments/benchmarks/catalog-prefix-progress-protocol.md; not present at origin/main)
tests a simpler alternative to completion notifications: consume scoped join
handles in label order and reclaim after each join. Later completions cannot
advance an oldest-prefix boundary while an earlier label is missing. Full memory,
query and service-rate census remains outstanding; this does not complete C5.