Historical report imported 2026-10-10 from docs/research/cross-system-operations-dissection.md. Original SHA-256: 1f01339937efeb5ce973938817f36293a6de972ae7e5d24a02071c2af491ac67. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Operations, progress and failure ownership across systems

Status: selected implementation source inspection, 2026-10-08. This is research under the cross-system program, not an adopted architecture, verification-policy change or performance result. No upstream code, project build, test or workload was executed for this inspection. The evidence below concerns the selected files, not complete repository audits.

The object is a durable ownership transition: accepted exact Batch bytes move from a producer to the server journal, then to a published Segment. Resource ownership and request lifetime overlap that transition but do not define its durability. Scheduling, admission, reclamation and failure exploration are different axes; a favorable result on one does not establish the others.

Evidence and provenance

The source acquisition receipts retain resolved revisions, member hashes, selected archive hashes, license files and readback outcomes beneath the source bundle (local evidence omitted from stage). source_audit: false in acquisition receipts means retrieval did not itself audit source; the implementation observations in this document were made subsequently. Historical retrieval failures remain failures.

System	Frozen revision	Selected implementation and observation boundary
Vector	Historical 58caf13adb5cd52a7a8e923108f2c61a84367a60; inspected replacement 47e05c4749b6b1af460af521279dc4681a912bb3	Historical receipt (local evidence omitted from stage) retains curl exit 22. First supplement (local evidence omitted from stage) contains adaptive concurrency controller/future; second supplement (local evidence omitted from stage) adds actual lib/vector-buffers/src/variants/disk_v2/{mod,checkpoint_recovery,writer,reader,ledger}.rs.
OTLP Arrow receiver	84512703ba39b6d3eed45d9e4940b7f5ffd73ffa	Archive (local evidence omitted from stage): receiver/otelarrowreceiver/internal/arrow/arrow.go, including admission, conversion, consumption and response ownership.
FoundationDB	Historical 55b4a06d2de16e47d52c0138b2a4a288fd9e7f2d; inspected replacement daae46ce69b1fdddefdbb985e03e5a12576b3077	Historical receipt (local evidence omitted from stage) retains curl exit 22. First supplement contains deterministic random implementation/interfaces; second adds actual fdbrpc/sim2.cpp, AsyncFileNonDurable.cpp and include/fdbrpc/{simulator,AsyncFileNonDurable}.h. Old .actor.cpp requests remain failed.
Shuttle	094be2b9cafa5bfba93df99d7e5cc8a581a430b2	Archive (local evidence omitted from stage): shuttle-engine/src/runtime/{runner,execution,failure}.rs. Requested old src/runtime/mod.rs path was absent; actual runtime files were selected by inventory.
Loom	948c8cc78b178ede6eeff3afc7d97f2f4ea08559	Archive (local evidence omitted from stage): src/model.rs, src/rt/{execution,scheduler,thread}.rs.
fail-rs	2cb57636adf466510d901e45293233226e3f9f13	Archive (local evidence omitted from stage): src/lib.rs, action execution, scenario cleanup and feature-gated macros.
proptest	4b0594eb07c53695dc3c24d3fa06bdceb98540e9	Archive (local evidence omitted from stage): proptest/src/test_runner/runner.rs and failure_persistence/file.rs.
Prometheus	21fd457f2b67fe417e181817d814dba812471e9f	Archive (local evidence omitted from stage): tsdb/chunkenc/xor.go, histogram.go and related chunk implementations.
SlateDB	c7d2b7b067295198af4ed8b7d81ff4d5f2f58d71	Archive (local evidence omitted from stage): slatedb/src/manifest/{mod,invariants}.rs, garbage_collector/manifest_gc.rs and compactor files. Supplement adds actual slatedb/src/manifest/store.rs and db.rs; store wrapper inspected, complete database module not audited.
differential-dataflow	aa8745f93ea8abe131104fc7885ba4fd47e63902	Archive (local evidence omitted from stage): differential-dataflow/src/trace/implementations/{spine_fueled,merge_batcher}.rs. Supplement adds actual ord_neu.rs; old requested paths were absent.

The supplement receipt (local evidence omitted from stage) records new Vector/FoundationDB resolutions separately from their failed historical pins. Its tar members have repository-identity prefixes; those prefixes are not upstream paths. It also contains the inventory-selected Shuttle shuttle-engine/src/scheduler/mod.rs and shuttle-schedulers/src/{dfs,random,replay}.rs at the original Shuttle pin. The second supplement receipt (local evidence omitted from stage) records the nine subsequently selected Vector disk-buffer and FoundationDB simulator/file implementation files at those replacement pins. Neither bundle was executed as an upstream workload.

Response feedback does not observe every bottleneck

Vector's controller combines an EWMA of successful-request RTT with explicit backpressure. It only increases concurrency when demand reached the current limit, the current RTT is no worse than the past mean and backpressure is absent; it adds one permit. Backpressure or increased RTT above a variance-scaled threshold triggers a multiplicative decrease, bounded below by one (218–254). Retry classification decides backpressure; only successful outcomes contribute RTT samples (271–303). Thus RTT sampling and retry pressure have different populations.

Its response future owns a semaphore permit and calls the controller adjustment when the inner future polls ready (29–65). Dropping this wrapper releases its permit through ownership; the selected wrapper has no equivalent drop-time response adjustment. Whether cancellation can desynchronize the controller's separate in_flight counter depends on callers not selected here; this is a boundary to investigate, not an established Vector defect.

Fabric's native commit RTT ends before sealing completes. A controller driven only by ACK latency can therefore raise intake while retained journal backlog grows. Existing Store byte limits remain the custody protection, but an eventual adaptive policy needs observations of backlog age, oldest releasable progress and resource debt in addition to latency. The controller files do not establish Vector's disk-buffer durability semantics, and they supply no Fabric capacity measurement. No adaptive default or wire protocol change is adopted.

The second supplement's writer separates publishing readable progress from periodic full synchronization. flush_inner flushes buffered writes and publishes progress; only when its ledger timer or forced-flush condition requires it does it call data-file sync_all followed by ledger flush (1913–1940). flush_record delegates a write to the buffered writer and updates its size; it does not itself invoke the separate sync_all method (699–742, 814–815). An adjacent comment labels that record durable, but the selected call path is not evidence that every returned record underwent Fabric's data-and-marker syncs. The delegated buffered writer and finalizer status types were not selected here; broader source tracing would be needed to reconstruct all source-level acknowledgment configurations.

One concrete semantic difference is already visible. A permanently unwritable record follows a branch that disarms the error guard, increments dropped-record accounting and returns Ok(Ok(0)) (1754–1790). The comment explicitly intends source acknowledgment/checkpoint progress for that drop. Fabric must not import this branch as a durable ACK: exact accepted Batch custody and explicit gap evidence have their own contract. A library's successful write outcome is not a universal custody promise.

The reader first resolves fully acknowledged records and eligible files, advances every eligible file checkpoint, then flushes that checkpoint (609–719). The ledger publishes the cleanup boundary only after successful flush; its background cleanup deletes files outside the checkpoint window and retries deletion errors (452–503). This is a useful separation of logical reclaim progress from physical deletion. Fabric already checkpoints stream state before removing the oldest sealed file. Moving its deletion asynchronously is a separate hypothesis: bytes are not free until deletion happens, and foreground custody progress cannot silently outrun a quota that still counts those bytes.

Vector's checkpoint recovery distinguishes a torn tail, a complete corrupt frame inside the checkpoint window, and a provably post-checkpoint frame. It preserves a complete corrupt frame when its position remains within the recovery window, but truncates a torn tail or a frame proven outside/crossing the writer checkpoint (561–612, 665–708). Its named durable-truncation helper delegates to the filesystem trait (729–734); that trait's native implementation was not selected. The useful Fabric test is an evidence matrix of complete-corrupt versus incomplete frames on each side of the committed marker. Recovery must not repair apparent consistency by discarding an acknowledged Batch. Existing FrameLog quarantine, recovery and independent delivery checks remain the oracle; no Vector corruption policy is copied into production.

Admission is a lifetime contract, and decode can precede it

The receiver implementation has a concrete separation between receiving, decoding, pipeline consumption and sending status. anyStream creates one Arrow consumer, closes it on exit, runs receive/send loops and waits for their termination (lines 358–414). This is a stream-level state lifetime; a batch-level accounting token is not all the state owned by that stream.

recvOne calls consumeBatch before boundedQueue.Acquire (636–650). consumeBatch materializes signal-specific pdata through MetricsFrom, LogsFrom or TracesFrom, then computes serialized protobuf size using the corresponding ProtoMarshaler (788–842). Thus the queue's requested size is a post-conversion estimate. It does not establish a cap on compressed input, decoder dictionaries or transient conversion allocations. The separate decoder memory-limit error is handled, but its implementation is outside this selected receiver file; no complete decoder-memory bound is inferred.

An inFlightData refcount spans receiver and consumption goroutines. A successful handoff adds a reference; anyDone releases the admission token only when the last reference leaves (520–542, 658–670). This is a useful ownership mechanism. consumeDone queues the pipeline outcome for the sender; an absent error becomes an OK status in sendOne (495–517, 705–740). Those functions establish the receiver's pipeline response boundary. They do not establish that downstream storage performed Fabric's two durable syncs.

Fabric already holds query permits with actual blocking work and joins server workers through cancellation: lifecycle_workers.rs (local checkout reference crates/fabric-server/src/lifecycle_workers.rs; not present at origin/main) retains both worker handles in a blocking cleanup task. Therefore the transfer is a targeted inventory of residual owners and allocations, not a new generic permit system or a rewrite of that existing lifecycle correction.

Exploration has a stop reason, not just a successful return

Loom's Builder implementation sets thread, branch, preemption, permutation and duration bounds. In check, the duration/permutation checks run inside the checkpoint-interval condition (166–188). Both can return normally before execution.step() reports no further path (212–216). Consequently process success alone cannot distinguish exhausted exploration from a deliberately bounded stop, and an interval-based duration check does not replace external containment.

Shuttle's runner stops for max_time or for a scheduler returning no next execution and reports an execution count (75–114). Its execution implementation distinguishes FailAfter from ContinueAfter: the latter marks scheduling stopped and returns Ok at the step bound (977–990). Its failure persistence records a serialized schedule, uses create_new for unique files and falls back to stderr when file persistence fails (24–83). Schedule serialization state is separate from borrowed execution state so a panic during that borrow can still be reported (execution.rs, 42–80).

Supplementary DFS code returns no next execution both at its iteration bound and when choices are exhausted. Its enabled random-data path uses a fixed sequence rather than enumerating all data choices (23–29, 51–64, 100–104). The random scheduler seeds a PCG generator per execution and selects a runnable task with that generator (78–99). The replay scheduler normally rejects a schedule ending early or naming a non-runnable task; allow_incomplete permits those stops (79–125). Record that option with the trace: successful partial replay does not establish that the original failure was reproduced.

These observations support preserving seeds, actual schedules, bounds and stop reasons. They establish no coverage of Fabric until its operations are mapped to the modeled primitives. Filesystem sync, storage-device behavior, runtime cancellation and native I/O remain separate experiments. A scheduler or memory model checker cannot certify physical power-loss durability from a thread model.

FoundationDB's retrieved random generator seeds a Boost MT19937-64 engine, keeps the next generated word explicitly, offers reset/peek and can log selected generated values (28–44, 152–159). Its header chooses the Boost engine for consistency across standard libraries. The engine choice alone does not make every method or whole simulation portable: randomSkewedUInt32 uses std::uniform_real_distribution directly on that engine, and randomBytes copies native word bytes (95–101, 129–134). Nor does a seed fix which components call the generator in which order. For Fabric, preserve explicit sampled actions and sampler versions, and separate workload-generation choices from scheduling choices when a comparison changes call paths.

The second supplement supplies actual simulator and simulated persistence implementation at the same replacement pin. In sim2.cpp, killProcess_internal separates an instant process failure, injected faults and disk failure. Injected faults initialize deterministic fault state; a failed disk sets a distinct flag (1790–1843). Public process/interface kills exclude protected addresses, and canKillProcesses can delegate to a simulation policy (1755–1765, 1876–1891). Thus a requested fault and an actually applied fault are different events. A Fabric campaign should record both rather than count a filtered request as exercised coverage. This inspection does not characterize every network scheduling, topology or failure policy in the simulator.

The non-durable file implementation tracks pending overlapping modifications and separates a write's start response from its delayed underlying modification (218–230, 337–389). A durable sync waits for pending modifications and syncs the underlying file. A kill requests the non-durable path; pending modifications may be retained correctly, omitted or partially corrupted, using 4,096-byte page and 512-byte sector units for that model (245–267, 394–470, 584–655). The model also includes a path intended to represent missing directory synchronization before a newly created file is first synced (637–644). These are explicit modeled choices, not discoveries about Fabric's filesystem or a real disk's failure distribution.

That distinction makes the smallest next Fabric test concrete: construct owned, bounded FrameLog fixtures from the same accepted Batch bytes at cuts before data sync, between data/marker sync, after both syncs, and between Segment rename, checkpoint and journal deletion. Apply deterministic omitted/torn modification images to copies, preserve the exact trace/image, and use unchanged delivery and query oracles to check the contract. Register the fault-image assumptions and required negative control before execution. This is a simulator investigation; physical power-loss claims still need native device evidence, and changes to durability or sync order remain outside the experiment's scope.

The fail-rs implementation provides early return, sleep, panic and pause actions. Pause waits on a condition variable; scenario cleanup wakes paused points before clearing configuration (469–502, 556–609). Its macro expands to an empty body when the failpoints feature is disabled (823–848). Any proposed fixture must therefore demonstrate that its injected defect actually fired and that every blocked worker was released and joined. Existing Fabric callback barriers and owned checkpoint failure fixtures already cover several relevant cuts without a new dependency.

The proptest runner replays persisted seeds before generating fresh cases, then saves a failing generated seed with its shrunken value (583–650). Shrinking can stop at iteration or elapsed-time limits (761–848). A finite shrunk counterexample is useful even when minimality is unestablished. Seed replay depends on the strategy and runner; Fabric should also preserve decoded deterministic actions, the violated contract and relevant source hashes, as required by its counterexample rule.

Scheduling fuel can multiply, and consolidation changes meaning

The differential spine does not present apply_fuel as one global budget: it copies the supplied value for each merge layer, allows work to overspend and does not debit a shared remaining balance (563–579). Thus a caller's nominal fuel is neither aggregate work nor wall time. Its merge state retains both input batches while incomplete, and returns those resources with a completed result (848–861). A smaller work quantum can extend the byte-time for which both inputs and partial output live.

The supplementary ordered merger checks effort between whole-key operations and calculates effort from the change in staging.total() (396–422, 898–924). A key operation can traverse its complete values/update lists before that check (copy_key, 435–445; equal-key merging, 954–980). The output/consolidation accounting is therefore distinct from input decode/comparison work and from the largest indivisible operation. This source finding supplies an adverse workload: UpdsBuilder::seal returns before incrementing total when consolidation empties the stash (211–229). Deduced from those paths, fully canceling input can consume work without increasing this fuel charge. No measured upstream latency defect is claimed.

The merge batcher combines equal (data,time) weights and emits nothing for a zero sum (286–339). That algebra is appropriate to its incremental collections. Fabric's delivery decisions distinguish a lost ACK retry from distinct accepted Batches and reject changed bytes at the current sequence. Equal telemetry observations in distinct accepted Batches remain distinct. Transport deduplication cannot be replaced by weighted consolidation.

Numeric encoding and reclamation have semantic prerequisites

Prometheus XOR chunks write the first float's bits, then XOR successive bit patterns, optionally reusing leading/trailing zero windows; decoding XORs into the prior bit pattern (161–218, 466–503, 568–570). Timestamps use delta-of-delta with wider fallbacks. The append count is a 16-bit field and an attempted append at maximum count panics. These are actual codec mechanics, not a whole-engine efficiency claim.

Fabric's Number distinguishes Int(i64) and Double; its observation cells already have integer and float-bit handling. Converting all integers to floats loses exactness above the exactly representable integer range. A transferable sidecar experiment must retain the value tag, units, attributes, ordering and original Batch bytes. It must test irregular timestamps, repeated timestamps, negative zero, supported nonfinite-value behavior and integers beyond 2^53. No supported-signal or production-format change follows from this inspection. Histogram reset/layout code was located, but compatibility with Fabric's supported metrics has not been established.

SlateDB's manifest GC excludes the latest manifest, excludes versions referenced by checkpoints, then applies minimum age. When enabled, it advances a boundary before the optional GC filter and deletion (93–147). Its manifest invariant rejects newly introduced physical SST identifiers whose timestamp falls strictly below the current L0 watermark; equal timestamps and already present SSTs remain allowed (52–88). That distinction prevents a view's identifier from substituting for the physical object's identity at this particular validation boundary.

Fabric uses journal/group labels and ordered checkpoint-before-delete, rather than adopting that wall-clock identifier contract. Its page tokens expire when retention removes required data. Checkpoint-pinned retention from another engine would change that behavior. The useful transfer is an explicit distinction between logical identity, published physical ownership and the reclamation boundary. The supplementary manifest store constructs separate writer/compactor epoch selectors for a fenceable transactional object, installs manifest invariants on the simple object, and delegates updates and conflict retries to slatedb_txn_obj (35–68, 178–182, 426–448). This establishes the wrapper boundary; the delegated object's implementation was not selected, so the complete conditional-write/retry algorithm remains outside this inspection.

Four discriminating Fabric hypotheses

1. Charge decode overlap separately from admitted work. The receiver ordering predicts that a post-decode reservation can look small while conversion peak is large. Census Fabric input bytes, decode allocations, query owners and builder buffers alongside requested allocations, RSS and cgroup memory. Keep admission unchanged for the first measurement. A no-conversion control and a wide mixed-signal Batch distinguish conversion overlap from generic allocator retention. Reject a reservation-only explanation if RSS/allocation evidence disagrees. This extends M1, not a transport migration.
2. Prefer release of a real oldest prefix, then measure its competing work. Ordered joins can reduce retained journal byte-time while earlier checkpoint work competes with ACK processing on the same commit thread. The native prefix probe uses actual Store, builders and candidate sealer, a historical example-only scheduler, balanced/skewed three-worker cases and a one-worker negative case. Measure observed release intervals and ACK tails together; do not infer visibility or sustained capacity from early deletion. An adverse ACK tail can reject unconditional scheduler superiority even when byte-time improves. If low ACK latency coexists with a growing oldest backlog, reject an ACK-only adaptive intake policy. A feedback extension must retain unchanged custody and separate stream/byte admission from builder concurrency. This is D1's existing continuation, not another scheduler proposal.
3. Explore ownership cuts with explicit incomplete outcomes. Compose modeled schedules with existing native fixture replay: lost response after commit, canceled query while blocking work lives, failed checkpoint after publication, and reclaim error while sibling builders are active. Extend only uncovered persistence cuts with deterministic owned-file images, distinguishing a requested fault from an applied one. Record complete action traces, modeled bounds and termination reason, including incomplete-replay options and sampler versions. Seed persisted regressions before fresh exploration and inject one representative early-release defect. A bounded stop remains inconclusive beyond its visited schedules. Prefer existing barriers before adding dependencies; never put a checker in the pure core or treat it as a physical durability oracle.
4. Compare global consumed-work accounting with output-based fuel. For a standalone future merge diagnostic, count consumed rows/bytes and indivisible decode/write steps across all workers, not a per-layer copied budget or output count. Include near-zero selection and large indivisible rows, with no raw consolidation. Record retained input/output byte-time and ACK/query interference alongside completion time. If smaller quanta increase retained lifetime or backlog, that falsifies a simple smaller-quantum-is-better prediction. A production builder scheduling change requires its own scoped protocol.

These candidates occupy different axes: allocation lifetime, a synchronization barrier, schedule evidence and aggregate work accounting. Their interactions matter, but none is selected by upstream novelty or reputation. Source retrieval is evidence of provenance; the above hypotheses need finite registered Fabric experiments before they support a behavior or performance claim.

Native probe preparation counterexample

The native-build-01 receipt (local evidence omitted from stage) and compiler output (local evidence omitted from stage) retain cargo exit 101 for cargo build --offline --locked --release -p fabric-server --example readiness_memory_lab --example prefix_load_probe. Rust rejected the example's reference to the server's pub(crate) SystemClock with E0603. The correction implements public fabric_ports::Clock in the example with the same wall-clock/error conversion, keeping the product adapter private. This was a compile failure, not a failed custody experiment. Rebuild outcome is not yet recorded here.

A separate source audit found that observing a successful oneshot reply does not prove Store::run has executed its subsequent committed-group counter store. The probe now drains and joins that thread, reopens Store and captures the replay-initialized group counter before quiet post-publication queries. This removes a timing assumption without changing product source or sync ordering. Neither correction has been built or run by the inspecting agent.

The prefix-load-01 receipt (local evidence omitted from stage) and startup output (local evidence omitted from stage) retain child exit 1 for a redundant child attempt to acquire the coordinator's already-held campaign lock. Failure occurred before the native probe launched. The runner now relies on the parent's existing lock and requires its inherited FABRIC_CROSS_SYSTEM_COORDINATED=1 marker. The historical failed receipt and retained scratch status are unchanged; a subsequent run needs its own receipt.