Historical report imported 2026-10-10 from docs/experiments/benchmarks/service-recovery-research-findings.md. Original SHA-256: 331adb7b60df0e9226fb3bf89212fd62f63f176f82e05a7ff64ea41166de9ec1. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Bounded service residency and storage recovery findings

The registered diagnostic (local checkout reference docs/experiments/benchmarks/service-recovery-research-protocol.md; not present at origin/main) completed both native service cases and all eighteen storage recovery cases. The current service preserved exact logs/ACK custody and complete query answers across four Segment publications per case. Resident memory settled during the three-minute quiet period. This is new combined-service evidence; the registered long soak, full bounded-sealer acceptance and target-host qualification remain open.

Workload and evidence boundaries

Twenty real Spindles offered 60,000 deterministic 900-byte logs over three 60-second phases at 200/600/200 aggregate logs/s. A further 180 seconds kept queries and 15-second host metrics active without new source logs. Native TLS, one sealer, 16 MiB journal files and the normal 16 MiB sort-run cap were used. Scan ran first, Walk second, each in fresh state. One ordered case per plan is descriptive evidence, not a randomized comparison or a default nomination.

The native server had two CPU equivalents, 768 MiB memory high and 1 GiB max; all twenty Spindles shared two CPU equivalents, 384/512 MiB high/max. Both enforced zero swap and task caps inside the 16/20 GiB laboratory cgroup. The server used CPUs 0–1, collectors/harness CPUs 2–3 on the twelve-CPU Linux host. This differs from the target qualification profile and from the earlier 1000/3000 logs/s small workload. Raw environment, limits and binary identities are in each Scan (local evidence omitted from stage) and Walk (local evidence omitted from stage) record.

Current release binaries were rebuilt without experimental features and bound to an exact source snapshot (local evidence omitted from stage). Protocol commit 8e5e1a0 preceded execution. Existing dirty and untracked product source is captured; the commit alone is not the full measured source identity. The later harness correction is separate from the captured timed sources.

Native observations

Measurement	Scan	Walk
Exact recovered logs	60,000	60,000
ACKed Batches matched by hash	3,879	3,880
Published Segments	4	4
Server peak sampled process RSS	165.73 MiB	92.85 MiB
First quiet-window median server RSS	153.54 MiB	70.05 MiB
Last quiet-window median server RSS	153.54 MiB	71.14 MiB
Server CPU over sampled service period	23.24 s	10.68 s
Mean server CPU equivalents	0.064	0.029
Collection → observed ACK p99, weighted by log records	23.31 ms	21.03 ms
Source write → receive p99	994.52 ms	1,002.06 ms
Recent-log query p99	46.49 ms	35.28 ms
Absent-text query p99	115.30 ms	58.15 ms
CPU-metric query p99	45.42 ms	35.00 ms
Sampled source → visible maximum, 36 probes	1,292.91 ms	714.83 ms
Sampled observed ACK → visible maximum	401.90 ms	300.93 ms

Every registered diagnostic gate passed. Final source backlog and pending Batches were zero; unexpected gaps, changed live rows and query errors were zero. The six complete final query chains matched the unchanged independent Python oracle; four missing/changed-row controls were rejected. Live returned rows were checked separately: 3,084/3,083 unique rows matched durable replay. Source body hashes and ACK hashes cover all offered logs and acknowledged Batches; the complete-query fixtures deliberately cover narrower selections.

Spool acceptance was approximately twenty successful Batches/s during offered load in both cases. Payload acceptance rose from about 198 logs/s in the first window to 596 in the burst, followed by about 204 during recovery; boundary carryover drained afterward. Every source record was accounted for. There is no complete Spool append-attempt denominator, so no append-acceptance percentage is claimed. Producer timing p99 lag was below 0.5 ms in both runs.

The quiet RSS discriminator was H1-compatible in both cases. Cgroup anonymous memory supported the same pattern: Scan median 142.70 → 142.70 MiB; Walk 59.44 → 60.53 MiB. File-cache medians rose about 0.625 MiB in each case while metrics continued. These are distinct counters; process RSS, anonymous memory and charged file cache must not be added as independent owners.

The raw per-node maxima include the short Python launcher before exec; they are not clean native Spindle peaks. A post hoc inspection restricted to offered load and later samples after every node emitted a first Batch found sampled per-node RSS approximately 5.61/5.62 MiB. Summed RSS counts shared mappings repeatedly; the aggregate node cgroups are the enforcement boundary. This attribution issue is retained explicitly rather than changing the original samples or gates.

All child memory high/max/OOM counters and swap observations were zero. Whole service-job peaks were about 687/684 MiB, including the coordinator, offline decoder/oracle and page cache. Those are not server RSS. The complete reduction (local evidence omitted from stage) links raw summaries, resource samples, clocks, rates, answers and grades.

Storage failures and interrupted recovery

The current bounded builder passed all seventeen existing selected syscall/cut cases plus one repeated-recovery case. The latter killed the initial spill, killed two recovery attempts immediately before publication, then retried without injection. Every case had an actual injection marker, failed execution, unchanged original input SHA-256, exact final reference manifest and no remaining building/run paths after successful recovery. The fault-free and unmatched-path controls behaved as expected; the classifier rejected four representative false-pass records. See the outcomes (local evidence omitted from stage) and summary (local evidence omitted from stage).

The interposer injects ENOSPC on selected write/sync/rename calls, EIO on reads or after short IO, and SIGKILL at selected boundaries. This establishes those error/retry behaviors, not actual disk exhaustion, hardware power-loss recovery, every crash interleaving, or all of BS-4/BS-5. Exact manifest equality here is stronger than a row count; it still uses the existing native verifier rather than a new independent full-query oracle for each fault state.

Evidence defects found and repaired

The original runner wrote an artifact map before its final resource snapshot. Both datasets consequently have one stale digest, for cgroup-final.json. The registered repair (local checkout reference docs/experiments/benchmarks/service-evidence-repair-proposal.md; not present at origin/main) preserves both original maps and authenticates the current artifacts in a separate reconciliation (local evidence omitted from stage). No other listed artifact changed. The overwritten earlier snapshots cannot be recreated from a digest; this limitation remains recorded.

Final manifest creation now follows final receipts. Failure preservation now accounts member metadata within the byte allowance, preserves empty directories, rejects duplicate/nonregular members and re-inventories originals before removal. The real archive path accepted valid preservation and rejected six missing, changed, added, duplicate, special-member and budget-overflow controls. Budget failure retained the original tree. These repairs concern research evidence; no product algorithm, default, durability order or independent oracle changed.

Synthesis and next decisions

The service evidence is consistent with a finite resident plateau after repeated sealing, including concurrent consumers. It narrows the gap between isolated builder measurements and an application pipeline. The approximately one-second source-to-receive tail also shows why durable ACK latency alone is an incomplete end-to-end metric: collection cadence remains visible.

The lower Walk residency and CPU suggest query work belongs in the shared memory model. Query allocation history, pruning/materialization, cache state and trial order remain competing explanations. The smallest next discriminator is a balanced repeated plan comparison on frozen data and fresh processes, with query-off controls. This round does not establish which explanation dominates. Neither repeated source surveys nor another early-free optimization is needed before testing that distinction.

Persistent experimental state, staged checks and bounded delegation from the agent-method research were applied here. Two short inspection assignments, one implementation assignment and a brief independent review supported serial execution. There is no measured token-usage total or agent-efficiency ablation; no inference-cost saving percentage is claimed. Executable evidence, not agent agreement, decided the outcomes and exposed the stale-map defect.

Remaining readiness gaps include the unchanged 5,460-second soak, the full bounded-sealer acceptance matrix, retention/cancellation/overload interactions, mixed-signal service pressure including traces, supported installation and target-profile qualification. Successful raw service fixtures were reduced to their recorded clocks, hashes, query answers and grades after validation; complete raw replay is not retained for later independent re-grading. Failure preservation was exercised on controlled archive fixtures, not a failed live service case. A release or completion of the broader engineering goal is not claimed.

Verification and cleanup

The exact job commands, exits, durations and resource counters are in the coordinator records (local evidence omitted from stage). Service, recovery, classifier and archive-control jobs exited zero. Successful native scratch, fault states and archive-control scratch were removed; source/counterexample evidence and shared Cargo caches remain on the data drive.

cargo xtask checks --profile fast passed all seventeen checks, exit 0, under two build/test threads in service-fast-01 (local evidence omitted from stage). The three manual documentation checks passed, exit 0, in service-docs-01 (local evidence omitted from stage). No Bun hook or CI was enabled. A final report reconciliation recheck is recorded separately as service-docs-02 in the coordinator directory.

The closeout (local evidence omitted from stage) authenticated eleven completed job receipts and absent owned scratch, rejecting seven representative receipt defects. Those jobs charge 1,308.70 seconds; closeout's own duration and the final documentation recheck are separately charged. The goal-edit documentation work was explicitly imported once, with its actual 20.0634 seconds, so no historical allowance disappeared or reset. Before closeout's own duration, 2,479.47 seconds remained in the native-round allowance and 4,852.05 in the shared frontier. The unchanged 5,460-second soak cannot fit those remaining allowances and was not shortened or relabeled.

The same census records 36,450,304/98,799,616/31,776,768 allocated bytes in the native memory/query/coordinator categories, all within their unchanged caps. Later closeout/check metadata has its own coordinator census. No experiment processes or transient workload units remained after closeout. Shared Cargo artifacts were retained; no frozen binary copies or remote workloads were added.