Experiment benchmarks rca journal findings - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/rca-journal-findings.md. Original SHA-256: fa79193aa46579fa5e0fecf43ee78fa1cac5ad213e420ceab3aa5655a3461b0c. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Native RCA journal and restart findings

Status: finite native evidence/query checks completed on 2026-10-09. One pilot and fourteen case/plan cells passed. This extends the tabletop design (local checkout reference docs/experiments/benchmarks/rca-flow-design.md; not present at origin/main) and application use-case plan; no causal investigator, application/browser adapter or deployment qualification is claimed.

Executed scope

The pilot protocol (local checkout reference docs/experiments/benchmarks/rca-native-pilot-protocol.md; not present at origin/main), committed as 4bcb2d8, ran rca-native-01 using the prepared late-span fixture and Walk. Its 25 complete query chains matched the unchanged Python oracle, three corrupted-answer controls were rejected, and all three producer Batches recovered exactly. Pilot summary (local evidence omitted from stage), oracle verdicts (local evidence omitted from stage), controls (local evidence omitted from stage).

Only then did the matrix protocol (local checkout reference docs/experiments/benchmarks/rca-journal-matrix-protocol.md; not present at origin/main), committed as 9eb7cc9, expand to seven cases × Scan/Walk × three observations: initial, after delivery and after graceful restart. There are 42 observation sets of eight full queries, plus fourteen held initial trace chains, for 350 complete chains. Every cell and its enclosing coordinator/launcher exited 0. The two ordinary corruption controls ran in every cell, and the late-span corruption ran in both late-span cells: 30 rejected controls.

Fixture Scan evidence Walk evidence What the actual answer checks establish
Pool wait 01 (local evidence omitted from stage) 01 (local evidence omitted from stage) 800 ms acquisition span, 20 ms dependency span and occupancy/capacity values survive delivery/restart
Downstream wait 02 (local evidence omitted from stage) 02 (local evidence omitted from stage) 5 ms acquisition and 800 ms dependency evidence remain distinct from a coincident deployment log
Counter reset 03 (local evidence omitted from stage) 03 (local evidence omitted from stage) The actual rate answer is 6/s then reset/null, rather than negative throughput
Missing evidence 04 (local evidence omitted from stage) 04 (local evidence omitted from stage) Missing dependency spans and the explicit gap persist; node freshness and complete retained scans still do not imply complete instrumentation
Late span 05 (local evidence omitted from stage) 05 (local evidence omitted from stage) Held old chain stays at two spans; fresh query sees three; restart retains all three
Clock skew 06 (local evidence omitted from stage) 06 (local evidence omitted from stage) Parent identity, the two-second timestamp offset and 800 ms local duration survive exactly
Host-pressure confounder 07 (local evidence omitted from stage) 07 (local evidence omitted from stage) Low available memory and incomplete application-span coverage remain observable together; a causal conclusion was not graded

For cases without a late Batch, the after-delivery observation is an unchanged population control. Each cell replays one initial Batch identically, obtains fresh-process durable dumps before and after restart, compares producer/recovery identity and SHA-256, and checks all response pages/envelopes independently. All fifteen unique fixture Batches were exercised under each plan in isolated server states. Duplicate delivery did not add another recovered logical Batch.

What a useful investigation can now rely on

The native late-span answers reproduce the proposed investigation sequence: initially only the request and acquisition spans are present; an investigator lacks dependency evidence. A new 800 ms dependency span then becomes available to fresh queries, while the previous page chain keeps its earlier boundary. After restart, that added evidence remains queryable. The result supports revision of an investigation without retroactively changing its old evidence.

The other cases establish that Fabric returns the intended discriminating facts. They do not establish that an agent or human will draw the correct conclusion. Pool saturation still needs a controlled intervention to identify its cause; low memory or a deployment timestamp remains correlation. Blind interpretation, abstention calibration and application interventions remain separate experiments.

Resource and timing observations

The driver body took 0.721–0.762 s per matrix cell; coordinator admission and source accounting add overhead recorded separately. Observed server CPU was 0.15–0.17 s per cell across both server processes. Process high-water RSS was 22.6–23.5 MiB. Maximum sampled fixture disk use was 9,988 bytes; all matrix HTTP responses together were 435,574 bytes. These tiny warm fixtures are correctness probes, not a capacity curve or service-latency qualification.

Child cgroup peaks, process RSS and aggregate laboratory peaks are separate observations. In these cells, child charged-memory peaks were smaller than RSS; do not substitute one for the other or infer a cold-start memory requirement. Source/binary hashing ran before the service and the code pages were reused. No OOM or swap occurred, and the server's page-aligned 4 decimal GB limit, two-CPU quota and task limit were verified. The whole laboratory stayed within its separate 20 GiB cap on the data drive. Exact limits and final counters are in every environment.json and cgroup-final.json.

The pilot's late delivery took 6.14 ms from send to observed ACK; its first completed matching fresh query was observed 29.15 ms after ACK. These are single observations, not p99 estimates. Query order contributes to the second number; it is an upper bound on when the driver observed visibility, not the precise instant data became available. A null visibility entry for the initial dependency Batch means no recipe returned a row from that Batch; it is not evidence of loss. ACK observations (local evidence omitted from stage), visibility observations (local evidence omitted from stage).

Each cell stores query pages, exact producer inputs, two recovery dumps, oracle verdicts, rejected corruptions, source/binary hashes and resource observations. The driver (local checkout reference tools/bench/labs/rca/native.py; not present at origin/main) remains a controller, not a blinded investigator. Original source snapshots stay beside each result. The independent oracle was not edited. All owned native fixture trees and server child groups were removed after retained evidence copies were hashed; launcher receipts confirm outer temporary cleanup. The earlier read-only preflight failure remains in the original design record.

The matrix summaries retained a stale pilot-only limitation string saying "one Walk plan", even in Scan cells. Their explicit case/plan fields, captured commands and response evidence identify the actual execution. Those historical files remain unchanged; the driver now formats this descriptive string from its arguments. No query verdict or expected answer changed.

Remaining product work

These tests use direct TLS Batch delivery, custom synthetic Gauge/Sum records and journal-backed queries. They do not exercise an application SDK → Spindle trace-export path, arbitrary application metric intake, browser queuing/loss, Nomos integration or sealed publication. The broader 56-episode design includes a published-storage phase that this 42-observation matrix deliberately lacks.

Next discriminators are actual publication with bounded padding, real local application export through Spindle, and an investigation client that preserves coverage while comparing applied configuration and website outcomes. Keep correct query results, defensible interpretation and a usable interface as separately verified responsibilities.