Experiment benchmarks rca journal findings - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/benchmarks/rca-journal-findings.md. Original SHA-256:fa79193aa46579fa5e0fecf43ee78fa1cac5ad213e420ceab3aa5655a3461b0c. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Native RCA journal and restart findings
Status: finite native evidence/query checks completed on 2026-10-09. One pilot
and fourteen case/plan cells passed. This extends the
tabletop design (local checkout reference docs/experiments/benchmarks/rca-flow-design.md; not present at origin/main) and
application use-case plan; no causal
investigator, application/browser adapter or deployment qualification is claimed.
Executed scope
The pilot protocol (local checkout reference docs/experiments/benchmarks/rca-native-pilot-protocol.md; not present at origin/main), committed as 4bcb2d8, ran
rca-native-01 using the prepared late-span fixture and Walk. Its 25 complete
query chains matched the unchanged Python oracle, three corrupted-answer
controls were rejected, and all three producer Batches recovered exactly.
Pilot summary (local evidence omitted from stage),
oracle verdicts (local evidence omitted from stage),
controls (local evidence omitted from stage).
Only then did the matrix protocol (local checkout reference docs/experiments/benchmarks/rca-journal-matrix-protocol.md; not present at origin/main), committed
as 9eb7cc9, expand to seven cases × Scan/Walk × three observations: initial,
after delivery and after graceful restart. There are 42 observation sets
of eight full queries, plus fourteen held initial trace chains, for 350
complete chains. Every cell and its enclosing coordinator/launcher exited 0.
The two ordinary corruption controls ran in every cell, and the late-span
corruption ran in both late-span cells: 30 rejected controls.
| Fixture | Scan evidence | Walk evidence | What the actual answer checks establish |
|---|---|---|---|
| Pool wait | 01 (local evidence omitted from stage) | 01 (local evidence omitted from stage) | 800 ms acquisition span, 20 ms dependency span and occupancy/capacity values survive delivery/restart |
| Downstream wait | 02 (local evidence omitted from stage) | 02 (local evidence omitted from stage) | 5 ms acquisition and 800 ms dependency evidence remain distinct from a coincident deployment log |
| Counter reset | 03 (local evidence omitted from stage) | 03 (local evidence omitted from stage) | The actual rate answer is 6/s then reset/null, rather than negative throughput |
| Missing evidence | 04 (local evidence omitted from stage) | 04 (local evidence omitted from stage) | Missing dependency spans and the explicit gap persist; node freshness and complete retained scans still do not imply complete instrumentation |
| Late span | 05 (local evidence omitted from stage) | 05 (local evidence omitted from stage) | Held old chain stays at two spans; fresh query sees three; restart retains all three |
| Clock skew | 06 (local evidence omitted from stage) | 06 (local evidence omitted from stage) | Parent identity, the two-second timestamp offset and 800 ms local duration survive exactly |
| Host-pressure confounder | 07 (local evidence omitted from stage) | 07 (local evidence omitted from stage) | Low available memory and incomplete application-span coverage remain observable together; a causal conclusion was not graded |
For cases without a late Batch, the after-delivery observation is an unchanged population control. Each cell replays one initial Batch identically, obtains fresh-process durable dumps before and after restart, compares producer/recovery identity and SHA-256, and checks all response pages/envelopes independently. All fifteen unique fixture Batches were exercised under each plan in isolated server states. Duplicate delivery did not add another recovered logical Batch.
What a useful investigation can now rely on
The native late-span answers reproduce the proposed investigation sequence: initially only the request and acquisition spans are present; an investigator lacks dependency evidence. A new 800 ms dependency span then becomes available to fresh queries, while the previous page chain keeps its earlier boundary. After restart, that added evidence remains queryable. The result supports revision of an investigation without retroactively changing its old evidence.
The other cases establish that Fabric returns the intended discriminating facts. They do not establish that an agent or human will draw the correct conclusion. Pool saturation still needs a controlled intervention to identify its cause; low memory or a deployment timestamp remains correlation. Blind interpretation, abstention calibration and application interventions remain separate experiments.
Resource and timing observations
The driver body took 0.721–0.762 s per matrix cell; coordinator admission and source accounting add overhead recorded separately. Observed server CPU was 0.15–0.17 s per cell across both server processes. Process high-water RSS was 22.6–23.5 MiB. Maximum sampled fixture disk use was 9,988 bytes; all matrix HTTP responses together were 435,574 bytes. These tiny warm fixtures are correctness probes, not a capacity curve or service-latency qualification.
Child cgroup peaks, process RSS and aggregate laboratory peaks are separate
observations. In these cells, child charged-memory peaks were smaller than RSS;
do not substitute one for the other or infer a cold-start memory requirement.
Source/binary hashing ran before the service and the code pages were reused.
No OOM or swap occurred, and the server's page-aligned 4 decimal GB limit,
two-CPU quota and task limit were verified. The whole laboratory stayed within
its separate 20 GiB cap on the data drive. Exact limits and final counters are
in every environment.json and cgroup-final.json.
The pilot's late delivery took 6.14 ms from send to observed ACK; its first completed matching fresh query was observed 29.15 ms after ACK. These are single observations, not p99 estimates. Query order contributes to the second number; it is an upper bound on when the driver observed visibility, not the precise instant data became available. A null visibility entry for the initial dependency Batch means no recipe returned a row from that Batch; it is not evidence of loss. ACK observations (local evidence omitted from stage), visibility observations (local evidence omitted from stage).
Each cell stores query pages, exact producer inputs, two recovery dumps, oracle
verdicts, rejected corruptions, source/binary hashes and resource observations.
The driver (local checkout reference tools/bench/labs/rca/native.py; not present at origin/main) remains a controller, not
a blinded investigator. Original source snapshots stay beside each result.
The independent oracle was not edited. All owned native fixture trees and
server child groups were removed after retained evidence copies were hashed;
launcher receipts confirm outer temporary cleanup. The earlier read-only
preflight failure remains in the original design record.
The matrix summaries retained a stale pilot-only limitation string saying
"one Walk plan", even in Scan cells. Their explicit case/plan fields,
captured commands and response evidence identify the actual execution. Those
historical files remain unchanged; the driver now formats this descriptive
string from its arguments. No query verdict or expected answer changed.
Remaining product work
These tests use direct TLS Batch delivery, custom synthetic Gauge/Sum records and journal-backed queries. They do not exercise an application SDK → Spindle trace-export path, arbitrary application metric intake, browser queuing/loss, Nomos integration or sealed publication. The broader 56-episode design includes a published-storage phase that this 42-observation matrix deliberately lacks.
Next discriminators are actual publication with bounded padding, real local application export through Spindle, and an investigation client that preserves coverage while comparing applied configuration and website outcomes. Keep correct query results, defensible interpretation and a usable interface as separately verified responsibilities.