Experiment benchmarks catalog mixed failure cleanup - kmosoti/FabricO11y GitHub Wiki

Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/catalog-mixed-failure-cleanup.md Original SHA-256: 28bdb19f55562c89d394ddcf50ea260188f2fcafc4bba314f754c1f62db13c10 Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Owned mixed-source failure archival and cleanup manifest

Status: root archival/cleanup completed; preparation agent performed no deletion. Owner: root coordinator. Failure catalog-mixed-sources-01, Cargo exit 101, remains failed after its fixture-only ordering correction and separate retry. Findings and command receipt (local evidence omitted from stage) identify the origin. No qualification or test expectation changes.

Completed receipt (local evidence omitted from stage) records byte_verified=true, scratch_removed=true, 4,223,853 removed source bytes and archive recovery/failure-catalog-mixed-sources-01.tar.gz. Root's scoped driver exited 0 after verifying each archived regular-file size/SHA-256 against source. Only this failed job was selected; SDK and shared build cache were preserved. The discovery-race failure root remains outside this completed cleanup.

Owned retained root: $FABRIC_DATA/evidence/fabric-work-af68cf661d834efb824302b338c05ef2. Its source was the launcher-owned work-ysgfd1bo scratch tree, relocated on failure and now archived/removed. Only this exact retained root was eligible; build caches, SDK venv, other evidence units and shared scratch are excluded.

Essential evidence to preserve:

  • Coordinator receipt, stdout/stderr, source archive, working-tree diff and registered protocol copies/hashes already under repository evidence.
  • Failed fixture origin.json, pre-IO expected ledger, all 60 numbered oracle directories (context/query/ledger/pages/stdout/stderr/exit) and any failure trace.
  • Entire fixture state: journal files, Segment 1/3 manifests and authenticated data/filter files. These reproduce the directory-order counterexample and coverage cut; do not prune state before byte-verified archival.
  • Outer resource-containment receipt naming this unit and relocated path.

Preferred approach: root runs a finite contained archival job using mounted data-drive scratch and no project workload. Walk the retained root read-only, reject unexpected symlinks/special files, and record every relative regular-file path, size and SHA-256 in an inventory. Preserve empty directories too. Create one compressed full-root archive in a new owned archive directory, record its size/SHA-256, then extract into a fresh owned data-drive verification directory. Compare extracted relative paths/sizes/SHA-256 against the original inventory. Include inventory/archive/verification hashes and exact exit codes in a receipt. An archive SHA alone does not verify archive contents against source files.

Before deletion, verify no descendant/process holds the owned fixture and that archive/inventory are durable outside the soon-deleted retained root. A mismatch, unsupported file type, archive failure, missing space or remaining process stops cleanup and preserves both source and diagnostic output. After a successful byte comparison, root removes only the exact retained root and its owned extraction scratch, then records nonexistence, archive location and remaining shared caches. Do not call failed scratch cleaned until these operations actually complete.

Suggested limits: existing 16/20 GiB no-swap launcher, coordinator 300 seconds, data-drive 16 GiB free reserve, at most 1 GiB archival/extraction scratch. Preserve archive under data-drive evidence with a repository inventory/receipt pointer; report actual bytes and cleanup. Success retry oracle evidence remains separately archived; it never replaces this failed state's provenance.

Future mixed-source executions now additionally archive sorted exact raw replay JSONL and its SHA-256. This does not retroactively add an artifact to run 02, and no repeat is requested solely to generate it.