Replacement native installation custody: first failure and repair

Status: source correction verified; native installation acceptance incomplete. All records below belong to the exact replacement packages, not historical8fb. DATA means /run/media/kmosoti/data/FabricO11y.

Actual first guest attempt

Failure record, actual merged ancestor binding. Clean harness da250b7ab135374d1231ab66f2a454e367884fec was verified ancestor of actual main 76d3da6fc91ebe155009a17acefe48077cfe9073 before launch. Product freeze e078ba23/source28bcdf7, exact DEB SHA256 2a555bec6334fde6774d21c414d7d42e041c39ca170e3a30fd3676e134b9f593.

Exact command, through registered e492bff0 native driver/ac372c38 owner and required resource launcher:

python3 -B tools/qualification/install/run-qemu.py --package DATA/results/installation-package-build/replacement-tls-01-deb-a/fabrico11y_0.1.0~alpha.1_amd64.deb --source-commit 28bcdf7ac833ab634d3b130b85cb2a0aa47eb7a8 --run-id replacement-deb-life-01 --family debian --upgrade-from DATA/results/packages/predecessor/fabrico11y_0.1.0~alpha.0_amd64.deb --memory-stressor parallel --native-custody-owner DATA/results/installation-native-custody/native-pipeline-replacement-01/native-owner-debian.json

Started04:15:35Z, stopped04:19:30Z on2026-10-16. Guest acceptance A1–A12 exited0, fails=0 not_run=0; lifecycle upgrade failed before first native snapshot with ValueError: unexpected or failed Spool input. Native transfer admission was incomplete. Actual outerexit2, stop_confirmed=true, QEMU cleanup=true. No second cell started. Failed original disk remains at DATA/scratch/installation-native-custody-replacement-deb-life-01/guest.qcow2; raw export/logs/receipts remain in DATA/results/installation-native-custody/replacement-deb-life-01.

Outer3d97d9c8 receiptSHA256 e5e6f75d9d4af52de50ecb8685d2d9a02d942bc35413921a75f90f5021cb5e3f; logSHA256 f2a81fb0afd4b2222e95ca4d711150393bb77856dfa4e4f9522702f8920d0924. Whole-DATA census admitted9GiB beneath95GB; host7/8GiB, no swap, VM2vCPU/6144MiB/8GiB, lifecycle180s, resource deadline1800s.

Retained stopped-disk diagnostics

Both diagnostics used read-only qemu-img conversion and owned sparse partition copies, original-hash before/after and strict18GiB reservation. No original writes or journal replay. Disposable copies removed in finally.

Diagnostic01 stopped04:27:41Z, outerexit0/stoptrue, inconclusive: debugfs returned0 despite a block-bitmap-checksum failure and filesystem not open. Its empty inventory/export is not evidence of input availability. Outer45465970 receiptSHA256 2d1c2beca23c0229df1ab0e5a147652d141d85f7ef91cbbf47caa3b05a31cdc2.

Diagnostic02 stopped04:32:28Z, outerexit0/stoptrue; debugfs1.47.3 -n bypassed metadata checksums on disposable copies only. It required actual nonempty inventory/export; retained sfdisk sector units512, root offset134217728, original ext superblock, versions/features and export hashes. The ext4 image needed journal recovery and exported ACK inode was stale. Therefore these are diagnostic bytes only, not a valid stopped snapshot or filesystem integrity. Private guest UID/GID987 mode0600 applied-config.json contained134bytes:

{"revision":1,"paused":false,"logs":["/var/log/fabric-accept/allowed.log","/var/log/fabric-accept/denied.log"],"metric_interval_s":15}

Source RemoteView independently defines those typed fields and synced rename. Both inspections verified original qcowSHA256 unchanged: 7d87a7288baa941cfe030eae9a00f87154c57e2f1aabc7bea084e0bcd014fc12. Diagnostic02 outera16f7786 receiptSHA256 e022ddbece8d6e855469c719ec482fe91577aef05452a91c3d0ff9245c8e712d; logSHA2560cf78fd146945bb7ac64ad567861f68b66ef3c7ff016761becbb91aac32a3d03. Original paths stopped-inspection-01 and stopped-inspection-02 retain every command/exit/stdout/stderr, semantic limitation and cleanup receipt.

Separate source correction and verification

Prospective commands. Protocol/control commit3fa2d77 precedes implementation8bda65e. Unknown/temp/failure markers still refuse. The reader validates exact typed, duplicate-safe/private bounded configuration metadata and retains original bytes. The finite #41 lifecycle has no configuration mutation; unchanged metadata joins remain required. Raw root export now precedes semantic parsing. Package bytes/oracles unchanged. Measured reader545/ceiling550; producer236/ceiling240; caller355/ceiling355.

Actual red command at clean3fa2d77:

env PYTHONPATH=tools/qualification/install:tools/qualification python3 -B -m unittest -v test_custody_native.AppliedConfigurationControls.test_original_applied_configuration_is_preserved test_custody_capture.CaptureAdmissionControls.test_original_raw_tree_survives_semantic_reader_refusal

Two methods exited1/stoptrue04:38:24Z: one actual sidecar-shape assertionfailure, one FileNotFoundError from absent pre-parser raw export. Neither is a detected invalid-custody negative. LogSHA256 eee78241a6a0a66272cb92b614b825b71e96e5f2ff8819286a2506dd1a1944b1; outere9f61cc5 receiptSHA256 da3c37ba04d35f2591847ef191a221f7f96cced9d7013fc1b5faf705b77f1d1f.

Actual green command at clean8bda65e44f428d3f6c99404f164a32cc6951b416:

env PYTHONPATH=tools/qualification/install:tools/qualification python3 -B -m unittest -v test_custody_witness test_custody_native test_custody_capture

62methods exited0, no skips, stoptrue04:38:39Z, allfive new methods plus historical methods in those modules retained. Required resource launcher,1GiB fresh reserve, whole-DATA census/no swap and clean source before/after; launcher scratch removed. LogSHA2563eb43037b3325b112b07f7c8142ea328f692671534ac6a1fb160d0b60eb7a72a; outer20d5f512 receiptSHA256 d79bc849b962b6c022d31541baa1dd880f5fe18521726562ebce024c2c20717d.

Fresh X224-repair-02 conditional pipeline retains the same four cases/resources with unique02runIDs. Actual merged runtime, fresh source/owner/driver bindings and exclusive scheduler clearance remain required before any restart. No source result supplies native acceptance.

Independent host configuration join

Prospective control, actual result. Only install/run-qemu.py and synthetic tests call admit_transfer; owner-guide configuration revisions are unaffected. Prospective28228da updates complete literal bundles to include original configuration and registers coherently rehashed absent/paused=true copies across every phase. On28228da the positive baseline passed, but both invalid copies were accepted: two actual assertion failures/no errors, exit1/stoptrue04:50:54Z. This independently exposed a missing host join; no arbitrary API/setup failure counted as negative detection. Redouter433b3f4b receiptSHA256 0cd3e97c731881395242d0b833db72aec9c626efe6660786b781dc2644224a79; logSHA256e01cc385619442e90cccabb25cdae7d9afe5a84e8c9789ad0aa47cc0022052b7.

Implementationad49505/115e4a4 adds explicit host presence, original byte/metadata and shared exact profile validation. Actual final pin 3b485bee022198086c0a70175751391a71f6f060 ran the same three custody modules: 63methods, exit0, no skips, stoptrue04:51:15Z. The preceding62-method result remains attributed to8bda. Greenouterfa74e644 receiptSHA256 9309f15235dcd46bdab0d393abb87f2ebd86c7e519eacc5bfb30dad2d81b69eb; logSHA256da43ad26f9650094d75b25a5e90f961a7bed2a74a7e8cdfbae7b2cab09902db2. Required launcher/strictcensus/fresh1GiB/no swap, clean source before/after and stopped launcher temporary cleanup remain recorded. Actual line counts: reader547(max550), producer238(max240), caller360(max365), combined598(max605), prospectively registered in the separate control commit. No package bytes, oracles, resource/deadline or historical controls changed.

The exact02cases/commands remain; only expected repaired runtime hashes changed in preserved conditional-plan-host-join.json SHA256 859f738ee431027fa5fc3efeeb65bca01ff2c5f206b2bc46ba86fc54daf82169. Merged-runtime/fresh-owner/driver binding and actual native execution remain pending; source success is not package acceptance.

Replacement Debian02 removal counterexample and correction (2026-10-16)

PR74 merged as 3494391403d33734debdd521085240ac2a5a337f; clean native source c14092e89f86cb8e35d57b6aa3fafd646bfa0863 was independently admitted as its ancestor. First cell replacement-deb-life-02 ran under the sole scheduler lease and the exact native02 owner/driver/plan. Guest A1–A12 exited0, upgrade/reboot gates passed, and all four original stopped native snapshots transferred and were independently admitted (bundle SHA256 9da9c35e74375cac4e27b3bdfdc4b1a7ea9137d9bf160b4b273bee281c8a1687,2334720 bytes). The complete cell nevertheless failed exit2 after package removal: ACCEPT A13a FAIL stopped-state observation rc=4: 'inactive\ninactive'. This is partial custody evidence, not native installation acceptance. No A13a/purge/full pass is claimed. Other three02 cells were not run.

Actual outer stop was confirmed at06:00:59Z, QEMU cleanup true; failed guest scratch and original raw remain retained. Outer receipt SHA256 0598fd80c79bad4a223d85a342604c1d0ea3689eb7b85b66259f341dc5713a69; log SHA256 34b93a67aad3b1d5800083e8076099367609c003c5d670f371400b5fe496dbc2. Actual issue41 result. All01/02 failure evidence remains immutable.

Local installed systemctl documentation describes LSB4 as no such unit and cautions against inferring state from LSB status alone. A never-existing user-unit probe on local systemd259.9 actually observed is-active4/inactive and show0/not-found/inactive/dead/MainPID0/empty ControlGroup. This is local mechanism evidence only; guest257 must provide its own original observations. Probe exit0 and outer stop06:06:01Z.

Separate prospective trust/control commit 2d479243315a8b79f0ba5e523dab499b87eacb69 registers complete independent removed-unit observations and synthetic counterexamples; implementation 08da2f7 and count-document child 56868710e4553e6142ae92734d1a524aa9d5cddf deploy/freeze a90-line shared observer. Both guest callers collect exact per-unit Id/load/active/substate/MainPID/ControlGroup, full ps/proc executable inventory, and project cgroup descendants. Missing units require strict show0/not-found/inactive/dead/noPID/no-cgroup. QEMU retains/regrades original JSON on the host. Native owner source joins include the helper. Product/package identity, independent delivery/query oracles, resource/time limits and all other gates remain unchanged. Caller465/470, producer238/240, reader547/550; prospective source-ceiling amendment is in the separate trust commit.

Actual source verification:

• Old-guard regression at2d47924:1 assertion failure/exit1, specifically valid removed-unit rc4 rejection, not an API/setup error.
• Full installation source discovery at5686871:81 tests/exit0; isolated production loader:1 test/exit0.
• Frozen Bun install and direct documentation links:exit0. The first checker failed because its child bun executable was absent from PATH; those spawn errors are setup absence, not detected negatives. Hooks were not run in that failed pipeline. Overall exit1/outerstop06:16:33Z retained.
• Retried only checker and hooks using the unchanged, previously proven explicit Bun PATH/cache wrappers:55/55 acceptance probes and10 hook tests, each exit0 with outer stop06:20:40Z/06:20:50Z. Source remained clean.

Exact original commands, source pins, log/receipt hashes, cleanup and outer stops are retained in DATA/results/installation-native-custody/removal-observer-repair-01/actual-source-results.json, with source and retry plans alongside. Each command ran serially under the required launcher,1GiB admission reserve and strict namespace census. No Rust-suite rerun was substituted or needed for this Python/shell harness change.

Ready PR76 preserves the failures. Fresh native03 preflight stages unique03 outputs and the same four cases with exact seven-path owner source joins. It is NOT RUN: required hosted checks, actual merged-main ancestor, fresh admission/board and sole scheduler CLEAR remain mandatory. No historical package refreeze or earlier result relabeling.

Native03 actual Debian success / Fedora readiness counterexample

PR76 hosted fast and extended checks succeeded at 56868710, merged as 4ce8f4b8178310237fb160045d9a4b8841aa8300 on 2026-10-16 at 07:12:32Z. Clean source ancestry, exact source/tool/owner/package/driver hashes and the full board were independently checked before launch. Actual admission.

replacement-deb-life-03 completed with exit 0: initial acceptance, four original stopped custody phases, independent host admission, strict removal unit/process/cgroup observation and purge. Bundle: 2385920 bytes, SHA256 4be4af4858a663756a84c4e5edffe62eac8958cdc58738a541d2f7efbf4e9ee6. Outer stop: 07:19:45Z; QEMU cleanup and scratch removal confirmed. Outer receipt 0e08dd36654bbc67f5431851e38f6b4b1e07494a0961c2c3ecf1d07ad93787b5; log aa49fff36cd2f383735899cebd7462b90203702a317518b770b5730acf3f3fa5. Its original source path DATA/scratch/native-removal-observer remains clean at 56868710; accepted source, owner and receipts are unchanged.

replacement-rpm-life-03 failed overall with exit 2, despite initial guest acceptance exit 0. The first drain query after unit start returned exit 2, empty stdout and exactly fabricctl: transport: io: Connection refused. The old helper raised before the existing 40-second readiness retry could run. Four stopped phases were incomplete and not admitted. Outer stop: 07:27:56Z; QEMU cleanup confirmed and original failed disk retained. Outer receipt 6b62942bd1fe15fc5a2ee038ddea14800e2885abc88cde4e37dd32e4d189036a; log 7268aea0dc8442e9a52243c8bba05e9503855d6e8307659abca445d3cf681953. The two dependent custody refusal cells were NOT RUN. Exact commands, receipts and stops are in the actual #41 checkpoint and DATA/results/installation-native-custody/readiness-repair-01/native03-actual-results.json.

Separate prospective control/protocol commit 6c8f1c93 retains the actual trace. Implementation 36657085 leaves Guest.query unchanged. A local wait retries only the exact first-query argv, exit 2, empty stdout and observed refusal stderr; every raw attempt is retained. Partial pages and other errors remain fatal. Commands and late successful reads are bounded by the original single absolute 40-second/phase deadline, restored in finally. Final full answers, raw row conservation, ACK/process joins and independent oracles are unchanged. Measured producer 258/265, caller 465/470, reader 547/550; no product refreeze.

Actual checks through the required launcher:

• Old guard: one method, two assertion failures (drain and reboot), exit 1; outer stop 07:35:58Z. These are mechanism counterexamples, not API absence.
• Corrected source: 86 installation tests and one production loader test, frozen dependencies, documentation links, 55/55 checker probes and ten hooks all exited 0. Outer stop 07:36:31Z; receipt dc608d1bd371f720de436029cd8e3a2b1bb5ce5f653eb761134912a8f8a12a86, log cdc37f6bee173122e935f99f7e2872b06b5214719fadeb4bd50e8980a5cbe7b0.

Exact commands and receipts are in readiness-repair-01/plan.json and actual-source-results.json. Ready PR77 hosted checks are pending at this checkpoint. Fresh native04 bindings use separate DATA/scratch/native-custody-readiness at 36657085 for Fedora and the two original custody refusals. They are NOT RUN until hosted checks, actual main ancestry and fresh scheduler clearance. Valid Debian03 is not rerun for this readiness correction; its original successful source and observations remain bound.

The coverage audit also confirmed that all three packaging mutations on both families remain mandatory and unrun on the replacement candidate. X240 registered three RPM repack/manifest prerequisites and six VM controls at 07:31:40Z, with a new 60-minute window eligible 08:31:40Z absent findings. No historical timer waiver is assumed. Required defects remain root-user → A6, no-collision-check → A2a+A2b, no-memory-max → A8; unrelated errors cannot count. The repack preserves canonical ELF, console and unrelated regular payload bytes, without Rust recompilation or product refreeze. Actual RPM hashes and manifests must be bound before dependent VM execution. #41 remains incomplete pending fresh Fedora, custody refusals and all six packaging mutations.

Final actual installation closeout — 2026-10-18

The exact replacement candidate e078ba23b543126011a2136dd1f7e98ca65370eb, package source 28bcdf7ac833ab634d3b130b85cb2a0aa47eb7a8, completed its registered finite Debian 13/Fedora 44 installation/lifecycle cells, two custody producer refusals and all six named package-mutation controls. This is owner-lab VM evidence, not a release/tag or general deployment qualification.

Canonical DEB SHA256 2a555bec6334fde6774d21c414d7d42e041c39ca170e3a30fd3676e134b9f593; canonical RPM SHA256 12c03f92e96c7d1a848a5198f017c21a1e17d485cdfca7feb8464c3a425971c1. No canonical product package was rebuilt to resolve harness failures. The three separately mutated RPMs retain their original strict payload-verification manifests and stopped build receipts, bound in the recovered-artifact record.

The deterministic actual evidence index is DATA/results/installation-native-custody/installation-actual-terminal-evidence-01.json, SHA256 0df8844154a879e416a3fb1181bb4e350a4c396fbd54262d699945cb30879c20 (168775 bytes). DATA is /run/media/kmosoti/data/FabricO11y. It records each cell's complete driver/inner argv, execution-plan and source identities, owner/driver/log/raw hashes, expected and actual guest outcomes, actual host/resource exit and whole stop, and cleanup. It is an index of the existing graders and original bytes, not a new acceptance gate.

Actual cell	Harness	Required outcome	Guest / host exit	Whole outer stop UTC
custody-replacement-deb-life-03	5686871	positive	0 / 0	2026-10-16T07:19:45.908814+00:00
custody-replacement-rpm-life-04	3665708	positive	0 / 0	2026-10-17T23:36:06.789031+00:00
custody-replacement-deb-pending-refusal-04	3665708	pending-not-produced	0 / 0	2026-10-18T00:01:42.167930+00:00
custody-replacement-deb-writer-refusal-04	3665708	live-writer	0 / 0	2026-10-18T00:06:54.256714+00:00
replacement-deb-mut-root-user-01	5686871	A6	1 / 0	2026-10-18T00:14:30.433962+00:00
replacement-rpm-mut-root-user-01	5686871	A6	1 / 0	2026-10-18T00:23:32.168569+00:00
replacement-deb-mut-no-collision-check-02	5686871	A2a, A2b	1 / 0	2026-10-18T00:54:18.344821+00:00
replacement-rpm-mut-no-collision-check-01	5686871	A2a, A2b	1 / 0	2026-10-18T01:03:44.148318+00:00
replacement-deb-mut-no-memory-max-01	5686871	A8	1 / 0	2026-10-18T01:09:15.890918+00:00
replacement-rpm-mut-no-memory-max-01	5686871	A8	1 / 0	2026-10-18T01:17:39.687293+00:00

The retained Debian03 and Fedora04 positives each recorded the 17 installation/removal PASS IDs: A1, A2a, A2b, A3, A3b, A4, A5, A6, A7, A8, A8h, A9, A10, A11, A12, A13a and A13b. L1 recorded the strictly older predecessor-to-candidate upgrade with exact rows, configuration/account and console preservation; L2 recorded an actual changed boot identity and retained state. Debian purge removed its fixture state/configuration. RPM removal preserved state/configuration before the registered explicit guest cleanup. Both strict removal observers accepted with no remaining project process; the candidate server's dedicated Spindle and standalone edge Spindle are also evidenced by original journal/ACK records and complete drain/reboot query answers with both fabric-server-self and spindle-1 freshness. All four stopped custody phases were independently admitted for each positive.

Fedora's original acceptance output records F1–F4 PASS: enforcing SELinux at baseline, after delivery/restart/removal and across reboot, with no Fabric-process AVC observed by the existing audit check. These are the recorded original acceptance observations; separate guest audit files are not claimed independently exported.

Both producer controls ran predecessor baseline setup, then produced their exact named prepare-phase refusal: missing-committed-pending and stop observation: live or unreadable writer. The independently admitted exports contain only the prepare phase. No candidate upgrade/mutation occurred after those refusals. They are expected negatives, not positive lifecycle trials.

Every package mutation produced guest exit 1, all required named failed checks, no NOT-RUN checks, error=null and mutation_rejection_confirmed=true; the host grader and outer workload exited 0. Root-user controls also observed the related ownership/denied-log failures; collision controls also observed the related root-identity/account failures. Memory controls failed A8 only. Each intentional package and raw evidence was preserved before successful owned scratch cleanup. Actual QEMU and whole-workload stops are recorded for every accepted cell; no native workload remained at the final terminal boundary.

The original interrupted Debian collision01 remains interrupted exit 2 and receives no acceptance credit: a strict census raced successful owned scratch removal. Its raw named detection and all receipts remain unchanged. The separately registered exact-owned complete-rescan correction, independently exercised against real unlink/root-removal and unknown-path/permission/I/O controls, preceded fresh collision02. The corrected complete owner argv binding also preserves the earlier malformed administrative plan as NOTRUN. Earlier Debian/Fedora failures remain historical evidence; none was relabeled.

All native jobs retained the original 9 GiB storage reservation, strict full-DATA 95 GB admission/monitoring, 7/8 GiB host memory/no swap, two guest vCPUs, 6 GiB guest memory, 8 GiB guest disk and 1800-second outer bound. Runtime568/366 attribution remains per cell. No independent oracle, expected negative outcome, product/package bytes or original native acceptance requirement was weakened.