Experiment formal installation release run 02 - kmosoti/FabricO11y GitHub Wiki
Frozen-candidate installation campaign โ 2026-10-12
Status: failed; campaign stopped for an owner product-fix decision. Epic #33 supersedes #32. #40 was closed with explicit owner authorization after its three build conditions were ticked. #41 remains open; B8 is incomplete. No package has been uploaded and no tag or release has been created.
Candidate and registered procedure
Frozen source: ca84d9823b5e3f8f39f40647cedffef0acff3aac; minimized source snapshot 9f60e8f55746a2cface2d9ed9fea7b2da1d40904, bound to the frozen revision by the #40 manifest and empty diff. Candidate Debian 0.1.0~alpha.1 SHA-256 241679e2d63d75f0ec628376baaff80a1070dc44991c7ca8c430dd9c66542219; candidate RPM 0.1.0-0.alpha.1 SHA-256 43ee9a896713e60ae9fd26d92757c16661c61dfcfd60e1bee48465058ce32bad. These are the original frozen packages, unchanged.
L1 Debian predecessor: registered historical payload re-versioned to 0.1.0~alpha.0, SHA-256 27a733e1bcef571ac9c1b110131b1afedefa6b3806233c24e639f6b0bf0e85c3.
Installation release protocol revision 4, with baseline revision 2, governs the unchanged acceptance scripts. The posted command plan registers lifecycle, fresh baseline and all six named mutation controls, one VM at a time. A12 explicitly selects the registered parallel fixture. Legacy-admin migration success would not satisfy production passkey/browser gates.
Actual lifecycle result
python3 -B tools/resource_group.py -- python3 -B tools/qualification/install/run-qemu.py --family debian --package /run/media/kmosoti/data/FabricO11y/results/installation-package-build/candidate-deb-a/fabrico11y_0.1.0~alpha.1_amd64.deb --source-commit 9f60e8f55746a2cface2d9ed9fea7b2da1d40904 --run-id rel-deb-life-01 --memory-stressor parallel --upgrade-from /run/media/kmosoti/data/FabricO11y/results/packages/predecessor/fabrico11y_0.1.0~alpha.0_amd64.deb
Executed from the clean detached frozen worktree on the owner's Fedora workstation. Harness and launcher exit 2, elapsed 306.112/309.568 seconds respectively. The predecessor baseline exited 0, RESULT fails=0 not_run=0; its fifteen executed checks are:
| Predecessor gate | Actual outcome |
|---|---|
| A1 | PASS |
| A2a | PASS |
| A2b | PASS |
| A3 | PASS |
| A3b | PASS |
| A4 | PASS |
| A5 | PASS |
| A6 | PASS |
| A7 | PASS |
| A8 | PASS |
| A8h | PASS |
| A9 | PASS |
| A10 | PASS |
| A11 | PASS |
| A12 | PASS |
Removal was deferred for the migration sequence. Installing the candidate succeeded, but its companion failed at startup; subsequent queries stayed empty. L1 failed, recorded ACCEPT LIFECYCLE FAIL phase=upgrade exit=1. L2 and A13a/A13b were not run. Candidate fresh A1โA12, A2c/A2d and P1 acceptance remain not run; the predecessor checks do not establish them. Fedora lifecycle/fresh baseline/F1โF4, all six mutation controls, and post-removal companion absence are not run. No #41 box is ticked.
The stopped guest journal shows the candidate listening and launching its companion, followed by fabric-node: Function not implemented (os error 38) and supervisor shutdown after the child exits 1. The cycle repeats under systemd's existing restart policy. Diagnostic installed binary hashes match #40 exactly: server ff44c3192f55da3b1f3b81bb81c8bceb92c02adc974ed668322a035403f386ea, node 70314cbd5b7b7cd0ec1594aefb42759ec538ffaaba55520c143ce37293cd44ce, CLI 0721d4dcbbf63cb7604054d6c608f18c74a8f67dc501d9954fe2f153100fd80b.
Cause and minimized reproduction
Observed source: managed companion startup calls the all-component secure log opener. Both installed units set RestrictSUIDSGID=yes. The systemd v257 seccomp implementation deliberately returns ENOSYS for openat2 under this restriction because its indirect argument cannot be filtered as a scalar.
A read-only diagnostic boot (rel-deb-life-diagnostic-01, collector exit 0) reproduced the same candidate companion/server failure and recovered the original boot journal. It changed no candidate byte or installed unit.
The first isolated probes (rel-deb-life-openat2-diagnostic-01) were inconclusive: both exited 2 before the syscall because fabricolly could not traverse the operator's private home. Those outputs remain retained. Collector exit 0 only attests collection and cleanup.
The corrected probes (rel-deb-life-openat2-diagnostic-02) use a root-owned mode-0644 Python fixture outside that private home. On the same Debian guest and exact regular server diagnostic log, both run as UID/GID 987 with NoNewPrivileges, memory high/max 128/256 MiB, zero swap, 32 tasks and a 15-second deadline. Only RestrictSUIDSGID differs. The ctypes fixture invokes x86_64 openat2 with the Rust opener's read-only flags (657408), mode 0 and RESOLVE_NO_SYMLINKS|RESOLVE_NO_MAGICLINKS (6).
| Probe | Actual exit | Actual syscall result |
|---|---|---|
| RestrictSUIDSGID=no | 0 | fd=3, errno=0 |
| RestrictSUIDSGID=yes | 1 | fd=-1, errno=38 (ENOSYS) |
This isolates the shipped hardening/log-opening conflict. It is a product acceptance defect, not a missing guest syscall or a qualification pass. The probes did not disable hardening in the installed candidate services. The campaign is stopped under handoff ยง7, pending owner authorization for a fix that preserves installed hardening and the path security contract. Any product fix requires a new merged candidate, rebuilt hashes and impacted gates rerun. No product, registered protocol, oracle or registered acceptance harness has been changed.
Host, containment and storage
Owner workstation fedora-blade, Intel i7-10750H, 6 cores/12 logical CPUs, 33,491,931,136 bytes RAM, kernel 7.2.8-200.fc44.x86_64; ext4 data drive mounted read/write. Clean detached worktree; owner's pre-existing modification in the usual checkout preserved.
All project workloads used the resource launcher, default 1,800-second outer deadline. QEMU's complete descendant cgroup was narrowed to high/max 7/8 GiB, zero swap; 2 vCPUs, 6,144 MiB guest RAM, 8 GiB disk, TCG. Guest: Debian 13.7, systemd 257.13, kernel 6.12.111+deb13-cloud-amd64, PID1 systemd, unified cgroup v2. Pinned dated image build 20261001-2618 matched the official HTTPS manifest and SHA-512 f46f0671a6e5bdec5291ab8972bae2f10e5408c2f64a74078f11efc2f06a436a9d0313ed50e0472542eeabf780e9f7c792ac0a314c6c20507fcd9fd81b468c3d.
Before every project invocation, preflight verified all six package hashes and both cached image hashes, recorded host/services and traversed the project tree without silently ignoring permission errors. Initial logical file-path usage 82,871,281,486 bytes plus reserved 9 GiB = 92,534,957,902 bytes, below 95 GB admission. Diagnostic reservations peaked around 93.02 GB; free physical disk exceeded 405 GB. This census counts hard-linked paths independently, unlike du's default inode accounting.
Lifecycle outer peak 5,616,852,992 bytes, zero host cgroup OOM/max/high events and zero swap. Intended A12 guest pressure recorded contained service/slice OOM group kills, task cap 128, unchanged service restart counts and continued delivery on the predecessor. Those intended guest events are distinct from outer containment. Three diagnostic collectors each stayed below 1.2 GB peak, returned 0 and had no outer OOM. Their individual probes retain their own exits. Heavy jobs ran strictly sequentially.
Controlled evidence and cleanup
Raw evidence is private under /run/media/kmosoti/data/FabricO11y/results, not publicly downloadable. Original VM result: installation-qemu/rel-deb-life-01; campaign archive: release-acceptance-ca84d982. The original receipt remains unchanged and its failed state is preserved. The exception path did not collect its service journal; the separately labeled diagnostic recovered that journal without turning the original run into a success.
The stopped 482,148,352-byte failure overlay has SHA-256 bff5435b3d23c1899c1fc6954a6f16b1926a0bf05e94b9e68fd2c9eed47a5032. Every diagnostic verified this same hash before and after its disposable copy-on-write boot. All QEMU processes and outer groups were stopped/reaped. Diagnostic scratch was removed after archival. The original failed fixture was moved intact into release-acceptance-ca84d982/retained-rel-deb-life-01 after verifying no live QEMU or mount reference, freeing its temporary scratch path. Its private key remains mode 0600; no credentials or whole VM directory are published. Cleanup receipt binds the old and archived paths. Launcher failure scratch remains in its controlled failure-evidence archive.
| Retained record | SHA-256 |
|---|---|
| receipt.json (rel-deb-life-01) | 46fddbfb5e8941976c46a1e7e3da3c07d23140ff5655a9dde4cfa7d8a28d694d |
| acceptance.txt (rel-deb-life-01) | ee78a20686d1f75da4a7c0b2d56332443c8903b0b057d340a650057523ebc127 |
| openat2-minimal-repro.py (release-acceptance-ca84d982) | f6f8b43af68c252e32fcbea6d4b15c02fdc26777a7461e0d3bd3ae82a1e8d8f7 |
| previous-server-journal.txt (rel-deb-life-diagnostic-01) | d81533909e22eecb7c8c989b8122e50ed87ea99dba23106223d83cc735ad2d67 |
| openat2-restrict-suid-no.txt (rel-deb-life-openat2-diagnostic-02) | cc041947936d136ab8c6ead2add597f66e9d2fc1936659a39b5fa59601db0af3 |
| openat2-restrict-suid-yes.txt (rel-deb-life-openat2-diagnostic-02) | c05c3223e602188f546328cc8324891b9b9ccd785b094003502eb55a8976c8b5 |
| receipt.json (rel-deb-life-openat2-diagnostic-02) | c44dc09bcc8a09a3c6e4f0e5d3d026a63e578bfa7d296bd54d8e3de8042e8117 |
| failure-retention-cleanup.json (release-acceptance-ca84d982) | 3a37c6cecff5bf63cbf9255608b6571c4b8c43a23dba31d190fec4816ae8e910 |
The controlled archive also contains exact collector commands, frozen scripts, host profiles, all resource receipts, raw read-only process snapshots and the persistent work queue. These are finite local migration measurements, not deployment qualification. Remaining uncertainty is the secure repair design and all unrun target/gate cells. #42โ#47 have not advanced.