Experiment formal installation custody controls plan 01 - kmosoti/FabricO11y GitHub Wiki
Prospective stopped Spool custody controls
Registered trust commitbf70ee0 on stacked checker fcd57f6. The independent custody adapter is absent; the named initial red is mechanism availability only. Five literal specific controls are NOT RUN until implementation. No VM/native upgrade or product failure is claimed. X089 requests pre-flight for the exact availability command.
pre-flight: #41 custody mechanism-availability control at cleanbf70ee0ff56ffeb2617d17dcddec675d7fe19f73 (separate trust +240test/+30docs, production0). Raw fixture outcomes registered before adapter; oracles unchanged.
Exact outer: python3 -B META/run-step-installation-custody-availability-red.py installation-custody-availability-red-01 1073741824 python3 -B tools/qualification/install/test_custody_witness.py MechanismAvailability.test_stopped_snapshot_and_conservation_are_available; META=DATA/results/candidate-build-fresh-8fb6f413179b-01, DATA=/run/media/kmosoti/data/FabricO11y, cwd=DATA/scratch/installation-custody-witness.
DriverSHAcf77916f4a9085ec8543e2c9b2cf6ad16c6639898c66e2a156bf66ed2d752e44; actual inner default python3 -B tools/resource_group.py -- python3 -B tools/qualification/install/test_custody_witness.py MechanismAvailability.test_stopped_snapshot_and_conservation_are_available; serial/default16/20GiB/no-swap/1800s/full1GiB/fresh95GB admission.
Expected red is absent independent stopped_snapshot/conserve adapter, NOT a product/corrupt-frame bug. Five specific tiny synthetic CustodyWitnessControls remain NOT RUN until adapter exists; no Rust/oracle encoder/systemd/native/VM calls or actual Spool mutation.
TestsSHA944a3fb9f1da6a6efa3d0ef57fc0258d324e193d60efcce7039229afda9270b7/protocolSHA1b79d6410f3de57d5beb473b156c1cdb2fc5ca55663d550701260e4c68eef775; corrected private planSHAba753ba20505cf2d84918dc818304e7a8a679267e5d40e00fd475ec40187cf10.
Later adapter budget~300production lines, unchanged oracle reuse+separate full envelope/wire/u64/terminalACK/fulllatestCursor/prefix/Btrfs guards and exact candidate PID/unit/boot/time ACK binding. Native integration/credential/config/reboot witnesses and final merged harness/19VM cells stay pending.
Shared Cargo-cache deletion staysHOLD perC047; these controls do not admit #41 VM or #43 and never alter source/package/receipt/secret evidence.
# Prospective installation Spool and unread-backlog upgrade witness
Private preparation only. No adapter, test, shell change, import, compile or VM
has run for this plan. Olo must review/register the prospective controls and
protocol, capture a meaningful contained red, implement the witness separately,
and bind the final clean merged harness before a fresh native lifecycle cell.
The custody lane starts from checker evidence commit
`fcd57f62581a243fad34c61a258acb760cbdfce8`, with checker production repair
`ce387f75610bc4bac82955fbb722c9d61e4fcbc1` already verified by Olo. The new
`milestone/installation-custody-witness` lane is prospective and stacked; none of
those checker results establish custody-adapter or native-upgrade acceptance.
## Objective and observed gap
Issue #41 requires upgrade to preserve configuration, credentials, Spool and
history. Product contract custody rules require the same durable Strand identity
and exact encoded bytes until ACK; unread log bytes remain source backlog.
`acceptance.sh:342–357` exercises an outage and node restart, then drains the
outage line before upgrade. `lifecycle.sh:42–74` checks three already delivered
rows, `*.conf`, account identity and console hashes. It has no pending batch at
the package boundary and does not separately hash credential files.
This supplement keeps all existing A1–A13/F1–F4/L1/L2 expectations. It adds a
predecessor-produced pending source and an unread source suffix across L1.
Historical results do not count for the new candidate. File hashes across a
stopped native install establish retained bytes; actual replay, ACK and a query
after a distinct reboot establish recoverable delivery.
## Exact inputs observed during preparation
All paths below are under `/run/media/kmosoti/data/FabricO11y` (DATA). Rehash and
revalidate them at admission; this preparation is not a validator result.
| Input | Exact binding |
| --- | --- |
| Frozen package revision | `8fb6f413179b094288316e25845c137440ba23fa` |
| Source epoch | `1791841113` |
| Actual minimized package source snapshot | `73ff77867d10ef521ca985b094e1d177f7786b77` |
| Source manifest SHA256 | `534b601f904ce80f50cc545be8b8b40f47f8940975fd4cbf2e2e8c8cb54f2cff` |
| Source bundle SHA256 | `fc83ed1435b6ef8afadade2b7978c433d47d6cbdb9ad4a95c9ff57b25a07f1d0` |
| DEB A receipt | `results/installation-package-build/fresh-8fb6f413179b-01-deb-a/receipt.json`; SHA256 `b10a4e4c2d83d8dbed7c1d6e74673fd377a83123fc1bce31fffc97e034beb9af` |
| DEB A package | same directory, `fabrico11y_0.1.0~alpha.1_amd64.deb`; SHA256 `8edfc7709f4b5dc3da01c00a1d8c442470298e96f8c8e502cfb14e8c9e713e92` |
| RPM A receipt | `results/packages/fresh-8fb6f413179b-01-rpm-a/receipt.json`; SHA256 `eadbbd217804efdeae217fe04d4800c9482cc23650942860df4d756e45ac8434` |
| RPM A package | same directory, `fabrico11y-0.1.0-0.alpha.1.x86_64.rpm`; SHA256 `a54f55e7bf1b2230e026b49d6b86023778a9648ae49b11c6bcb0d558db34a840` |
| Exact helper manifest | DEB A directory, `qualification-helpers/manifest.json`; SHA256 `604fd0b90f80f061859967f967347375f1da20417fd576519485e8d7c58c43e6` |
| Exact `spool_dump` | same helper directory; SHA256 `e61babbb57249e25639e8413ce87752ec1a408cd7ad5d0b8c35d892a52b602bf` |
| Optional diagnostic `server_dump` | same helper directory; manifest SHA256 `68daf3fdbaed5579ff672c4916651905ec18e10d440b4c391931ff963592312f`; actual bytes must be rehashed before use |
| Historical Debian predecessor | `results/packages/predecessor/fabrico11y_0.1.0~alpha.0_amd64.deb`; SHA256 `27a733e1bcef571ac9c1b110131b1afedefa6b3806233c24e639f6b0bf0e85c3` |
| Constructed historical-binary RPM predecessor | `results/packages/predecessor/fabrico11y-0.1.0-0.alpha.0.x86_64.rpm`; SHA256 `3c0a64a10b088a9add12e8837c12ad2f8d6aad01c3ce3dab55cfd75baad469a3` |
| Historical source artifact binding | registered original Debian SHA256 `ac71995b3c8b88cbb6f7dd05d9854e347b7a783caadc1b602f6590d6460f4a2f`; Debian reversion preserves payload/scripts, RPM construction preserves historical binaries |
| Candidate CI admission | `results/candidate-build-fresh-8fb6f413179b-01/candidate-admission.json`, #40 record `https://github.com/kmosoti/FabricO11y/issues/40#issuecomment-6133313107`; this is CI admission only |
| Pending runtime harness | `INSTALL_HARNESS_REPO` and full `INSTALL_HARNESS_COMMIT` must be bound by Olo after repairs/controls merge; never infer them from mutable main |
Require successful exact DEB/RPM receipt state, cleanup, source equality, embedded
RPM-to-DEB receipt binding, manifest/bundle and helper byte equality. The helper
manifest must contain the same source object as DEB A. Helpers are uninstalled
diagnostics from the frozen package source, not an independent oracle.
`spool_dump` uses shared Rust inspection and protobuf reencoding; its output
cannot replace actual raw frame payloads or independently establish custody.
Source-inspection pins at the frozen checker branch (to be rebound at final
harness admission): `query_oracle.py` SHA256
`fc696ee1173093c258ff4451c4a796dd8e5832761f9e0ff49f89a922b411d23a`;
`delivery_oracle.py` SHA256
`09034939ed48e3c75937adc6087add8964c4971f2c2a3b32b565b0ecefca2fcc`.
Keep both oracles unchanged. Package-source manifest records frame source SHA256
`ddeff38d27982f988e35eb48b26ebaa0e857c82e9d49a68e9d0950bd45071e99`.
## Containment and admission
Serial, quiet host, one disposable guest only. Entire run uses the admitted
outer driver/resource_group, mounted DATA scratch, existing 1,800-second outer
deadline and existing 180-second lifecycle phase bound. Begin one monotonic
upgrade deadline before the first stop; every child uses the smaller of its
named cap and remaining phase time. No deadline resets or extra phase allowance.
Boot/acceptance/reboot phases retain their registered bounds; if the total
remaining outer budget is inadequate, stop and record incomplete rather than
extending it.
Keep 2 vCPUs, 6,144 MiB guest RAM, 8 GiB owned virtual disk, whole host cgroup
7 GiB high/8 GiB max/zero swap. Reserve **9,663,676,416 bytes (9 GiB)** for each
guest, including the full disk plus 1 GiB image/log/helper/evidence overhead.
Actual strict DATA census must be **<=85,336,323,584 bytes** before each cell;
95 GB admission stop and 100 GB ceiling are unchanged. Count snapshot/export
bytes in that overhead. Save one complete pre-upgrade Spool archive; post-install
records rehash the live stopped tree without a duplicate raw archive, and final
stopped evidence requires a second complete Spool archive. Refuse
admission if measured archive/log/helper growth cannot fit the unchanged whole
reservation; an unexpected evidence overrun fails the run and preserves the
stopped fixture rather than omitting its source evidence. Preserve unexpected
failures and stop/reap QEMU before cleanup.
Use the registered exact Debian13 and Fedora44 images and their existing
run-qemu checksum/OS/PID1/cgroup checks. Fedora remains Enforcing with functioning
audit; unavailable prerequisites never support acceptance. The 20-second source
outage is a finite guest lifecycle fixture, not a destructive storage fault.
## Prospective native phase sequence
The outer command shape remains the existing native runner, once the new witness
is merged/admitted. It is not yet an executable witness invocation:
```bash
FABRIC_STORAGE_ROOT=/run/media/kmosoti/data/FabricO11y \
python3 -B "$INSTALL_HARNESS_REPO/tools/resource_group.py" -- \
python3 -B "$INSTALL_HARNESS_REPO/tools/qualification/install/run-qemu.py" \
--family debian --package "$INSTALL_DEB" \
--source-commit 73ff77867d10ef521ca985b094e1d177f7786b77 \
--run-id "$INSTALL_NEW_RUN_ID" --memory-stressor parallel \
--upgrade-from "$INSTALL_PRE_DEB"
```
For Fedora change only family, exact package/predecessor and unique run ID. The
guarded root driver must wrap this command and census/reserve the full 9 GiB;
do not launch it directly based on this draft. Olo binds a new unused run ID for
each family and freezes the exact complete command before publication.
Guest command fragments below describe the actual native operations. New
snapshot/stop/conservation grading is pending implementation and controls; no
invented helper command is represented as available.
1. Install the historical predecessor and complete its A1–A12 baseline with
deferred removal. Candidate-only fresh account/admission checks remain in
separate candidate baseline cells. Save the three existing exact history
rows, account identity and configuration/credential hash manifest while the
predecessor server is still reachable. Neither new line exists yet.
2. Start the upgrade phase deadline. Record the current journal cursor, then:
```bash
systemctl stop fabrico11y-server.service
printf 'accept-upgrade-pending-%s\n' "$INSTALL_NEW_RUN_ID" >> /var/log/fabric-accept/allowed.log
sleep 20
systemctl stop fabrico11y-node.service
```
Verify actual complete stopped states, MainPID0, empty owned unit cgroups and
no managed companion descendants. Record actual stop commands/exits/Result;
a missing/error query is not a stop witness. Require stopped state through
the snapshot/install/post-install comparison boundary.
3. Record source-log device/inode/size, append the second line only after stop:
```bash
printf 'accept-upgrade-unread-%s\n' "$INSTALL_NEW_RUN_ID" >> /var/log/fabric-accept/allowed.log
fabricctl inspect /etc/fabrico11y/node.conf > /root/accept/spool-pre.inspect
/root/accept/spool_dump /etc/fabrico11y/node.conf > /root/accept/spool-pre.jsonl
```
`spool_dump` was transferred and hash-verified from the exact helper manifest;
it is not installed as a product binary. Preserve its exit and bounded
stdout/stderr separately. Copy the complete stopped raw Spool and only the
synthetic allowed-log source bytes/metadata to a fresh evidence directory.
A future adapter must independently require a committed pending sentinel,
an ACK below its sequence, contiguous retained unacknowledged source bytes,
no recovery-required/interrupted-append marker, and an exact unread appended
interval whose second sentinel appears nowhere in committed sources.
Twenty seconds is not an oracle: if the predecessor did not commit the
pending line, preparation fails rather than allowing the candidate to create
a replacement. Record any genuine gaps; the existing denied-log gap is not
silently removed or recast as complete coverage.
4. Native install with both roles stopped:
```bash
dpkg -i /home/accept/fabrico11y.deb
# Fedora alternative, never run both:
rpm -Uvh /home/accept/fabrico11y.deb
```
The Fedora transfer name is the existing harness's guest filename. Reassert
no running service/companion after scriptlets. Before any candidate launch,
obtain candidate `fabricctl inspect`, shared-Rust `spool_dump`, the independent
frame inventory and configuration/credential hashes. Require same Spool
tree bytes, Strand identity/generation, ACK, sequence/source set, source-log
inode/consumed prefix/unread suffix and account/config/credential identities.
Native package binary/UI hashes must match the exact candidate separately.
A hash match alone establishes retention, not successful replay.
5. Keep the existing candidate-only explicit legacy-admin drop-in and run:
```bash
systemctl daemon-reload
systemctl start fabrico11y-server.service fabrico11y-node.service
fabricctl admin /root/accept/admin.conf query '{"kind":"logs","node":"spindle-1","from_ns":0,"to_ns":9223372036854775807,"limit":1000}'
```
Bound readiness/query polling to <=60 seconds and remaining upgrade time.
Require all three old rows unchanged. Pending sentinel must appear exactly
once with the independently captured node_id/sequence/index/observed_ns/body/
attributes; its raw Batch generation remains the captured Strand generation.
Query projection has no separate generation field, so never claim otherwise.
The unread sentinel appears exactly once from the same Strand after collection;
preserve its observed original source-log offset/attributes. Duplicate body,
new identity, missing row or changed projected fields fails conservation.
6. Capture candidate-only journal events from the saved cursor, excluding
predecessor successes. The message omits node_id and generation: bind each
event's real journal `_SYSTEMD_UNIT`, `_PID`, `_BOOT_ID` and monotonic event
time to the independently captured candidate node process, exact package
hash, run and bounded start/stop window. An isolated message string cannot
establish its sender's Strand. For every pre-upgrade pending Batch require actual
`delivery sequence=N sha256=H status=ack committed_through=K` with K>=N and H
equal to SHA256 of the archived raw frame payload. Stop node and server again
before final inspection; verify actual stop/companion absence, durable ACK
at least the pre-upgrade pending maximum, no recovery/append flags, and zero
unread allowed-log suffix. Record unrelated diagnostic sources separately;
do not infer an ACK from row count or from native inspect alone. Retain final
raw evidence, then start the registered services for the actual reboot.
7. Existing L2 must observe a genuinely different boot identity and both enabled
services active, then repeat all five exact row witnesses, account and
configuration/credential hash checks. Existing removal/purge/explicit Fedora
fixture reset follows only after these checks. An upgrade failure blocks all
later acceptance; preserve failed raw logs and stopped evidence.
## Evidence and credential boundary
Export raw source Spool files, synthetic source-log bytes/metadata, independent
frame/source identities, shared-Rust diagnostic stdout/stderr, candidate delivery
events, exact query observations, package/helper/source receipts, resource/stop
receipts and cleanup observations. Never export raw `/etc/fabrico11y` credential
files, private keys or tokens, or command/header logs containing them. For
credentials record only canonical filename, regular-file/type/owner/mode/bytes
and SHA256 before install, after stopped install and after reboot. Explicit
credential list includes server.key, admin-token and node-token; CA/server
certificate and configuration inventories are hash-bound too. No `cat` of a
secret or raw-credential archive is part of this witness.
Raw source inspection, exports and the new adapter must fail closed on symlinks,
nonregular unexpected files, outside-owned evidence paths, mismatched candidates,
extra active project units, incomplete stop or stale receipt. Artifacts remain
private; public evidence is compact hashes/results/wiki pointers. Existing VM
failure preservation stays a failure and grants no acceptance.
## Smallest independent boundary adapter estimate
Olo's scoped budget is approximately **300 net production lines** for the narrow
stopped raw-Spool/full-envelope/Cursor reader and snapshot/conservation join.
Integration scripts are a separate later budget after plan review. Literal
controls and registered protocol changes remain separate trust-boundary work;
no implementation or verification has run for this adapter.
Use only standard-library `struct`, `zlib.crc32`, `hashlib`, JSON and no-follow
regular-file reads; reuse unchanged `query_oracle.decode_batch`,
`decode_logs_request`, `materialize_record` and protobuf `parse_fields`. Their
decoding does **not** validate Batch version, known-field wire types, bounded
u32/u64 values, nonzero generation/sequence or the complete Cursor. The adapter
must check these independently before materialization, plus node_id length16,
nonempty telemetry/gaps and the existing field/count caps. A terminal sequence
or FAK1 ACK (`u64::MAX`) has no representable successor and is refused. Keep
Rust protobuf/wire semantics and all query/delivery/rate oracles unchanged.
Decode all Cursor fields: path/device/inode/offset, skipping_oversize,
prefix_len/prefix_crc and optional full Btrfs UUID/subvolume identity. For each
path select its **last occurrence in committed sequence order**, never the
maximum offset. Validate that latest consumed prefix against the archived
source's exact bytes and recorded identity (full Btrfs identity if present,
legacy device/inode otherwise); require bounded prefix_len<=64 and <=offset,
CRC32 agreement, offset<=size and normal newline boundaries. This fixture's
pending/unread sentinels cannot use skipping_oversize or an incomplete line.
Validate sentinel body and log.file path/device/inode/start/end attributes
against the archived original intervals, with pending captured in predecessor
Batch bytes and unread absent from every pre-upgrade committed Batch.
Actual framing evidence is `crates/fabric-frame/src/frame.rs:165–209,552–585`:
16-byte header = FAB1 + little-endian u32 length + CRC32(first8) + CRC32(payload),
nonzero payload <=1MiB, then exact payload, then 16-byte FAC1 marker with
little-endian u64 data-end offset and CRC32(first12). Marker offset is relative
to its own file. Require the complete boundary and every CRC; preserve actual
payload bytes/hash without protobuf reencoding. Fresh gracefully stopped
fixtures must end exactly on a commit boundary; do not repair/truncate a tail.
Inventory batches.faj and numerically ordered sealed-20digit.faj files, refusing
unexpected `.faj` names. Check sealed first-sequence labels, identity consistency
and contiguous sequences; no payload over1MiB and framed-file aggregate no larger
than the actual journal cap (`spool_bytes - 4096`, runtime.rs:37,169–170; this
fixture configures256MiB total Spool). Read FAI1 identity (28bytes: magic,16byte
ID,u64 nonzero generation)
and FAK1 ACK (20bytes: magic,u64 generation,u64 ACK; absent ACK means0 under the
registered format), generation match, pending cursor bound and absence of
recovery-required/append-in-progress. File inventory/hash custody and supplied
actual stopped-state receipt must precede decoding. Reject changed input
identity during reads. Never derive missing pre-upgrade sources from server
recovery, later candidate output or a query response.
## Literal prospective negative controls
All controls use small owned synthetic fixtures under contained DATA scratch,
default1,800-second outer deadline, no native service/package/VM invocation and
no mutation of actual Spools. Register inputs/outcomes before implementation;
capture actual red, then rerun unchanged controls against the implementation.
| Literal defect | Required refusal / independent discriminator |
| --- | --- |
| Empty pending set (ACK==last sequence) | Preparation fails before native install; preserved history/file count cannot satisfy pending custody |
| Pending sentinel exists only in unread source log | Fails before install; candidate cannot become the source of claimed predecessor custody |
| Reset post-install Spool, retain both log lines | Stream/source identity mismatch fails before start; later successful body reread cannot pass |
| Delete one pending committed frame/sealed file | Exact source set/continuity mismatch, even when remaining files are valid |
| Change one pending payload byte without CRC repair | Independent CRC refusal; shared-Rust output is not the oracle |
| Change pending payload and consistently recalculate both CRCs | Exact raw source bytes/hash mismatch despite valid frame syntax |
| Truncate a FAC1 marker or append an incomplete tail | Boundary refusal, no repair/truncation and no acceptance |
| FAK1 cursor advanced past a still-pending source or generation changed | Before-install ACK/Strand conservation refusal; no fabricated ACK |
| CRC-valid version2/wrong known-field wire type, zero/overflow identity field or terminal sequence/ACK | Full envelope/sidecar refusal before projection; query decoding alone is insufficient |
| Wrong consumed-prefix CRC, malformed full Btrfs identity, skipping cursor or later smaller cursor ignored | Exact latest full Cursor/prefix refusal; maximum-offset selection cannot support admission |
| Rewrite/drop unread suffix or move its committed cursor to EOF | Two-source-line/offset conservation refusal; unread sentinel must survive actual collection |
| Claim ACK with wrong payload SHA, only old predecessor ACKs, or missing candidate event | Delivery witness refusal independently of query body presence |
| ACK message copied from wrong PID/unit/boot or outside candidate run window | Sender-binding refusal even when sequence/hash/message are otherwise correct |
| Duplicate query sentinel or alter node_id/sequence/index/time/attributes | Exact row conservation refusal, not substring matching |
| Reconnect without a changed boot ID | Existing L2 refusal unchanged |
| Empty/error stop query, nonzero MainPID, extra companion/project process | Refuse snapshot/install admission before any mutation |
| Symlink/outside path, helper/candidate/receipt substitution | Refuse read/transfer/start admission; synthetic snapshots remain unchanged |
| Credential replaced while configs stay identical | Hash/type/owner/mode mismatch fails; no raw secret material exported |
Clean paired positives retain two-source-line conservation with real pending
identity, complete frames, unchanged stopped install snapshot, matching actual
candidate ACK, single exact rows and distinct reboot. Synthetic positives prove
only checker behavior; actual Debian/Fedora cells remain required.
## Prospective narrow API and literal selectors
`tools/qualification/install/custody_witness.py` is not implemented. Its two
planned entry points are:
```python
stopped_snapshot(root, stopped_receipt, archived_allowed_logs)
conserve(before, after_install, after_drain, queries, candidate_ack_events)
```
The snapshot root is an existing owned stopped raw Spool under contained DATA
scratch. `stopped_receipt` binds canonical spool_root, actual configured
spool_bytes268435456 (framed cap minus4096), actual boolean
stop_confirmed, boot_id, no extra project units, and both service observations
(inactive/failed, query_exit3, MainPID0, empty cgroups and no companions). Its
`binding` contains run_id, candidate_frozen, candidate_package_sha256,
package_source_commit and explicit harness_commit. The after-drain receipt also
contains independently observed `candidate_process`: exact unit/PID/boot,
monotonic start_ns/end_ns and the same binding. No candidate sender is inferred
from an ACK message. Native producer/transfer/validation for this receipt is a
later integration prerequisite; synthetic objects prove only checker logic.
`archived_allowed_logs` maps each guest source path to a no-follow archive_path
and original guest device/inode/full optional Btrfs identity. Snapshot output
retains node_id/generation/ack, complete file identities/hashes, exact raw
source payload/hash per sequence, full latest cursors, source archive metadata
and bytes, and unchanged-oracle materialized rows. `queries` supplies the
independently captured old rows, exact after_drain/after_reboot rows, old/new
boot IDs and unique pending_body/unread_body. ACK events retain journal
unit/PID/boot/time/message. `conserve` returns normally only on all required
agreements; every unavailable/invalid prerequisite raises ValueError. Exact
credential/account/config hash joins remain native integration obligations,
without raw credential export. This API performs no repair or cleanup.
The independent fixtures in `test_custody_witness.py` use hand-encoded tiny
protobuf/FAB1/FAC1 bytes, not Rust or oracle encoders. Existing oracle decoders
may be reused by the implementation only. Olo runs the initial named red:
```bash
python3 -B tools/qualification/install/test_custody_witness.py \
MechanismAvailability.test_stopped_snapshot_and_conservation_are_available
```
Absence is an explicit mechanism-availability failure, **not** a corrupt-frame
bug reproduction or product failure. Specific controls are NOT RUN while the
adapter is absent; later actual named controls are:
```bash
python3 -B tools/qualification/install/test_custody_witness.py \
CustodyWitnessControls
```
Selectors cover clean two-line custody, full framing/envelope refusals, full
Cursor/latest-prefix identity, stopped/no-follow admission, and exact install/
ACK/query/reboot conservation. These commands require Olo's serial admitted
resource driver, default1800 outer and1GiB containment reservation; this draft
does not authorize direct execution. No tests/imports/builds/native commands or
VM workloads ran during preparation.
## Remaining decisions and dependencies
Olo/Pip must register the additional witness within #41's prospective protocol,
review the independent adapter/control ownership and integrate a clean final
harness before any workload. Current run-qemu does not transfer the helpers or
archive stopped source Spools, so the plan is blocked on that implementation,
its independent red/green controls and exact-head validation. The frozen package
source remains unchanged; no rebuild/refreeze is inferred from a harness repair.
The historical predecessor's recorded A11 outage/restart supports the native
pending-source fixture mechanism, but the exact new two-line fixture has not
run. Its actual payload/source provenance and hashes remain mandatory. The
same-code alpha0 fixtures being prepared for #46 are native-metadata current-code
fixtures only; they cannot establish historical migration into this candidate.
The existing legacy-admin lifecycle route also does not establish production
passkey/browser owner acceptance. Unavailable predecessor collection, helper
compatibility, budget, audit or phase time is a real prerequisite failure, not
a reason to change limits or silently substitute current-code custody.
Source /run/media/kmosoti/data/FabricO11y/scratch/installation-custody-witness/tools/qualification/install/test_custody_witness.py; SHA256 944a3fb9f1da6a6efa3d0ef57fc0258d324e193d60efcce7039229afda9270b7.
"""Prospective stopped-custody controls; fixture bytes never come from Rust."""
import copy
import hashlib
import importlib.util
import os
from pathlib import Path
import stat
import struct
import tempfile
import unittest
import zlib
ADAPTER = Path(__file__).with_name('custody_witness.py')
SOURCE = '/var/log/fabric-accept/allowed.log'
NODE = bytes(range(1, 17))
PENDING = b'accept-upgrade-pending-literal\n'
UNREAD = b'accept-upgrade-unread-literal\n'
BINDING = dict(run_id='custody-literal',
candidate_frozen='8fb6f413179b094288316e25845c137440ba23fa',
candidate_package_sha256='8edfc7709f4b5dc3da01c00a1d8c442470298e96f8c8e502cfb14e8c9e713e92',
package_source_commit='73ff77867d10ef521ca985b094e1d177f7786b77',
harness_commit='fcd57f62581a243fad34c61a258acb760cbdfce8')
PROCESS = dict(unit='fabrico11y-node.service', pid=55, boot_id='literal-boot-a',
start_ns=100, end_ns=200, **BINDING)
def varint(value):
out = bytearray()
while value >= 128:
out.append((value & 127) | 128); value >>= 7
return bytes(out + bytes([value]))
def vi(tag, value):
return varint(tag << 3) + varint(value)
def ld(tag, value):
return varint(tag << 3 | 2) + varint(len(value)) + value
def cursor(offset, *, prefix_crc=None, skipping=0, identity=b''):
prefix = (PENDING + UNREAD)[:min(offset, 64)]
return (ld(1, SOURCE.encode()) + vi(2, 7) + vi(3, 9) + vi(4, offset)
+ vi(5, skipping) + vi(6, len(prefix))
+ vi(7, zlib.crc32(prefix) if prefix_crc is None else prefix_crc)
+ (ld(8, identity) if identity else b''))
def batch(sequence=4, *, version=1, node=NODE, generation=1, body=PENDING,
offset=None, cursor_bytes=None):
offset = len(PENDING) if offset is None else offset
attrs = {'log.file.path': SOURCE, 'log.file.device': '7', 'log.file.inode': '9',
'log.file.offset.start': str(offset - len(body)),
'log.file.offset.end': str(offset)}
record = b'\x59' + struct.pack('<Q', 123) + ld(5, ld(1, body[:-1]))
for key, value in attrs.items():
record += ld(6, ld(1, key.encode()) + ld(2, ld(1, value.encode())))
logs = ld(1, ld(2, ld(2, record)))
return (vi(1, version) + ld(2, node) + vi(3, generation) + vi(4, sequence)
+ ld(6, logs) + ld(7, cursor(offset) if cursor_bytes is None else cursor_bytes))
def frame(payload, at=0):
first = b'FAB1' + struct.pack('<I', len(payload))
marker = b'FAC1' + struct.pack('<Q', at + 16 + len(payload))
return (first + struct.pack('<II', zlib.crc32(first), zlib.crc32(payload))
+ payload + marker + struct.pack('<I', zlib.crc32(marker)))
def adapter():
spec = importlib.util.spec_from_file_location('custody_witness', ADAPTER)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
class MechanismAvailability(unittest.TestCase):
def test_stopped_snapshot_and_conservation_are_available(self):
self.assertTrue(ADAPTER.is_file(), 'NOT RUN: independent custody adapter is absent')
module = adapter()
self.assertTrue(callable(getattr(module, 'stopped_snapshot', None)))
self.assertTrue(callable(getattr(module, 'conserve', None)))
class CustodyWitnessControls(unittest.TestCase):
@classmethod
def setUpClass(cls):
if not ADAPTER.is_file():
raise unittest.SkipTest('NOT RUN: independent custody adapter is absent')
cls.module = adapter()
def setUp(self):
self.temp = tempfile.TemporaryDirectory(dir=os.environ['FABRIC_SCRATCH_ROOT'])
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
archive = self.root / 'allowed.log'; archive.write_bytes(PENDING + UNREAD)
self.logs = {SOURCE: dict(archive_path=str(archive), device=7, inode=9,
btrfs_identity=None)}
def snapshot(self, name, payloads=None, *, ack=3, change=None, receipt_change=None):
root = self.root / name; root.mkdir()
payloads = [batch()] if payloads is None else payloads
raw = b''
for payload in payloads: raw += frame(payload, len(raw))
(root / 'identity').write_bytes(b'FAI1' + NODE + struct.pack('<Q', 1))
(root / 'acked').write_bytes(b'FAK1' + struct.pack('<QQ', 1, ack))
(root / 'writer.lock').touch()
(root / 'batches.faj').write_bytes(raw)
receipt = dict(binding=dict(BINDING), spool_root=str(root), spool_bytes=268435456,
stop_confirmed=True,
boot_id='literal-boot-a', extra_project_units=[],
units={unit: dict(active_state='inactive', query_exit=3, main_pid=0,
cgroup_empty=True, companions=[])
for unit in ('fabrico11y-server.service', 'fabrico11y-node.service')},
candidate_process=copy.deepcopy(PROCESS) if name == 'drain' else None)
if change: change(root)
if receipt_change: receipt_change(receipt)
def inventory():
return {path.name: (path.lstat().st_mode,
str(path.readlink()) if path.is_symlink() else
path.read_bytes() if stat.S_ISREG(path.lstat().st_mode) else None)
for path in root.iterdir()}
original = inventory()
try:
return self.module.stopped_snapshot(root, receipt, self.logs)
finally:
self.assertEqual(inventory(), original, 'snapshot reader mutated raw source')
def chain(self):
before = self.snapshot('before')
installed = self.snapshot('installed')
drained = self.snapshot('drain', [batch(), batch(5, body=UNREAD,
offset=len(PENDING + UNREAD))], ack=5)
old = [dict(node='spindle-1', node_id=NODE.hex(), sequence=seq, index=0,
observed_ns=120, body=body, attributes={}) for seq, body in enumerate(
('accept-line-before-start', 'accept-line-during-outage',
'accept-line-after-pressure'), 1)]
rows = copy.deepcopy(old)
for sequence, body, start in ((4, PENDING, 0), (5, UNREAD, len(PENDING))):
rows.append(dict(node='spindle-1', node_id=NODE.hex(), sequence=sequence,
index=0, observed_ns=123, body=body[:-1].decode(), attributes={
'log.file.path': SOURCE, 'log.file.device': '7', 'log.file.inode': '9',
'log.file.offset.start': str(start), 'log.file.offset.end': str(start + len(body))}))
queries = dict(before=old, after_drain=copy.deepcopy(rows),
after_reboot=copy.deepcopy(rows), boot_before='literal-boot-a',
boot_after='literal-boot-b', pending_body=PENDING[:-1].decode(),
unread_body=UNREAD[:-1].decode())
events = [dict(unit=PROCESS['unit'], pid=55, boot_id=PROCESS['boot_id'], at_ns=150,
message=f'delivery sequence={seq} sha256={hashlib.sha256(payload).hexdigest()} '
f'status=ack committed_through={seq} elapsed_us=1')
for seq, payload in ((4, batch()), (5, batch(5, body=UNREAD,
offset=len(PENDING + UNREAD))))]
return before, installed, drained, queries, events
def test_clean_raw_snapshot_and_two_line_conservation(self):
chain = self.chain()
self.assertEqual(chain[0]['node_id'], NODE.hex())
self.assertEqual(chain[0]['ack'], 3)
self.assertEqual(chain[0]['sources'][0]['sha256'], hashlib.sha256(batch()).hexdigest())
self.assertEqual(chain[0]['cursors'][SOURCE]['offset'], len(PENDING))
self.module.conserve(*chain)
def test_complete_frame_and_envelope_refusals(self):
wrong_wire = ld(1, b'\x01') + batch()[2:]
variants = [batch(version=2), wrong_wire, batch(node=NODE[:-1]),
batch(generation=0), batch(sequence=0), batch(sequence=2**64 - 1),
batch(generation=2**64), batch(cursor_bytes=cursor(len(PENDING), skipping=2))]
for index, payload in enumerate(variants):
with self.subTest(payload=index), self.assertRaises(ValueError):
self.snapshot(f'envelope-{index}', [payload])
changes = [lambda root: (root / 'batches.faj').write_bytes(frame(batch())[:-1]),
lambda root: (root / 'batches.faj').write_bytes(frame(batch()) + b'F'),
lambda root: (root / 'batches.faj').write_bytes(frame(batch())[:16] + b'X' + frame(batch())[17:]),
lambda root: (root / 'recovery-required').write_bytes(b'failure'),
lambda root: (root / 'append-in-progress').write_bytes(b'interrupted'),
lambda root: (root / 'acked').write_bytes(b'FAK1' + struct.pack('<QQ', 1, 2**64 - 1)),
lambda root: (root / 'acked').write_bytes(b'FAK1' + struct.pack('<QQ', 2, 0))]
for index, change in enumerate(changes):
with self.subTest(frame=index), self.assertRaises(ValueError):
self.snapshot(f'frame-{index}', change=change)
def test_full_cursor_prefix_identity_and_sequence_order(self):
older = (vi(1, 1) + ld(2, NODE) + vi(3, 1) + vi(4, 4)
+ ld(7, cursor(len(PENDING + UNREAD))) + ld(8, b'older cursor fixture'))
snapshot = self.snapshot('latest', [older, batch(5)])
self.assertEqual(snapshot['cursors'][SOURCE]['offset'], len(PENDING))
self.assertEqual(snapshot['cursors'][SOURCE]['prefix_crc'], zlib.crc32(PENDING))
for index, raw in enumerate((cursor(len(PENDING), prefix_crc=1),
cursor(len(PENDING), skipping=1),
cursor(len(PENDING), identity=ld(1, b'x') + vi(2, 1)))):
with self.subTest(cursor=index), self.assertRaises(ValueError):
self.snapshot(f'cursor-{index}', [batch(cursor_bytes=raw)])
self.logs[SOURCE]['btrfs_identity'] = dict(uuid=NODE.hex(), subvolume_id=7)
full = ld(1, NODE) + vi(2, 7)
valid = self.snapshot('btrfs', [batch(cursor_bytes=cursor(len(PENDING), identity=full))])
self.assertEqual(valid['cursors'][SOURCE]['btrfs_identity'], self.logs[SOURCE]['btrfs_identity'])
def test_stop_and_path_admission_refusals(self):
mutations = [lambda receipt: receipt.update(stop_confirmed=False),
lambda receipt: receipt.update(extra_project_units=['unexpected.service']),
lambda receipt: receipt['units'].pop('fabrico11y-node.service'),
lambda receipt: receipt['units']['fabrico11y-node.service'].update(main_pid=55),
lambda receipt: receipt['units']['fabrico11y-node.service'].update(query_exit=1)]
for index, mutation in enumerate(mutations):
with self.subTest(stop=index), self.assertRaises(ValueError):
self.snapshot(f'stop-{index}', receipt_change=mutation)
target = self.root / 'outside-identity'; target.write_bytes(b'FAI1' + NODE + struct.pack('<Q', 1))
def symlink(root):
(root / 'identity').unlink(); (root / 'identity').symlink_to(target)
with self.assertRaises(ValueError): self.snapshot('symlink', change=symlink)
self.assertEqual(target.read_bytes(), b'FAI1' + NODE + struct.pack('<Q', 1))
def test_install_ack_query_and_reboot_conservation_refusals(self):
chain = self.chain()
mutations = [lambda args: args[0].update(ack=4),
lambda args: args[1]['binding'].update(run_id='other-run'),
lambda args: args[1]['binding'].update(candidate_package_sha256='00' * 32),
lambda args: args[1].update(node_id='00' * 16),
lambda args: args[1]['sources'].clear(),
lambda args: args[1]['sources'][0].update(sha256='00' * 32),
lambda args: args[1]['cursors'][SOURCE].update(offset=len(PENDING + UNREAD)),
lambda args: args[2].update(ack=0),
lambda args: args[4].clear(),
lambda args: args[4][0].update(pid=44),
lambda args: args[4][0].update(unit='other.service'),
lambda args: args[4][0].update(boot_id='predecessor-boot'),
lambda args: args[4][0].update(at_ns=99),
lambda args: args[4][0].update(message=args[4][0]['message'].replace(hashlib.sha256(batch()).hexdigest(), '00' * 32)),
lambda args: args[3]['after_drain'].append(copy.deepcopy(args[3]['after_drain'][0])),
lambda args: args[3]['after_reboot'][0].update(observed_ns=124),
lambda args: args[3].update(boot_after='literal-boot-a')]
for index, mutation in enumerate(mutations):
args = copy.deepcopy(chain); mutation(args)
with self.subTest(conservation=index), self.assertRaises(ValueError):
self.module.conserve(*args)
if __name__ == '__main__':
unittest.main()