Experiment formal source prerequisite green plan 01 - kmosoti/FabricO11y GitHub Wiki

Source prerequisites: final finite green pre-flight

Prospective only. No new green or native acceptance inferred. Existing red logs and receipts remain retained.

source-prerequisite-green-plan-01.md

SHA256 d98621757427284d23170676af8496372b7d3dede043ac4efa8eb48203d58d85.

# Finite source prerequisites, prospective green verification

No candidate VM, browser, native clock binding, owner constructor or acceptance
workload runs under this plan. One serial Olo executor, fresh strict census and
full 1 GiB reservation per command, default16/20GiB/noSwap/1800s,95GB unchanged.
Read #59 before and after each command; only advance on an actual stopped
receipt. Neither source correction nor a finite passing test is qualification.

META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01

Clock clean head a74ab66bae81197d8f7a22f61ccba1a686b7a79e:

```sh
python3 -B META/run-step-console-clock-green.py console-clock-controls-green-01 1073741824 env PYTHONPATH=tools python3 -B -m unittest discover -s tools/ui -p test_session_clock.py -v

The four unchanged lifecycle methods previously produced eight real assertion failures at d33fb33, exit1/stopped. Two earlier import failures are separate setup failures. Frozen image hash/dev/inode/map witnesses and private process group shutdown/diagnostic retention were corrected; native cleanup failures now retain staging and capture existing ledger/frames regardless close error. Tests76f937331e2ebe6e2da9ee121a39a0411c823f38bd788c3c4a36107cd3dcec49, oracle4817a0d09c1c920a67242637cbbb3a6473d6bea42ce716d07a73e8d7122d1728, protocoldea94fa0109a4e757ac2c022b2829e2ec4073678feb077b1472b32a9ed6cdab3 are unchanged. Productionf183ce8caf2f73fe16e11f9de3342b4368a5990a17964e75861cecb4c4e68a34, net+67 within the extra85-line scope. Independent Fenn static review followed by its small archive correction; actual full controls still not run.

Installation checker clean head db5b9d86ff70795a4b7ecec046b60d07d8e4bb1a:

python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-controls-02 1073741824 python3 -B tools/qualification/install/test_run_qemu.py
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-shell-syntax-02 1073741824 bash -n tools/qualification/install/acceptance.sh tools/qualification/install/run.sh tools/qualification/install/lifecycle.sh
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-docs-check-02 1073741824 env PATH=/run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0:/usr/local/bin:/usr/bin:/bin BUN_INSTALL_CACHE_DIR=/run/media/kmosoti/data/FabricO11y/tools/docs-bun-cache /run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0/bun tools/docs/check.mjs
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-docs-tests-02 1073741824 env PATH=/run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0:/usr/local/bin:/usr/bin:/bin BUN_INSTALL_CACHE_DIR=/run/media/kmosoti/data/FabricO11y/tools/docs-bun-cache /run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0/bun tools/docs/check.test.mjs
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-docs-hooks-02 1073741824 python3 -B tools/docs/test_hooks.py

The final F3 control is literal readable positive versus grep-error2 on the real acceptance block, without errexit masking. At42e45f5 one intended failing subcase accepted grep2 as PASS (exit1/stopped); f437f0e now distinguishes rc0/1/ other. Unchanged test35bbbc144065a22bd37390aed7325ded8f56b3bdfbee672944fc19292cfba744; acceptancec287ad707ea97ea5e25033a6d78d425b4a324016b69144458255b07d95b30b73. Rev5 is prospective, result prose migrated to wiki and CURRENT describes behavior only. CX-INSTALLATION-BASELINE-AUDIT-READ registered separately. Full suite has ten methods. These actual greens precede ONE final PR66 push; Pip delta review and exact final-head hosted CI precede owner merge. Historical remote fcd57 hosted fast38333544753/extended38333544797 both succeeded, and belong only to that revision. No local cold Cargo checks in this plan.

Docs use the already locked DATA dependency directory through a real ignored node_modules directory with entry symlinks. No install/cache write is requested; post-move dependency view has not yet been checked. Existing lock/package hashes remain exact. All source-result data is wiki-only. Every command label is unique.


## run-step-installation-checker-final-02.py

SHA256 `84114964a3b9c595aa04ea4afb68798c29f740830973b33bbb197089e85761a1`.

```python
import datetime, hashlib, json, os, re, signal, subprocess, sys, time
from pathlib import Path
ROOT = Path('/run/media/kmosoti/data/FabricO11y')
REPO = ROOT / 'scratch/installation-checker-repair'
OUT = ROOT / 'results/service-opening-candidate'
STORE = ROOT / 'scratch/service-opening-census'
FROZEN = 'db5b9d86ff70795a4b7ecec046b60d07d8e4bb1a'
def require(value, reason):
    if not value:
        raise ValueError(reason)
label, reserve_text, *command = sys.argv[1:]
reserve = int(reserve_text)
delegated = command[:1] == ['--delegate']
if delegated: command = command[1:]
require(re.fullmatch(r'[a-z0-9-]{1,80}', label) and reserve > 0 and command, 'ordinary workload admission guard')
record_path = OUT / (label + '.json')
require(not record_path.exists(), 'unique run label required')
def census():
    # Rootless container files can belong to subordinate UIDs. Inspect all of
    # them in the same user namespace; never skip unreadable directories.
    require(not STORE.is_symlink(), 'ordinary workload admission guard')
    require((STORE / '.owned-census-store').read_text() == 'service-opening-census-v1\n', 'ordinary workload admission guard')
    args = ['podman', '--root', str(STORE/'graph'), '--runroot', str(STORE/'run'),
            '--tmpdir', str(STORE/'tmp'), '--storage-driver', 'vfs',
            'unshare', 'python3', '-B', str(OUT/'census.py')]
    env = dict(os.environ, TMPDIR=str(STORE/'tmp'), TMP=str(STORE/'tmp'), TEMP=str(STORE/'tmp'))
    result = subprocess.run(args, env=env, text=True, capture_output=True, timeout=25)
    if result.returncode:
        raise RuntimeError(f'strict namespace census failed ({result.returncode}): {result.stderr}')
    observed = json.loads(result.stdout)
    require(observed['uid'] == 0 and observed['gid'] == 0, 'ordinary workload admission guard')
    require(type(observed['data_bytes']) is int and observed['data_bytes'] >= 0, 'ordinary workload admission guard')
    return observed
def git(*args):
    return subprocess.check_output(['git', *args], cwd=REPO, text=True).strip()
require(git('rev-parse', 'HEAD') == FROZEN and not git('status', '--porcelain'), 'clean frozen source required')
require(not subprocess.check_output(['systemctl', '--user', 'list-units', 'fabric-work-*', '--state=running', '--plain', '--no-legend'], text=True).strip(), 'another project workload is active')
initial = census()
used = initial['data_bytes']
record = {'label': label, 'command': command, 'delegated': delegated, 'frozen': None, 'scope': 'harness controls only; no candidate acceptance evidence', 'harness_revision': FROZEN, 'source_status_before': git('status', '--porcelain'), 'storage_root': str(ROOT), 'admission_bytes': used, 'admission_observation': initial, 'reserved_bytes': reserve, 'observations': [], 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat()}
if used + reserve > 95_000_000_000:
    record.update(state='not_run', reason='95 GB storage admission threshold')
    record_path.write_text(json.dumps(record, indent=2)+'\n')
    raise SystemExit('storage admission denied; project command not launched')
receipts = REPO / 'target/resource-containment/runs'
before = set(receipts.glob('*.json'))
env = dict(os.environ, FABRIC_STORAGE_ROOT=str(ROOT), CARGO_HOME=str(ROOT/'toolchain-cache/cargo'), RUSTUP_HOME=str(ROOT/'toolchain-cache/rustup'), CARGO_BUILD_JOBS='2')
env['PATH'] = str(ROOT/'tools/candidate-dpkg/bin') + ':' + env['PATH']
log_path = OUT / (label + '.log')
record.update(state='running', log=str(log_path))
record_path.write_text(json.dumps(record, indent=2)+'\n')
def find_unit():
    with log_path.open() as stream:
        prefix = stream.read(65536)
    match = re.search(r'Running as unit: (fabric-work-[0-9a-f]+)\.service', prefix)
    return match.group(1) if match else None
def stop_owned(process):
    unit = find_unit()
    if unit:
        stopped = subprocess.run(['systemctl', '--user', 'stop', unit],
                                 text=True, capture_output=True, timeout=25)
        record['stop'] = {'unit': unit, 'exit': stopped.returncode, 'stderr': stopped.stderr}
        if stopped.returncode:
            # Interrupt the launcher too; its finally block independently
            # confirms that this exact service has no surviving descendants.
            process.send_signal(signal.SIGINT)
    else:
        process.send_signal(signal.SIGINT)
        record['stop'] = {'unit': None, 'launcher_interrupted': True}
with log_path.open('wb') as log:
    launcher = ['python3', '-B', 'tools/resource_group.py', *(['--delegate'] if delegated else []), '--', *command]
    process = subprocess.Popen(launcher, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT)
    next_census = time.monotonic() + 30
    try:
        while process.poll() is None:
            time.sleep(1)
            if time.monotonic() >= next_census:
                observed = census()
                record['observations'].append({'unix':time.time(), **observed})
                record_path.write_text(json.dumps(record, indent=2)+'\n')
                if observed['data_bytes'] >= 95_000_000_000:
                    record['interruption'] = '95 GB threshold reached'
                    break
                next_census = time.monotonic() + 30
    except BaseException as error:
        record['interruption'] = f'{type(error).__name__}: {error}'
    finally:
        if record.get('interruption') and process.poll() is None:
            try:
                stop_owned(process)
            except BaseException as error:
                record['stop_error'] = f'{type(error).__name__}: {error}'
                process.send_signal(signal.SIGINT)
        status = process.wait()
record['launcher_exit'] = status
if record.get('interruption'): status = 2
with log_path.open('rb') as stream:
    log_digest = hashlib.file_digest(stream, 'sha256').hexdigest()
record.update(state='interrupted' if record.get('interruption') else 'completed', exit=status, source_status_after=git('status', '--porcelain'), finished_utc=datetime.datetime.now(datetime.timezone.utc).isoformat(), log_sha256=log_digest, resource_receipts=[])
archive = OUT/'resource-containment'; archive.mkdir(exist_ok=True)
for p in sorted(set(receipts.glob('*.json')) - before):
    value=json.loads(p.read_text())
    if value['command'] != command: continue
    q=archive/p.name; q.write_bytes(p.read_bytes())
    record['resource_receipts'].append({'path':str(q),'sha256':hashlib.sha256(q.read_bytes()).hexdigest(),'exit':value['exit'],'stop_confirmed':value['stop_confirmed'],'retained_failure_evidence':value['retained_failure_evidence']})
if not record['resource_receipts'] or any(not r['stop_confirmed'] or r['exit'] != status for r in record['resource_receipts']):
    record['receipt_error'] = 'missing or inconsistent outer resource receipt'
    status = 2
    record['exit'] = status
record_path.write_text(json.dumps(record, indent=2)+'\n')
print(json.dumps(record, indent=2))
if status: print(log_path.read_text()[-8000:])
raise SystemExit(status)

run-step-console-clock-green.py

SHA256 7e2e1038a4262a3bcdbfac9c6f19c5ab658666757ed1b1cce9a3292ac4a58dcf.

import datetime, hashlib, json, os, re, signal, subprocess, sys, time
from pathlib import Path
ROOT = Path('/run/media/kmosoti/data/FabricO11y')
REPO = ROOT / 'scratch/console-access-acceptance'
OUT = ROOT / 'results/service-opening-candidate'
STORE = ROOT / 'scratch/service-opening-census'
FROZEN = 'a74ab66bae81197d8f7a22f61ccba1a686b7a79e'
def require(value, reason):
    if not value:
        raise ValueError(reason)
label, reserve_text, *command = sys.argv[1:]
reserve = int(reserve_text)
delegated = command[:1] == ['--delegate']
if delegated: command = command[1:]
require(re.fullmatch(r'[a-z0-9-]{1,80}', label) and reserve > 0 and command, 'ordinary workload admission guard')
record_path = OUT / (label + '.json')
require(not record_path.exists(), 'unique run label required')
def census():
    # Rootless container files can belong to subordinate UIDs. Inspect all of
    # them in the same user namespace; never skip unreadable directories.
    require(not STORE.is_symlink(), 'ordinary workload admission guard')
    require((STORE / '.owned-census-store').read_text() == 'service-opening-census-v1\n', 'ordinary workload admission guard')
    args = ['podman', '--root', str(STORE/'graph'), '--runroot', str(STORE/'run'),
            '--tmpdir', str(STORE/'tmp'), '--storage-driver', 'vfs',
            'unshare', 'python3', '-B', str(OUT/'census.py')]
    env = dict(os.environ, TMPDIR=str(STORE/'tmp'), TMP=str(STORE/'tmp'), TEMP=str(STORE/'tmp'))
    result = subprocess.run(args, env=env, text=True, capture_output=True, timeout=25)
    if result.returncode:
        raise RuntimeError(f'strict namespace census failed ({result.returncode}): {result.stderr}')
    observed = json.loads(result.stdout)
    require(observed['uid'] == 0 and observed['gid'] == 0, 'ordinary workload admission guard')
    require(type(observed['data_bytes']) is int and observed['data_bytes'] >= 0, 'ordinary workload admission guard')
    return observed
def git(*args):
    return subprocess.check_output(['git', *args], cwd=REPO, text=True).strip()
require(git('rev-parse', 'HEAD') == FROZEN and not git('status', '--porcelain'), 'clean frozen source required')
require(not subprocess.check_output(['systemctl', '--user', 'list-units', 'fabric-work-*', '--state=running', '--plain', '--no-legend'], text=True).strip(), 'another project workload is active')
initial = census()
used = initial['data_bytes']
record = {'label': label, 'command': command, 'delegated': delegated, 'frozen': None, 'scope': 'harness controls only; no candidate acceptance evidence', 'harness_revision': FROZEN, 'source_status_before': git('status', '--porcelain'), 'storage_root': str(ROOT), 'admission_bytes': used, 'admission_observation': initial, 'reserved_bytes': reserve, 'observations': [], 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat()}
if used + reserve > 95_000_000_000:
    record.update(state='not_run', reason='95 GB storage admission threshold')
    record_path.write_text(json.dumps(record, indent=2)+'\n')
    raise SystemExit('storage admission denied; project command not launched')
receipts = REPO / 'target/resource-containment/runs'
before = set(receipts.glob('*.json'))
env = dict(os.environ, FABRIC_STORAGE_ROOT=str(ROOT), CARGO_HOME=str(ROOT/'toolchain-cache/cargo'), RUSTUP_HOME=str(ROOT/'toolchain-cache/rustup'), CARGO_BUILD_JOBS='2')
env['PATH'] = str(ROOT/'tools/candidate-dpkg/bin') + ':' + env['PATH']
log_path = OUT / (label + '.log')
record.update(state='running', log=str(log_path))
record_path.write_text(json.dumps(record, indent=2)+'\n')
def find_unit():
    with log_path.open() as stream:
        prefix = stream.read(65536)
    match = re.search(r'Running as unit: (fabric-work-[0-9a-f]+)\.service', prefix)
    return match.group(1) if match else None
def stop_owned(process):
    unit = find_unit()
    if unit:
        stopped = subprocess.run(['systemctl', '--user', 'stop', unit],
                                 text=True, capture_output=True, timeout=25)
        record['stop'] = {'unit': unit, 'exit': stopped.returncode, 'stderr': stopped.stderr}
        if stopped.returncode:
            # Interrupt the launcher too; its finally block independently
            # confirms that this exact service has no surviving descendants.
            process.send_signal(signal.SIGINT)
    else:
        process.send_signal(signal.SIGINT)
        record['stop'] = {'unit': None, 'launcher_interrupted': True}
with log_path.open('wb') as log:
    launcher = ['python3', '-B', 'tools/resource_group.py', *(['--delegate'] if delegated else []), '--', *command]
    process = subprocess.Popen(launcher, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT)
    next_census = time.monotonic() + 30
    try:
        while process.poll() is None:
            time.sleep(1)
            if time.monotonic() >= next_census:
                observed = census()
                record['observations'].append({'unix':time.time(), **observed})
                record_path.write_text(json.dumps(record, indent=2)+'\n')
                if observed['data_bytes'] >= 95_000_000_000:
                    record['interruption'] = '95 GB threshold reached'
                    break
                next_census = time.monotonic() + 30
    except BaseException as error:
        record['interruption'] = f'{type(error).__name__}: {error}'
    finally:
        if record.get('interruption') and process.poll() is None:
            try:
                stop_owned(process)
            except BaseException as error:
                record['stop_error'] = f'{type(error).__name__}: {error}'
                process.send_signal(signal.SIGINT)
        status = process.wait()
record['launcher_exit'] = status
if record.get('interruption'): status = 2
with log_path.open('rb') as stream:
    log_digest = hashlib.file_digest(stream, 'sha256').hexdigest()
record.update(state='interrupted' if record.get('interruption') else 'completed', exit=status, source_status_after=git('status', '--porcelain'), finished_utc=datetime.datetime.now(datetime.timezone.utc).isoformat(), log_sha256=log_digest, resource_receipts=[])
archive = OUT/'resource-containment'; archive.mkdir(exist_ok=True)
for p in sorted(set(receipts.glob('*.json')) - before):
    value=json.loads(p.read_text())
    if value['command'] != command: continue
    q=archive/p.name; q.write_bytes(p.read_bytes())
    record['resource_receipts'].append({'path':str(q),'sha256':hashlib.sha256(q.read_bytes()).hexdigest(),'exit':value['exit'],'stop_confirmed':value['stop_confirmed'],'retained_failure_evidence':value['retained_failure_evidence']})
if not record['resource_receipts'] or any(not r['stop_confirmed'] or r['exit'] != status for r in record['resource_receipts']):
    record['receipt_error'] = 'missing or inconsistent outer resource receipt'
    status = 2
    record['exit'] = status
record_path.write_text(json.dumps(record, indent=2)+'\n')
print(json.dumps(record, indent=2))
if status: print(log_path.read_text()[-8000:])
raise SystemExit(status)

session_clock.py

SHA256 f183ce8caf2f73fe16e11f9de3342b4368a5990a17964e75861cecb4c4e68a34.

"""Disposable target-only clock fixture and contained native binding admission."""
import argparse
import ctypes
import hashlib
import json
import os
from pathlib import Path
import platform
import select
import shutil
import signal
import socket
import subprocess
import sys
import tempfile
import threading
import time
import session_clock_oracle as oracle

HERE=Path(__file__).resolve().parent
sys.path.insert(0,str(HERE.parent))
from resource_group import STORAGE, require_limits


def digest(path):
    with Path(path).open('rb') as stream:return hashlib.file_digest(stream,'sha256').hexdigest()


class KernelTime(ctypes.Structure):_fields_=[('sec',ctypes.c_long),('nsec',ctypes.c_long)]
def kernel_time():
    value=KernelTime();oracle.require(ctypes.CDLL(None,use_errno=True).syscall(228,0,ctypes.byref(value))==0,'fixture-raw-clock');return value.sec*1000000000+value.nsec


def freeze_images(executable,helper):
    frozen={}
    for name,path in (('executable',executable),('helper',helper)):
        before=path.stat();frozen[name+'_sha256']=digest(path);after=path.stat()
        oracle.require((before.st_dev,before.st_ino,before.st_size,before.st_mtime_ns,before.st_ctime_ns)==(after.st_dev,after.st_ino,after.st_size,after.st_mtime_ns,after.st_ctime_ns),'isolation-'+('target' if name=='executable' else 'helper')+'-frozen')
        frozen[name+'_identity']={'device':before.st_dev,'inode':before.st_ino}
    return frozen


def process_binding(pid, executable, helper, intercepted=True, *, expected=None):
    proc=Path(f'/proc/{pid}');actual=proc/'exe'
    if expected is not None:
        for name,paths,message in (('executable',(actual,executable),'isolation-target-frozen'),('helper',(helper,),'isolation-helper-frozen')):
            for path in paths:
                stat=path.stat()
                oracle.require({'device':stat.st_dev,'inode':stat.st_ino}==expected[name+'_identity'] and digest(path)==expected[name+'_sha256'],message)
    oracle.require(actual.resolve()==executable.resolve() and digest(actual)==digest(executable),'isolation-target')
    rows=[line.split(maxsplit=5) for line in (proc/'maps').read_text().splitlines()]
    mappings={(row[3],row[4],row[5]) for row in rows if len(row)==6 and row[5]==str(helper)};stat=helper.stat()
    if expected is not None and mappings:
        frozen=expected['helper_identity']
        oracle.require(mappings=={(f'{os.major(frozen["device"]):02x}:{os.minor(frozen["device"]):02x}',str(frozen['inode']),str(helper))},'isolation-helper-frozen')
    oracle.require(mappings==({(f'{os.major(stat.st_dev):02x}:{os.minor(stat.st_dev):02x}',str(stat.st_ino),str(helper))} if intercepted else set()),'isolation-interception')
    libc=next(Path(row[5]) for row in rows if len(row)==6 and row[5].endswith('/libc.so.6'))
    return {'pid':pid,'executable_sha256':digest(actual),'helper_sha256':digest(helper) if mappings else None,'libc':str(libc),'libc_sha256':digest(libc),
            'start_ticks':int((proc/'stat').read_text().rsplit(')',1)[1].split()[19]),'cgroup':(proc/'cgroup').read_text(),'helper_maps':sorted(mappings)}


def stop_group(process):
    try:os.killpg(process.pid,signal.SIGTERM)
    except ProcessLookupError:process.wait(timeout=3);return
    try:process.wait(timeout=3)
    except subprocess.TimeoutExpired:pass
    try:os.killpg(process.pid,signal.SIGKILL)
    except ProcessLookupError:pass
    process.wait(timeout=3);deadline=time.monotonic()+3
    while True:
        try:
            while os.waitpid(-process.pid,os.WNOHANG)[0]:pass
        except ChildProcessError:pass
        try:os.killpg(process.pid,0)
        except ProcessLookupError:return
        oracle.require(time.monotonic()<deadline,'fixture-process-group-stop');time.sleep(.01)


class SessionClock:
    def __init__(self, executable, helper, directory, deadline, initial=None, socket_type=socket.SOCK_SEQPACKET, preload=True, invalid_fd=False):
        self.deadline=deadline;self.frames=[];self.rows=[];self.bytes=0;self.error=None;self.buffer=b''
        env=dict(os.environ);oracle.require(not any(name in env for name in ('LD_PRELOAD','LD_AUDIT','FABRIC_CLOCK_FD')),'fixture-environment')
        self.parent,self.child=socket.socketpair(socket.AF_UNIX,socket_type|socket.SOCK_NONBLOCK|socket.SOCK_CLOEXEC)
        self.fd=self.child.fileno();self.socket_identity=os.readlink(f'/proc/self/fd/{self.fd}')
        self.parent.send(initial if initial is not None else self.frame(0));self.stderr=directory.open('xb')
        if preload:env.update(LD_PRELOAD=str(helper),FABRIC_CLOCK_FD=str(1048575 if invalid_fd else self.fd))
        self.argv=[str(executable),'interactive',str(self.fd),str(helper),self.socket_identity]
        try:self.process=subprocess.Popen(self.argv,env=env,pass_fds=(self.fd,),stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=self.stderr,start_new_session=True)
        except BaseException:self.parent.close();self.child.close();self.stderr.close();raise
        self.child.close();self.receiver=threading.Thread(target=self.receive,daemon=True);self.receiver.start()
    def frame(self,offset):
        frame=b'FCLKv1\0\0'+len(self.frames).to_bytes(8,'little')+offset.to_bytes(8,'little')
        oracle.check_frames(self.frames+[frame]);self.frames.append(frame);return frame
    def advance(self,offset):oracle.require(self.parent.send(self.frame(offset))==24,'fixture-control-send')
    def receive(self):
        try:
            while True:
                if not select.select([self.parent],[],[],.1)[0]:continue
                raw=self.parent.recv(257)
                if not raw:return
                self.bytes+=len(raw);oracle.require(len(raw)<=256 and self.bytes<=1048576,'fixture-ledger-bound')
                self.rows.append(json.loads(raw))
        except BaseException as error:
            self.error=type(error).__name__+': '+str(error)
            try:os.killpg(self.process.pid,signal.SIGTERM)
            except ProcessLookupError:pass
    def line(self):
        while b'\n' not in self.buffer:
            oracle.require(self.error is None,'fixture-ledger-error')
            remaining=self.deadline-time.monotonic();oracle.require(remaining>0,'fixture-deadline')
            oracle.require(select.select([self.process.stdout],[],[],remaining)[0],'fixture-deadline')
            raw=os.read(self.process.stdout.fileno(),4096);oracle.require(bool(raw),'fixture-probe-eof')
            self.buffer+=raw;oracle.require(len(self.buffer)<=65536,'fixture-probe-bound')
        raw,self.buffer=self.buffer.split(b'\n',1);return json.loads(raw)
    def command(self,name):self.process.stdin.write((name+'\n').encode());self.process.stdin.flush()
    def close(self):
        try:stop_group(self.process)
        finally:
            self.receiver.join(timeout=1);self.parent.close();self.process.stdin.close();self.process.stdout.close();self.stderr.close()
        oracle.require(not self.receiver.is_alive(),'fixture-receiver-stop')
        try:os.killpg(self.process.pid,0)
        except ProcessLookupError:return
        raise ValueError('fixture-process-group-stop')


def native_admission(executable,helper,output,deadline,report):
    def close(current,evidence):
        try:current.close()
        except BaseException as error:
            report['native_cleanup_error']=type(error).__name__+': '+str(error);raise
        finally:
            evidence['ledger']=list(current.rows);evidence['frames']=[frame.hex() for frame in current.frames]
    target=SessionClock(executable,helper,output/'positive.stderr',deadline);report['clocks']=[];report['system_time']=[]
    try:
        ready=target.line();report['ready']=ready
        oracle.require(ready=={'kind':'ready','pid':target.process.pid,'env_clear':True,'cloexec':True,'socket_type':5} and os.readlink(f'/proc/{target.process.pid}/fd/{target.fd}')==target.socket_identity and int(Path(f'/proc/{target.process.pid}/fdinfo/{target.fd}').read_text().split('flags:',1)[1].split()[0],8)&os.O_CLOEXEC,'isolation-control')
        report['target']=process_binding(target.process.pid,executable,helper,expected=report['frozen_images']);report['controller']=process_binding(os.getpid(),Path(sys.executable),helper,False);report['host_clocks']=[]
        report['control_fd']={'identity':target.socket_identity,'fdinfo':Path(f'/proc/{target.process.pid}/fdinfo/{target.fd}').read_text()};report['probe_argv']=target.argv
        for sequence,offset in enumerate((0,1,1799,1800,28799,28800)):
            if sequence:target.advance(offset)
            if sequence==2:
                target.command('fork');fork,child,done=(target.line() for _ in range(3));report['fork']=[fork,child,done]
                row=fork['row'];oracle.require(row['pid']==child['pid']==done['fork_pid']!=target.process.pid and done['status']==0,'isolation-fork-pid')
                oracle.require(row['offset_s']==1 and row['sequence']==done['sequence_before']==1 and row['returned_ns']==row['raw_ns'],'isolation-fork-consumption')
                oracle.require(row['syscall_before_ns']<=row['raw_ns']<=row['syscall_after_ns'] and row['syscall_after_ns']-row['syscall_before_ns']<=1000000000,'isolation-host-clock')
                oracle.require(child['kind']=='exec' and child['env_clear'] and not child['helper_mapped'] and not child['control_fd_present'],'isolation-exec-escape')
                oracle.require(child['syscall_before_ns']<=child['clock_ns']<=child['syscall_after_ns'] and child['syscall_after_ns']-child['syscall_before_ns']<=1000000000,'isolation-host-clock')
            target.command('clock')
            for _ in range(3):
                value=target.line();oracle.require(value['kind']=='clock','clock-binding');report['clocks'].append(value['row'])
            system=target.line();oracle.require(system['kind']=='system','clock-binding');report['system_time'].append(system['row'])
            before=kernel_time();clock=time.time_ns();after=kernel_time();oracle.require(before<=clock<=after and after-before<=1000000000 and (not report['host_clocks'] or before>=report['host_clocks'][-1]['syscall_after_ns']),'isolation-host-clock');report['host_clocks'].append({'sequence':sequence,'syscall_before_ns':before,'clock_ns':clock,'syscall_after_ns':after})
        target.command('sleep');sleep=target.line();oracle.require(sleep['kind']=='sleep','sleep-fields');report['sleep']=sleep['row']
        oracle.check_frames(target.frames);oracle.check_probe_clocks(report['clocks'],target.process.pid);oracle.check_sleep(report['sleep'])
        for row,clock in zip(report['system_time'],report['clocks'][::3]):
            oracle.require((row['pid'],row['sequence'],row['offset_s'],row['clock_id'])==(clock['pid'],clock['sequence'],clock['offset_s'],0),'clock-binding')
            oracle.require(row['syscall_before_ns']<=row['raw_ns']<=row['syscall_after_ns'] and row['syscall_after_ns']-row['syscall_before_ns']<=1000000000,'clock-raw-bracket')
            oracle.require(row['returned_ns']==row['raw_ns']+row['offset_s']*1000000000,'clock-realtime-only')
        target.command('quit');oracle.require(target.process.wait(timeout=max(.001,deadline-time.monotonic()))==0,'fixture-probe-exit')
    finally:
        close(target,report)
    oracle.require(target.error is None,'fixture-ledger-error')
    for row in report['clocks']+report['system_time']:
        oracle.require({key:value for key,value in row.items() if not key.startswith('syscall_')} in report['ledger'],'clock-ledger-binding')
    report['controls']=[]
    initial=b'FCLKv1\0\0'+bytes(16)
    controls=[('short',initial[:-1],'FCLK-FRAME'),('oversize',initial+b'x','FCLK-FRAME'),('version',b'bad-v001'+bytes(16),'FCLK-FRAME'),
              ('initial-sequence',initial[:8]+(1).to_bytes(8,'little')+bytes(8),'FCLK-SEQUENCE'),('initial-offset',initial[:16]+(1).to_bytes(8,'little'),'FCLK-OFFSET'),
              ('skip',None,'FCLK-SEQUENCE'),('duplicate',None,'FCLK-SEQUENCE'),('decrease',None,'FCLK-OFFSET'),('cap',None,'FCLK-OFFSET'),('overflow',None,'FCLK-OFFSET'),
              ('count',None,'FCLK-COUNT'),('death',None,'FCLK-EOF'),('closed',None,'FCLK-RECV'),('descriptor',None,'FCLK-FD'),('type',None,'FCLK-TYPE'),('unpreloaded',None,'isolation-interception')]
    for name,packet,expected in controls:
        error_path=output/(name+'.stderr');current=SessionClock(executable,helper,error_path,deadline,initial=packet,socket_type=socket.SOCK_STREAM if name=='type' else socket.SOCK_SEQPACKET,preload=name!='unpreloaded',invalid_fd=name=='descriptor')
        evidence={'case':name,'refusal':expected,'exit':None,'stderr':None};report['controls'].append(evidence)
        try:
            if name=='unpreloaded':
                oracle.require(current.line()=={'kind':'absent','pid':current.process.pid},'fixture-unpreloaded')
                try:process_binding(current.process.pid,executable,helper,expected=report['frozen_images'])
                except ValueError as error:oracle.require(str(error)==expected,'fixture-wrong-refusal')
                else:raise ValueError('fixture-unpreloaded-admitted')
            else:
                if packet is None and name not in ('type','descriptor'):
                    current.line()
                    if name=='death':current.parent.shutdown(socket.SHUT_WR)
                    elif name=='decrease':current.parent.send(initial[:8]+(1).to_bytes(8,'little')+(1).to_bytes(8,'little'));current.parent.send(initial[:8]+(2).to_bytes(8,'little')+bytes(8))
                    else:
                        values=[(i,0) for i in range(1,65)] if name=='count' else [(2 if name=='skip' else 0 if name=='duplicate' else 1,28861 if name=='cap' else 2**64-1 if name=='overflow' else 0)]
                        if name!='closed':
                            for seq,value in values:current.parent.send(initial[:8]+seq.to_bytes(8,'little')+value.to_bytes(8,'little'))
                    current.command('close' if name=='closed' else 'clock')
                oracle.require(current.process.wait(timeout=max(.001,deadline-time.monotonic()))==78,'fixture-control-exit')
        finally:close(current,evidence)
        stderr=error_path.read_text();oracle.require(name=='unpreloaded' or stderr.strip()==expected,'fixture-wrong-refusal')
        evidence.update(exit=current.process.returncode,stderr=stderr)


def main():
    parser=argparse.ArgumentParser(description=__doc__);parser.add_argument('--output',type=Path,required=True);parser.add_argument('--rustc',required=True);parser.add_argument('--cc',default='gcc');args=parser.parse_args()
    require_limits();oracle.require(platform.system()=='Linux' and platform.machine()=='x86_64','fixture-platform')
    output=args.output;oracle.require(output.is_absolute() and output.resolve()==output and output.is_relative_to(STORAGE),'fixture-output');output.mkdir(mode=0o700)
    scratch=Path(os.environ['FABRIC_SCRATCH_ROOT']);oracle.require(scratch.is_dir() and scratch.is_relative_to(STORAGE),'fixture-scratch')
    directory=Path(tempfile.mkdtemp(prefix='session-clock-',dir=scratch));deadline=time.monotonic()+240;report={'scope':'native binding only; actual packaged browser session checks pending','state':'failed','exit':1,'commands':[]}
    def run(command):
        report['commands'].append(command);path=output/('tool-'+str(len(report['commands']))+'.log');timeout=max(.001,deadline-time.monotonic());failure=None
        with path.open('xb') as stream:
            process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT,start_new_session=True)
            try:
                while process.poll() is None:
                    if time.monotonic()>=deadline:raise subprocess.TimeoutExpired(command,timeout)
                    oracle.require(path.stat().st_size<=1048576,'fixture-tool-output');time.sleep(.01)
            except BaseException as error:failure=error;raise
            finally:
                try:stop_group(process)
                except BaseException as error:
                    report['tool_cleanup_error']=type(error).__name__+': '+str(error)
                    if failure is None:raise
                finally:
                    with path.open('rb') as captured:raw=captured.read(1048576)
                    report.setdefault('tool_outputs',[]).append({'command':command,'exit':process.returncode,'raw_log':str(path),'output':raw.decode(errors='replace')})
                    if isinstance(failure,subprocess.TimeoutExpired):failure.output=raw
        oracle.require(path.stat().st_size<=1048576,'fixture-tool-output');oracle.require(process.returncode==0,'fixture-tool-exit');return raw.decode()
    try:
        compiler=Path(shutil.which(args.cc)).resolve();rustc=Path(shutil.which(args.rustc)).resolve();version=run([str(rustc),'-vV'])
        oracle.require('release: 1.99.0' in version and 'commit-hash: b940084d7' in version,'fixture-toolchain')
        run([str(compiler),'--version']);helper=directory/'session-clock.so';executable=directory/'session-clock-probe'
        run([str(compiler),'-shared','-fPIC','-pthread','-O2','-Wall','-Wextra','-Werror','-o',str(helper),str(HERE/'session_clock.c'),'-ldl'])
        run([str(rustc),'--edition=2021','-C','opt-level=1','-o',str(executable),str(HERE/'session_clock_probe.rs')])
        dynamic=run(['readelf','--dyn-syms','--wide',str(executable)]);oracle.require('clock_gettime@' in dynamic,'fixture-dynamic-clock-import')
        interpreter=run(['readelf','-lW',str(executable)]).split('Requesting program interpreter: ',1)[1].split(']',1)[0]
        report['frozen_images']=freeze_images(executable,helper)
        report['hashes']={str(path):digest(path) for path in (compiler,rustc,helper,executable,Path(interpreter),HERE/'session_clock.py',HERE/'session_clock.c',HERE/'session_clock_probe.rs')}
        oracle.require(all(report['hashes'][str(path)]==report['frozen_images'][name+'_sha256'] for name,path in (('executable',executable),('helper',helper))),'fixture-image-hash')
        oracle.require(sum(path.stat().st_size for path in directory.rglob('*') if path.is_file())<=512*1024**2,'fixture-storage-bound')
        native_admission(executable,helper,output,deadline,report)
        report['build_id_notes']={str(path):run(['readelf','-nW',str(path)]) for path in (helper,executable,Path(interpreter),Path(report['target']['libc']))};report.update(state='completed',exit=0)
    except BaseException as error:report['error']=type(error).__name__+': '+str(error);raise
    finally:
        try:
            if report['state']=='completed':shutil.rmtree(directory)
            elif any(name in report for name in ('tool_cleanup_error','native_cleanup_error')):report['failure_evidence']=str(directory)
            else:
                retained=output/'failed-fixture';shutil.move(str(directory),str(retained));report['failure_evidence']=str(retained)
        except BaseException as error:
            report['cleanup_error']=type(error).__name__+': '+str(error);report['failure_evidence']=str(directory)
            if 'error' not in report:report.update(state='failed',exit=1);raise
        finally:
            report['cleanup']={'owned_fixture':str(directory),'absent':not directory.exists(),'failure_evidence':report.get('failure_evidence')}
            (output/'receipt.json').write_text(json.dumps(report,indent=2)+'\n');print(str(output/'receipt.json'))
    return report['exit']


if __name__=='__main__':raise SystemExit(main())

test_run_qemu.py

SHA256 35bbbc144065a22bd37390aed7325ded8f56b3bdfbee672944fc19292cfba744.

"""Named rejection must survive provenance and environment checks."""
import ast
import importlib.util
import os
from pathlib import Path
import unittest
import subprocess
import tempfile
from types import SimpleNamespace
from unittest.mock import call, patch

SPEC = importlib.util.spec_from_file_location('install_qemu', Path(__file__).with_name('run-qemu.py'))
MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(MODULE)


class MutationGradingTests(unittest.TestCase):
    def test_expected_rpm_mutation_cleans_only_confirmed_stopped_scratch(self):
        main = next(node for node in ast.parse(Path(MODULE.__file__).read_text()).body
                    if isinstance(node, ast.FunctionDef) and node.name == 'main')
        final = next(node.finalbody for node in main.body if isinstance(node, ast.Try))
        index = next(i for i, node in enumerate(final) if isinstance(node, ast.Assign)
                     and any(isinstance(t, ast.Name) and t.id == 'completed_mutation' for t in node.targets))
        code = compile(ast.Module(body=final[index:index + 2], type_ignores=[]), '<actual-cleanup>', 'exec')
        for output, stopped, error, expected in (
                ('ACCEPT A6 FAIL injected', True, None, True),
                ('ACCEPT A12 FAIL unrelated', True, None, False),
                ('ACCEPT A6 FAIL injected', False, None, False),
                ('ACCEPT A6 FAIL injected', True, 'helper hash mismatch', False)):
            with self.subTest(output=output, stopped=stopped, error=error), \
                    tempfile.TemporaryDirectory(dir=os.environ['FABRIC_SCRATCH_ROOT']) as root:
                namespace = dict(args=SimpleNamespace(mutate=None, expected_mutation='root-user'),
                    grading_mutation='root-user', mutation_package_archive=None,
                    mutation_rejected=MODULE.mutation_rejected, rc=1, acceptance_log=output,
                    error=error, cleanup_ok=stopped, process_cleanup_confirmed=stopped,
                    work_dir=Path(root), result_dir=Path(root), shutil=MODULE.shutil, json=MODULE.json,
                    guest_acceptance_exit=1, mutation_package_sha256=None)
                with patch.object(MODULE.shutil, 'rmtree') as remove:
                    exec(code, namespace)
                self.assertEqual(remove.call_args_list, [call(Path(root))] if expected else [])

    def test_reboot_witness_runs_after_privilege_transition(self):
        # Origin: Fedora upgrade03 really rebooted, but SSH flattened bash -c
        # argv, expanding protected paths too early and leaving a bare test.
        def fake_ssh(args, *, input_text=None, **kwargs):
            shell = '''sudo() { privileged=yes; "$@"; }
cat() {
  [ "$privileged" = yes ] || return 1
  case "$1" in
    /proc/sys/kernel/random/boot_id) echo current ;;
    /root/accept/lifecycle-boot-id) echo "$saved" ;;
    *) return 1 ;;
  esac
}
export -f cat
export privileged saved
'''
            return subprocess.run(['bash', '-c', shell + ' '.join(args[1:])],
                                  input=input_text, text=True, capture_output=True,
                                  env={'PATH': '/usr/bin:/bin', 'saved': saved})
        with patch.object(MODULE, 'run', fake_ssh):
            saved = 'before'
            self.assertEqual(MODULE.changed_boot(['fake-ssh']).returncode, 0)
            saved = 'current'
            self.assertNotEqual(MODULE.changed_boot(['fake-ssh']).returncode, 0)
            saved = 'before'
            old = fake_ssh(['fake-ssh', 'sudo', 'bash', '-c',
                           'test "$(cat /proc/sys/kernel/random/boot_id)" != "$(cat /root/accept/lifecycle-boot-id)"'])
            self.assertNotEqual(old.returncode, 0)

    def test_all_registered_named_defects_are_required(self):
        for mutation, checks in MODULE.MUTATION_EXPECTED_FAILURES.items():
            output = '\n'.join(f'ACCEPT {check} FAIL injected' for check in sorted(checks))
            self.assertTrue(MODULE.mutation_rejected(mutation, 1, output))
            for check in checks:
                with self.subTest(mutation=mutation, missing=check):
                    self.assertFalse(MODULE.mutation_rejected(
                        mutation, 1, output.replace(f'ACCEPT {check} FAIL', f'ACCEPT {check} PASS')))

    def test_unrelated_failure_is_not_detection(self):
        self.assertFalse(MODULE.mutation_rejected('root-user', 1, 'ACCEPT A12 FAIL timeout'))

    def test_bad_provenance_or_environment_cannot_pass(self):
        output = 'ACCEPT A6 FAIL injected'
        self.assertFalse(MODULE.mutation_rejected('root-user', 1, output, 'helper hash mismatch'))
        self.assertFalse(MODULE.mutation_rejected('root-user', 1, output + '\nACCEPT A8h NOT-RUN unavailable'))
        for code in (None, 0, 2, 3, -9):
            self.assertFalse(MODULE.mutation_rejected('root-user', code, output))


class InstallationShellControls(unittest.TestCase):
    """Origin: C-035 / C-aud-7; real shell decisions, isolated paths and command stubs."""
    def shell(self, program, family='fedora', *, unit='', state='inactive', avc='',
              retained=True, purge_keeps_state=False, systemctl_mode='normal', grep_error=False):
        with tempfile.TemporaryDirectory(dir=os.environ['FABRIC_SCRATCH_ROOT']) as directory:
            root = Path(directory)
            paths = {old: root / name for old, name in (
                ('/root/accept', 'accept'), ('/var/lib/fabrico11y', 'state'),
                ('/etc/fabrico11y', 'config'), ('/usr/bin/fabric-server', 'absent-binary'))}
            work, data, config = (paths[p] for p in ('/root/accept', '/var/lib/fabrico11y', '/etc/fabrico11y'))
            work.mkdir(); config.mkdir()
            if retained: (data / 'server').mkdir(parents=True)
            account = 'fabricolly:x:995:995:Fabric O11y service::/usr/sbin/nologin\n'
            (work / 'lifecycle-account').write_text(account)
            (config / 'server.conf').write_text('synthetic configuration\n')
            for old, new in paths.items(): program = program.replace(old, str(new))
            stubs = '''
rpm() { [ "$*" = '-e fabrico11y' ]; }
dpkg() {
  if [ "$1" = -P ]; then
    if [ "$PURGE_KEEP_STATE" != yes ]; then rm -rf "$DATA"; fi
    rm -rf "$CONFIG"
  fi
}
rm() {
  [ "$1" = -rf ] || return 97; shift
  for path; do
    case "$path" in "$DATA"|"$CONFIG") command rm -rf -- "$path" ;; *) return 97 ;; esac
  done
}
package_remove() { return 0; }
getent() { printf '%s' "$ACCOUNT"; }
getenforce() { echo Enforcing; }
auditctl() { echo 'enabled 1'; }
ausearch() { printf '%s\\n' "$AVC"; }
grep() {
  if [ "$GREP_ERROR" = yes ] && [ "$1" = -E ]; then
    echo 'synthetic audit read error' >&2; return 2
  fi
  command grep "$@"
}
systemctl() {
  case "$SYSTEMCTL_MODE" in
    empty) return 3 ;;
    incomplete) echo inactive; return 3 ;;
    error) printf 'inactive\\ninactive\\n'; echo 'synthetic systemctl query error' >&2; return 1 ;;
  esac
  local rc=3 value unit
  for unit; do
    case "$unit" in fabrico11y-*.service)
      value=inactive; if [ "$unit" = "$BAD_UNIT" ]; then value=$BAD_STATE; fi
      case " $* " in *' --quiet '*) ;; *) printf '%s\\n' "$value" ;; esac
      if [ "$value" = active ]; then rc=0; fi
    esac
  done
  return "$rc"
}
'''
            env = dict(os.environ, ACCOUNT=account, DATA=str(data), CONFIG=str(config),
                       BAD_UNIT=unit, BAD_STATE=state, AVC=avc,
                       PURGE_KEEP_STATE='yes' if purge_keeps_state else 'no',
                       SYSTEMCTL_MODE=systemctl_mode, GREP_ERROR='yes' if grep_error else 'no')
            return subprocess.run(['bash', '-s', '--', 'remove', family, 'synthetic-package'],
                input=stubs + program, env=env, text=True, capture_output=True, timeout=10)

    def test_lifecycle_remove_rejects_active_services_and_fabric_avc(self):
        program = Path(__file__).with_name('lifecycle.sh').read_text()
        for unit in ('fabrico11y-server.service', 'fabrico11y-node.service'):
            for state in ('active', 'activating', 'deactivating', 'reloading'):
                with self.subTest(unit=unit, state=state):
                    result = self.shell(program, unit=unit, state=state)
                    self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
                    self.assertNotIn('ACCEPT A13a PASS', result.stdout)
        for comm in ('fabric-server', 'fabric-node', 'fabricctl'):
            with self.subTest(comm=comm):
                result = self.shell(program, avc=f'type=AVC msg=audit(1.0:1): avc: denied {{ read }} comm="{comm}"')
                self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
                self.assertNotIn('ACCEPT F4 PASS', result.stdout)
        for family in ('debian', 'fedora'):
            with self.subTest(clean=family):
                result = self.shell(program, family)
                self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
                self.assertIn('ACCEPT A13a PASS', result.stdout)
                self.assertIn('ACCEPT A13b PASS', result.stdout)

    def test_lifecycle_remove_rejects_missing_state_and_incomplete_purge(self):
        program = Path(__file__).with_name('lifecycle.sh').read_text()
        for kwargs, forbidden in (({'retained': False}, 'ACCEPT A13a PASS'),
                                  ({'purge_keeps_state': True}, 'ACCEPT A13b PASS')):
            with self.subTest(kwargs=kwargs):
                result = self.shell(program, 'debian', **kwargs)
                self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
                self.assertNotIn(forbidden, result.stdout)

    def test_lifecycle_remove_rejects_unobserved_services_and_unreadable_audit(self):
        program = Path(__file__).with_name('lifecycle.sh').read_text()
        for mode in ('empty', 'incomplete', 'error'):
            with self.subTest(systemctl_mode=mode):
                result = self.shell(program, systemctl_mode=mode)
                self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
                self.assertNotIn('ACCEPT A13a PASS', result.stdout)
        result = self.shell(program, grep_error=True)
        self.assertIn('synthetic audit read error', result.stderr)
        self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
        self.assertNotIn('ACCEPT F4 PASS', result.stdout)

    def test_acceptance_remove_rejects_either_live_unit(self):
        source = Path(__file__).with_name('acceptance.sh').read_text()
        functions = '\n'.join(line for line in source.splitlines() if line.startswith(('pass()', 'fail()')))
        start = source.index('package_remove >')
        block = source[start:source.index('package_purge >', start)]
        for decision in ('stopped=$(systemctl is-active ', 'pass A13a ', 'fail A13a '):
            self.assertIn(decision, block, 'actual nonempty A13a decision required')
        program = 'set -u\nFAILS=0\nW=/root/accept\n' + functions + '\n' + block + '\nexit "$FAILS"\n'
        for unit in ('fabrico11y-server.service', 'fabrico11y-node.service'):
            for state in ('active', 'activating', 'deactivating', 'reloading', 'inactive'):
                with self.subTest(unit=unit, state=state):
                    result = self.shell(program, unit=unit, state=state)
                    self.assertEqual(result.returncode, 0 if state == 'inactive' else 1,
                                     result.stdout + result.stderr)
                    self.assertIn('ACCEPT A13a ' + ('PASS' if state == 'inactive' else 'FAIL'), result.stdout)
        for mode in ('empty', 'incomplete', 'error'):
            with self.subTest(systemctl_mode=mode):
                result = self.shell(program, systemctl_mode=mode)
                self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
                self.assertNotIn('ACCEPT A13a PASS', result.stdout)

    def test_acceptance_fedora_rejects_unreadable_audit(self):
        source = Path(__file__).with_name('acceptance.sh').read_text()
        functions = '\n'.join(line for line in source.splitlines() if line.startswith(('pass()', 'fail()')))
        start = source.index('  audit_status=$(auditctl -s 2>&1)')
        block = source[start:source.index('\nfi\necho "RESULT', start)]
        for decision in ('ausearch -m AVC,USER_AVC ', 'selinux-avc.log', 'pass F3 ', 'fail F3 '):
            self.assertIn(decision, block, 'actual nonempty F3 audit decision required')
        program = 'set -u\nFAILS=0\nW=/root/accept\n' + functions + '\n' + block + '\nexit "$FAILS"\n'
        for grep_error in (False, True):
            with self.subTest(grep_error=grep_error):
                result = self.shell(program, grep_error=grep_error)
                if grep_error:
                    self.assertIn('synthetic audit read error', result.stderr)
                    self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
                    self.assertNotIn('ACCEPT F3 PASS', result.stdout)
                else:
                    self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
                    self.assertIn('ACCEPT F3 PASS', result.stdout)


if __name__ == '__main__':
    unittest.main()

acceptance.sh

SHA256 c287ad707ea97ea5e25033a6d78d425b4a324016b69144458255b07d95b30b73.

#!/bin/bash
# Running-installation acceptance, executed as root INSIDE a disposable
# Debian 13 systemd container or VM (see run.sh and run-qemu.py). Never run it
# on a real host: it
# creates and deletes the fabricolly account, installs and purges the package
# and starts services. Protocol:
# docs/experiments/formal/installation-acceptance-protocol.md.
#
# Usage: acceptance.sh <DEB> [--self-spindle-ca <CA-PATH>]
#        [--memory-stressor tail|parallel]
# Prints one line per check, "ACCEPT <ID> PASS|FAIL|NOT-RUN <detail>". Exits 1
# if any check failed, 3 if none failed but one could not run in this
# environment, and 0 only when every check ran and passed.
set -u
DEB=${1:?usage: acceptance.sh <DEB> [--self-spindle-ca <CA-PATH>] [--memory-stressor tail|parallel]}
shift
SELF_SPINDLE_CA=
A12_MEMORY_STRESSOR=tail
PACKAGE_FAMILY=debian
DEFER_REMOVAL=0
LEGACY_SERVER=0
while [ "$#" -gt 0 ]; do
  case "$1" in
    --legacy-server) LEGACY_SERVER=1; shift ;;
    --defer-removal) DEFER_REMOVAL=1; shift ;;
    --package-family)
      [ "$#" -ge 2 ] || exit 2
      PACKAGE_FAMILY=$2; shift 2
      case "$PACKAGE_FAMILY" in debian|fedora) ;; *) exit 2 ;; esac
      ;;
    --self-spindle-ca)
      [ "$#" -ge 2 ] && [ -n "$2" ] || { echo "missing self-spindle CA path" >&2; exit 2; }
      SELF_SPINDLE_CA=$2; shift 2
      case "$SELF_SPINDLE_CA" in
        /*) ;;
        *) echo "self-spindle CA path must be absolute" >&2; exit 2 ;;
      esac
      case "$SELF_SPINDLE_CA" in
        *[!A-Za-z0-9_./-]*) echo "self-spindle CA path contains unsupported characters" >&2; exit 2 ;;
      esac
      ;;
    --memory-stressor)
      [ "$#" -ge 2 ] || { echo "missing A12 memory stressor" >&2; exit 2; }
      A12_MEMORY_STRESSOR=$2; shift 2
      case "$A12_MEMORY_STRESSOR" in
        tail|parallel) ;;
        *) echo "unsupported A12 memory stressor: $A12_MEMORY_STRESSOR" >&2; exit 2 ;;
      esac
      ;;
    *) echo "usage: acceptance.sh <DEB> [--self-spindle-ca <CA-PATH>] [--memory-stressor tail|parallel]" >&2; exit 2 ;;
  esac
done
FAILS=0
NOTRUN=0
UNIFIED=0; [ "$(stat -fc %T /sys/fs/cgroup)" = cgroup2fs ] && UNIFIED=1
W=/root/accept
mkdir -p "$W"
LOGDIR=/var/log/fabric-accept
SCRIPT_DIR=$(cd -- "$(dirname -- "$0")" && pwd)

pass() { echo "ACCEPT $1 PASS ${*:2}"; }
fail() { echo "ACCEPT $1 FAIL ${*:2}"; FAILS=$((FAILS + 1)); }
notrun() { echo "ACCEPT $1 NOT-RUN ${*:2}"; NOTRUN=$((NOTRUN + 1)); }
wait_for() { # wait_for <seconds> <command...>
  local limit=$1; shift
  local end=$((SECONDS + limit))
  until "$@" >/dev/null 2>&1; do
    [ $SECONDS -ge $end ] && return 1
    sleep 1
  done
}
admin() { fabricctl admin "$W/admin.conf" "$@"; }
query() { admin query "$1"; }
show() { systemctl show -P "$2" "$1"; }

package_install() {
  if [ "$PACKAGE_FAMILY" = fedora ]; then rpm -Uvh --replacepkgs "$DEB"; else dpkg -i "$DEB"; fi
}
package_remove() {
  if [ "$PACKAGE_FAMILY" = fedora ]; then rpm -e fabrico11y; else dpkg -r fabrico11y; fi
}
package_purge() {
  if [ "$PACKAGE_FAMILY" = fedora ]; then
    rpm -e fabrico11y >/dev/null 2>&1 || :
    rm -rf /var/lib/fabrico11y /etc/fabrico11y
  else dpkg -P fabrico11y; fi
}
package_extract() {
  if [ "$PACKAGE_FAMILY" = fedora ]; then
    mkdir -p "$x"
    (cd "$x" && rpm2cpio "$DEB" | cpio -idm --quiet)
  else dpkg-deb -x "$DEB" "$x"; fi
}
echo "ENV family=$PACKAGE_FAMILY systemd $(systemctl --version | head -1 | cut -d' ' -f2) kernel $(uname -r)"
cat /etc/os-release
if [ "$PACKAGE_FAMILY" = fedora ]; then
  [ "$(getenforce)" = Enforcing ] && pass F1 "SELinux enforcing" || fail F1 "SELinux must remain enforcing"
fi
echo "ENV cgroup $(stat -fc %T /sys/fs/cgroup) controllers: $(cat /sys/fs/cgroup/cgroup.controllers 2>/dev/null)"

# A1 sysusers: dry run and a temporary root, applied twice.
x=$W/pkg; rm -rf "$x"; package_extract
conf=$x/usr/lib/sysusers.d/fabrico11y.conf
r=$W/sysroot; rm -rf "$r"; mkdir -p "$r/etc"
dry=$(systemd-sysusers --dry-run --root="$r" "$conf" 2>&1); dry_rc=$?
dry_clean=0; [ -e "$r/etc/passwd" ] && dry_clean=1
systemd-sysusers --root="$r" "$conf" >/dev/null 2>&1 && first=$(grep '^fabricolly:' "$r/etc/passwd")
systemd-sysusers --root="$r" "$conf" >/dev/null 2>&1 && second=$(grep '^fabricolly:' "$r/etc/passwd")
if [ $dry_rc -eq 0 ] && echo "$dry" | grep -q fabricolly && [ $dry_clean -eq 0 ] \
   && [ -n "${first:-}" ] && [ "$first" = "${second:-}" ] \
   && echo "$first" | grep -Eq '^fabricolly:x:[0-9]+:[0-9]+:Fabric O11y service:/:/usr/sbin/nologin$'; then
  pass A1 "temporary root: $first"
else
  fail A1 "dry_rc=$dry_rc first='${first:-}' second='${second:-}'"
fi

# A2 collision refusal: a human account, then a non-system group, named fabricolly.
useradd -m -u 1500 -s /bin/bash fabricolly
out=$(package_install 2>&1); rc=$?
if [ $rc -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
  pass A2a "human account uid 1500 refused, nothing unpacked"
else
  fail A2a "rc=$rc"
fi
package_purge >/dev/null 2>&1; userdel -r fabricolly 2>/dev/null
groupadd -g 1600 fabricolly
out=$(package_install 2>&1); rc=$?
if [ $rc -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
  pass A2b "non-system group gid 1600 refused, nothing unpacked"
else
  fail A2b "rc=$rc"
fi
package_purge >/dev/null 2>&1; groupdel fabricolly 2>/dev/null

# Candidate-only root-ID collision checks. The historical predecessor's old
# preinstall is preserved in migration fixtures and is not credited with these.
if [ "$LEGACY_SERVER" = 1 ]; then
  root_before=$(getent passwd root); root_group_before=$(getent group root)
  groupadd --system fabricolly
  useradd -M -o -u 0 -g fabricolly -s /usr/sbin/nologin fabricolly
  out=$(package_install 2>&1); rc=$?
  if [ "$rc" -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
    pass A2c "duplicate root UID refused before unpack"
  else fail A2c "root UID rc=$rc"; fi
  package_purge >/dev/null 2>&1
  userdel -f fabricolly >/dev/null 2>&1
  groupdel fabricolly >/dev/null 2>&1
  groupadd -o -g 0 fabricolly
  out=$(package_install 2>&1); rc=$?
  if [ "$rc" -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
    pass A2d "duplicate root GID refused before unpack"
  else fail A2d "root GID rc=$rc"; fi
  package_purge >/dev/null 2>&1
  groupdel -f fabricolly >/dev/null 2>&1
  [ "$(getent passwd root)" = "$root_before" ] && [ "$(getent group root)" = "$root_group_before" ] || fail SETUP "duplicate-ID fixture altered root identity"
fi

# A3 install, then install again: same identity, exit 0 both times.
package_install >"$W/install1.log" 2>&1; rc1=$?
id1=$(getent passwd fabricolly)
package_install >"$W/install2.log" 2>&1; rc2=$?
id2=$(getent passwd fabricolly)
uid=$(id -u fabricolly 2>/dev/null); gid=$(id -g fabricolly 2>/dev/null); pg=$(id -gn fabricolly 2>/dev/null)
if [ $rc1 -eq 0 ] && [ $rc2 -eq 0 ] && [ "$id1" = "$id2" ] && [ "$uid" -lt 1000 ] && [ "$uid" -gt 0 ] \
   && [ "$pg" = fabricolly ] && [ "$(getent passwd fabricolly | cut -d: -f7)" = /usr/sbin/nologin ]; then
  pass A3 "installed twice; $id1"
else
  fail A3 "rc1=$rc1 rc2=$rc2 id1='$id1' id2='$id2'"
fi
etc_mode=$(stat -c '%a %U:%G' /etc/fabrico11y)
[ "$etc_mode" = "750 root:fabricolly" ] && pass A3b "/etc/fabrico11y $etc_mode" || fail A3b "/etc/fabrico11y $etc_mode"

# A4 systemd-analyze verify on the installed units and slice.
out=$(systemd-analyze verify /usr/lib/systemd/system/fabrico11y-node.service \
      /usr/lib/systemd/system/fabrico11y-server.service /usr/lib/systemd/system/system-fabrico11y.slice 2>&1); rc=$?
[ $rc -eq 0 ] && pass A4 "exit 0" || fail A4 "rc=$rc $out"

# Configuration: a private CA, the server certificate for 127.0.0.1, the admin
# token, and one authorized and one denied log.
cd "$W"
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -keyout ca.key -out ca.pem -days 2 \
  -subj "/CN=fabric accept CA" -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" 2>/dev/null
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -keyout server.key -out server.csr \
  -subj "/CN=127.0.0.1" 2>/dev/null
printf 'subjectAltName=IP:127.0.0.1\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=serverAuth\n' > san.ext
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.pem -days 2 -extfile san.ext 2>/dev/null
install -m 0644 -o root -g root ca.pem server.pem /etc/fabrico11y/
install -m 0640 -o root -g fabricolly server.key /etc/fabrico11y/
(umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > admin-token)
install -m 0640 -o root -g fabricolly admin-token /etc/fabrico11y/admin-token
cat > /etc/fabrico11y/server.conf <<EOF
listen=127.0.0.1:7443
tls_cert=/etc/fabrico11y/server.pem
tls_key=/etc/fabrico11y/server.key
admin_token_file=/etc/fabrico11y/admin-token
state_dir=/var/lib/fabrico11y/server
journal_bytes=21474836480
retention_s=86400
retention_bytes=21474836480
EOF
if [ -n "$SELF_SPINDLE_CA" ]; then
  printf 'self_spindle_ca=%s\n' "$SELF_SPINDLE_CA" >> /etc/fabrico11y/server.conf
fi
chmod 0644 /etc/fabrico11y/server.conf
if [ "$PACKAGE_FAMILY" = fedora ]; then
  restorecon -R /etc/fabrico11y
fi
printf 'server_url=https://127.0.0.1:7443\nserver_ca=/etc/fabrico11y/ca.pem\nadmin_token_file=%s/admin-token\n' "$W" > admin.conf
mkdir -p $LOGDIR && chmod 0755 $LOGDIR
install -m 0600 /dev/null $LOGDIR/allowed.log
install -m 0600 /dev/null $LOGDIR/denied.log
setfacl -m u:fabricolly:r $LOGDIR/allowed.log

if [ "$LEGACY_SERVER" = 1 ]; then
  mkdir -p /etc/systemd/system/fabrico11y-server.service.d
  printf '[Service]\nExecStart=\nExecStart=/usr/bin/fabric-server serve-legacy /etc/fabrico11y/server.conf\n' > /etc/systemd/system/fabrico11y-server.service.d/legacy-fixture.conf
  systemctl daemon-reload
fi
systemctl enable --now fabrico11y-server.service >/dev/null 2>&1
wait_for 30 curl -fsS --cacert /etc/fabrico11y/ca.pem -H "authorization: Bearer $(cat admin-token)" \
  https://127.0.0.1:7443/v1/admin/nodes || fail SETUP "server did not answer"
enroll=$(admin node add spindle-1 --log $LOGDIR/allowed.log --log $LOGDIR/denied.log --interval 15)
token=$(echo "$enroll" | grep -o '"token":"[^"]*"' | cut -d'"' -f4)
(umask 077; printf '%s\n' "$token" > node-token)
install -m 0640 -o root -g fabricolly node-token /etc/fabrico11y/node-token
cat > /etc/fabrico11y/node.conf <<EOF
spool_dir=/var/lib/fabrico11y/node/spool
metric_interval_s=15
spool_bytes=268435456
log=$LOGDIR/allowed.log
log=$LOGDIR/denied.log
server_url=https://127.0.0.1:7443
server_ca=/etc/fabrico11y/ca.pem
token_file=/etc/fabrico11y/node-token
EOF
chmod 0644 /etc/fabrico11y/node.conf
echo "accept-line-before-start" >> $LOGDIR/allowed.log
echo "secret-line-never-read" >> $LOGDIR/denied.log
systemctl enable --now fabrico11y-node.service >/dev/null 2>&1

# A5 TLS: the admin API refuses an unverified client and answers a verified one;
# the node delivers over TLS (rows below).
curl -sS -o /dev/null https://127.0.0.1:7443/v1/admin/nodes 2>/dev/null; untrusted=$?
code=$(curl -sS -o /dev/null -w '%{http_code}' --cacert /etc/fabrico11y/ca.pem \
  -H "authorization: Bearer $(cat admin-token)" https://127.0.0.1:7443/v1/admin/nodes)
[ $untrusted -eq 60 ] && [ "$code" = 200 ] && pass A5 "untrusted client exit 60; CA-verified 200" \
  || fail A5 "untrusted=$untrusted verified=$code"

logs_query() { query "{\"kind\":\"logs\",\"node\":\"spindle-1\",\"from_ns\":0,\"to_ns\":9223372036854775807,\"limit\":1000}"; }
has_line() { logs_query | grep -q "$1"; }

# A6 identity of the running services: non-root UID and primary GID, no capabilities.
wait_for 60 has_line accept-line-before-start
ok=1; detail=""
for unit in fabrico11y-node fabrico11y-server; do
  pid=$(show $unit.service MainPID)
  u=$(awk '/^Uid:/{print $2}' /proc/$pid/status); g=$(awk '/^Gid:/{print $2}' /proc/$pid/status)
  cap=$(awk '/^CapEff:/{print $2}' /proc/$pid/status); nnp=$(awk '/^NoNewPrivs:/{print $2}' /proc/$pid/status)
  detail="$detail $unit:uid=$u,gid=$g,CapEff=$cap,NoNewPrivs=$nnp"
  [ "$u" = "$uid" ] && [ "$g" = "$gid" ] && [ "$u" != 0 ] && [ "$cap" = 0000000000000000 ] && [ "$nnp" = 1 ] || ok=0
done
[ $ok = 1 ] && pass A6 "$detail" || fail A6 "$detail"

# A7 managed directories: owner fabricolly:fabricolly, mode 0700.
ok=1; detail=""
for d in /var/lib/fabrico11y/node /var/lib/fabrico11y/server /run/fabrico11y/node /run/fabrico11y/server; do
  m=$(stat -c '%a %U:%G' "$d" 2>/dev/null || echo missing)
  detail="$detail $d=$m"
  [ "$m" = "700 fabricolly:fabricolly" ] || ok=0
done
[ $ok = 1 ] && pass A7 "$detail" || fail A7 "$detail"

# A8 cgroup placement, effective limits and accounting. On the unified
# hierarchy the files are memory.max/memory.high/pids.max; on the legacy
# hierarchy memory.limit_in_bytes and pids.max, and MemoryHigh has no kernel
# counterpart, so its enforcement (A8h) is not run there.
ok=1; detail=""
cgroup_of() { # pid -> the service's cgroup path in the systemd tree
  if [ "$UNIFIED" = 1 ]; then sed -n 's/^0:://p' /proc/$1/cgroup; else sed -n 's/^[0-9]*:name=systemd://p' /proc/$1/cgroup; fi
}
mem_max_file() { [ "$UNIFIED" = 1 ] && echo "/sys/fs/cgroup$1/memory.max" || echo "/sys/fs/cgroup/memory$1/memory.limit_in_bytes"; }
pids_max_file() { [ "$UNIFIED" = 1 ] && echo "/sys/fs/cgroup$1/pids.max" || echo "/sys/fs/cgroup/pids$1/pids.max"; }
expect() { # unit high max tasks
  local unit=$1 pid cg
  pid=$(show "$unit" MainPID); cg=$(cgroup_of "$pid")
  detail="$detail $unit:cgroup=$cg"
  [ "$cg" = "/system.slice/system-fabrico11y.slice/$unit" ] || ok=0
  for kv in "EffectiveMemoryHigh=$2" "EffectiveMemoryMax=$3" "EffectiveTasksMax=$4"; do
    local got; got=$(show "$unit" "${kv%%=*}")
    detail="$detail ${kv%%=*}=$got"
    [ "$got" = "${kv#*=}" ] || ok=0
  done
  for acct in MemoryCurrent TasksCurrent; do
    local got; got=$(show "$unit" $acct)
    detail="$detail $acct=$got"
    case "$got" in ''|'[not set]'|18446744073709551615) ok=0 ;; esac
  done
  local got_max got_pids
  got_max=$(cat "$(mem_max_file "$cg")" 2>/dev/null); got_pids=$(cat "$(pids_max_file "$cg")" 2>/dev/null)
  detail="$detail kernel-memory-max=$got_max kernel-pids-max=$got_pids"
  [ "$got_max" = "$3" ] && [ "$got_pids" = "$4" ] || ok=0
}
expect fabrico11y-node.service $((128 << 20)) $((256 << 20)) 128
expect fabrico11y-server.service $((2560 << 20)) $((3072 << 20)) 512
slice=/system.slice/system-fabrico11y.slice
s_max=$(cat "$(mem_max_file $slice)" 2>/dev/null); s_pids=$(cat "$(pids_max_file $slice)" 2>/dev/null)
detail="$detail slice kernel-memory-max=$s_max kernel-pids-max=$s_pids"
[ "$s_max" = $((3328 << 20)) ] && [ "$s_pids" = 640 ] || ok=0
[ $ok = 1 ] && pass A8 "$detail" || fail A8 "$detail"
if [ "$UNIFIED" = 1 ]; then
  n_high=$(cat "/sys/fs/cgroup$(cgroup_of "$(show fabrico11y-node.service MainPID)")/memory.high")
  s_high=$(cat /sys/fs/cgroup$slice/memory.high)
  [ "$n_high" = $((128 << 20)) ] && [ "$s_high" = $((2816 << 20)) ] \
    && pass A8h "node memory.high=$n_high slice memory.high=$s_high" || fail A8h "node=$n_high slice=$s_high"
  echo "REPORT controllers slice=[$(cat /sys/fs/cgroup$slice/cgroup.controllers)]"
else
  notrun A8h "legacy cgroup hierarchy: MemoryHigh has no kernel counterpart (systemd reports the value only)"
fi
echo "REPORT IOReadBytes node=$(show fabrico11y-node.service IOReadBytes) (I/O accounting and weights are reported, never counted as enforcement)"

# A9 host metrics and the authorized log, read inside the sandbox.
metric_query='{"kind":"metrics","node":"spindle-1","name":"system.memory.available","from_ns":0,"to_ns":9223372036854775807,"limit":10}'
wait_for 60 sh -c "fabricctl admin $W/admin.conf query '$metric_query' | grep -q '\"time_ns\"'"
m=$(query "$metric_query")
if echo "$m" | grep -q '"time_ns"' && has_line accept-line-before-start; then
  pass A9 "system.memory.available points and the authorized log line delivered"
else
  fail A9 "metrics: $(echo "$m" | head -c 300)"
fi

# A10 the denied log is a visible gap and its content is never delivered.
wait_for 60 sh -c "fabricctl admin $W/admin.conf query '{\"kind\":\"logs\",\"node\":\"spindle-1\",\"from_ns\":0,\"to_ns\":9223372036854775807,\"limit\":1000}' | grep -q 'denied.log'"
answer=$(logs_query)
gap=$(echo "$answer" | grep -o 'log source unavailable [^"]*denied.log[^"]*' | head -1)
if [ -n "$gap" ] && ! echo "$answer" | grep -q secret-line-never-read; then
  pass A10 "gap: $gap"
else
  fail A10 "no denied-log gap or secret delivered"
fi

# A11 graceful shutdown and restart with a retained Spool.
systemctl stop fabrico11y-server.service
server_stop=$(show fabrico11y-server.service Result)
echo "accept-line-during-outage" >> $LOGDIR/allowed.log
sleep 20
systemctl restart fabrico11y-node.service
node_stop=$(show fabrico11y-node.service Result)
spool_files=$(find /var/lib/fabrico11y/node/spool -type f | wc -l)
systemctl start fabrico11y-server.service
wait_for 90 has_line accept-line-during-outage; delivered=$?
has_line accept-line-before-start; kept=$?
if [ "$server_stop" = success ] && [ "$node_stop" = success ] && [ "$spool_files" -gt 0 ] \
   && [ $delivered -eq 0 ] && [ $kept -eq 0 ]; then
  pass A11 "server stop Result=$server_stop; node restart Result=$node_stop; spool files=$spool_files; outage line delivered after restart; earlier line retained"
else
  fail A11 "server=$server_stop node=$node_stop spool=$spool_files delivered=$delivered kept=$kept"
fi

# A12 memory and task pressure are contained in the slice; the services survive.
if [ "$A12_MEMORY_STRESSOR" = parallel ] && [ "$UNIFIED" -eq 0 ]; then
  echo "FIXTURE A12 memory-stressor=parallel helper_sha256=unavailable reason=requires-cgroup-v2"
  notrun A12 "parallel memory stressor requires unified cgroup v2; use the historical tail default on legacy hierarchies"
else
n_before=$(show fabrico11y-node.service NRestarts); s_before=$(show fabrico11y-server.service NRestarts)
# hog <unit> <limit bytes> <worker bytes> <require kernel OOM> <properties...>:
# runs the selected fixture and samples MemoryCurrent until it ends. Contained means
# the peak stayed within the limit and the kernel killed it: Result=oom-kill
# on the unified hierarchy; on the legacy one systemd cannot see OOM events,
# so a SIGKILL (Result=signal, status 9) before RuntimeMaxSec is required.
hog() {
  local unit=$1 limit=$2 worker_bytes=$3 require_kernel_oom=$4 expected_high=$5 expected_tasks=$6
  shift 6
  local before_max=0 before_group=0 after_max=0 after_group=0
  if [ "$require_kernel_oom" = 1 ] && [ "$UNIFIED" = 1 ]; then
    local slice_events=/sys/fs/cgroup/system.slice/system-fabrico11y.slice/memory.events
    while read -r event value; do
      case "$event" in max) before_max=$value ;; oom_group_kill) before_group=$value ;; esac
    done < "$slice_events"
  fi
  if [ "$A12_MEMORY_STRESSOR" = parallel ]; then
    systemd-run --quiet --unit="$unit" --slice=system-fabrico11y.slice \
      -p RuntimeMaxSec=300 -p OOMPolicy=kill -p MemorySwapMax=0 "$@" \
      python3 "$SCRIPT_DIR/a12-memory-pressure.py" --parallel-bytes "$worker_bytes" \
        --memory-high-bytes "$expected_high" --memory-max-bytes "$limit" --tasks-max "$expected_tasks"
  else
    systemd-run --quiet --unit="$unit" --slice=system-fabrico11y.slice -p RuntimeMaxSec=300 "$@" tail /dev/zero
  fi
  local peak=0 cur
  while [ "$(systemctl is-active "$unit" 2>/dev/null)" = active ]; do
    cur=$(show "$unit" MemoryCurrent); case "$cur" in ''|'[not set]') ;; *) [ "$cur" -gt "$peak" ] && peak=$cur ;; esac
    sleep 0.2
  done
  local result status max_events=0 oom_group_events=0 events_state=not-checked
  result=$(show "$unit" Result); status=$(show "$unit" ExecMainStatus)
  if [ "$require_kernel_oom" = 1 ] && [ "$UNIFIED" = 1 ]; then
    local slice_events=/sys/fs/cgroup/system.slice/system-fabrico11y.slice/memory.events event value
    while read -r event value; do
      case "$event" in max) after_max=$value ;; oom_group_kill) after_group=$value ;; esac
    done < "$slice_events"
    max_events=$((after_max - before_max))
    oom_group_events=$((after_group - before_group))
    events_state="slice-delta max=$max_events oom_group_kill=$oom_group_events"
  fi
  systemctl reset-failed "$unit" 2>/dev/null
  local contained=0
  local event_check=1
  if [ "$require_kernel_oom" = 1 ] && [ "$UNIFIED" = 1 ] \
     && { [ "$max_events" -le 0 ] || [ "$oom_group_events" -le 0 ]; }; then event_check=0; fi
  if [ "$peak" -le "$limit" ] && [ "$event_check" = 1 ] \
     && { [ "$result" = oom-kill ] || { [ "$UNIFIED" = 0 ] && [ "$result" = signal ] && [ "$status" = 9 ]; }; }; then
    contained=1
  fi
  echo "$contained $result/$status peak=$peak limit=$limit events=$events_state"
}
positive_control='not-used'; positive_control_ok=1
if [ "$A12_MEMORY_STRESSOR" = parallel ]; then
  helper=$SCRIPT_DIR/a12-memory-pressure.py
  helper_sha=$(sha256sum "$helper" | awk '{print $1}')
  page_bytes=$(getconf PAGESIZE)
  node_worker_bytes=$(((((2 * (256 << 20) + 95) / 96 + page_bytes - 1) / page_bytes) * page_bytes))
  slice_worker_bytes=$(((((2 * (3328 << 20) + 95) / 96 + page_bytes - 1) / page_bytes) * page_bytes))
  echo "FIXTURE A12 memory-stressor=parallel helper_sha256=$helper_sha workers=96 node_worker_bytes=$node_worker_bytes slice_worker_bytes=$slice_worker_bytes"
  if command -v python3 >/dev/null 2>&1 && [ -f "$helper" ]; then
    systemd-run --quiet --unit=fabric-accept-mem-control --slice=system-fabrico11y.slice \
      -p Type=exec -p RemainAfterExit=yes \
      -p RuntimeMaxSec=15 -p MemoryHigh=128M -p MemoryMax=256M -p MemorySwapMax=0 -p TasksMax=128 \
      python3 "$helper" --control --memory-high-bytes $((128 << 20)) \
        --memory-max-bytes $((256 << 20)) --tasks-max 128
    control_peak=0; cur=0; control_end=$((SECONDS + 20))
    while [ "$(show fabric-accept-mem-control SubState)" = running ] && [ "$SECONDS" -lt "$control_end" ]; do
      cur=$(show fabric-accept-mem-control MemoryCurrent)
      case "$cur" in ''|'[not set]') ;; *) [ "$cur" -gt "$control_peak" ] && control_peak=$cur ;; esac
      sleep 0.2
    done
    control_result=$(show fabric-accept-mem-control Result)
    control_status=$(show fabric-accept-mem-control ExecMainStatus)
    control_state=$(show fabric-accept-mem-control SubState)
    systemctl stop fabric-accept-mem-control 2>/dev/null
    systemctl reset-failed fabric-accept-mem-control 2>/dev/null
    positive_control_ok=0
    if [ "$control_peak" -ge $((24 << 20)) ] && [ "$control_peak" -lt $((128 << 20)) ] \
       && [ "$control_state" = exited ] && [ "$control_result" = success ] \
       && [ "$control_status" = 0 ]; then positive_control_ok=1; fi
    positive_control="$control_result/$control_status state=$control_state peak=$control_peak high=$((128 << 20))"
  else
    positive_control_ok=0; positive_control='python3-or-fixture-unavailable'
  fi
else
  echo "FIXTURE A12 memory-stressor=tail acceptance_sha256=$(sha256sum "$0" | awk '{print $1}')"
fi
if [ "$A12_MEMORY_STRESSOR" = parallel ] && [ "$positive_control_ok" -ne 1 ]; then
  echo "A12 parallel positive control failed: $positive_control" >&2
fi
if [ "$A12_MEMORY_STRESSOR" = parallel ]; then
  mem=$(hog fabric-accept-mem $((256 << 20)) "$node_worker_bytes" 1 $((128 << 20)) 128 \
    -p MemoryHigh=128M -p MemoryMax=256M -p TasksMax=128)
  slice_hog=$(hog fabric-accept-slice $((3328 << 20)) "$slice_worker_bytes" 1 $((2816 << 20)) 640)
else
  mem=$(hog fabric-accept-mem $((256 << 20)) 0 0 0 0 -p MemoryHigh=128M -p MemoryMax=256M -p TasksMax=128)
  slice_hog=$(hog fabric-accept-slice $((3328 << 20)) 0 0 0 0)
fi
# A process tree that tries to exceed the Spindle's task limit.
systemd-run --quiet --unit=fabric-accept-tasks --slice=system-fabrico11y.slice -p TasksMax=128 \
  bash -c 'for i in $(seq 300); do sleep 120 & done; wait' >/dev/null 2>&1
tasks_now=0; end=$((SECONDS + 20))
while [ $SECONDS -lt $end ]; do
  cur=$(show fabric-accept-tasks.service TasksCurrent 2>/dev/null)
  case "$cur" in ''|*[!0-9]*) ;; *) [ "$cur" -gt "$tasks_now" ] && tasks_now=$cur ;; esac
  sleep 0.2
done
fork_errors=$(journalctl -q -u fabric-accept-tasks.service --no-pager 2>/dev/null | grep -ci "fork")
systemctl stop fabric-accept-tasks.service 2>/dev/null; systemctl reset-failed fabric-accept-tasks.service 2>/dev/null
n_after=$(show fabrico11y-node.service NRestarts); s_after=$(show fabrico11y-server.service NRestarts)
alive=$(systemctl is-active fabrico11y-node.service fabrico11y-server.service | tr '\n' ' ')
echo "accept-line-after-pressure" >> $LOGDIR/allowed.log
wait_for 60 has_line accept-line-after-pressure; after=$?
if [ "${mem%% *}" = 1 ] && [ "${slice_hog%% *}" = 1 ] && [ "$positive_control_ok" = 1 ] \
   && [ "$tasks_now" -le 128 ] && [ "$tasks_now" -ge 100 ] \
   && [ "$fork_errors" -gt 0 ] && [ "$n_before" = "$n_after" ] && [ "$s_before" = "$s_after" ] \
   && [ "$alive" = "active active " ] && [ $after -eq 0 ]; then
  pass A12 "memory-stressor=$A12_MEMORY_STRESSOR; positive control=$positive_control; service-limit hog: ${mem#* }; slice-limit hog: ${slice_hog#* }; task tree peaked at TasksCurrent=$tasks_now with $fork_errors fork-failure lines; services active, NRestarts unchanged, delivery continued"
else
  fail A12 "memory-stressor=$A12_MEMORY_STRESSOR positive-control=[$positive_control]; mem=[$mem] slice=[$slice_hog] tasks=$tasks_now forks=$fork_errors restarts node $n_before->$n_after server $s_before->$s_after alive='$alive' after=$after"
fi
fi

if [ "$DEFER_REMOVAL" = 0 ]; then
# A13 remove keeps data, configuration and the account; purge removes data and
# configuration.
package_remove >"$W/remove.log" 2>&1; rc=$?
stopped_rc=0
stopped=$(systemctl is-active fabrico11y-node.service fabrico11y-server.service) || stopped_rc=$?
stopped_ok=0
case "$stopped_rc:$stopped" in
  $'3:inactive\ninactive'|$'3:inactive\nfailed'|$'3:failed\ninactive'|$'3:failed\nfailed') stopped_ok=1 ;; esac
if [ $rc -eq 0 ] && [ ! -e /usr/bin/fabric-server ] && [ -d /var/lib/fabrico11y/server ] \
   && [ -f /etc/fabrico11y/server.conf ] && getent passwd fabricolly >/dev/null && [ "$stopped_ok" -eq 1 ]; then
  pass A13a "remove: binaries gone, services '$stopped', state, configuration and account kept"
else
  fail A13a "rc=$rc stopped_rc=$stopped_rc stopped='$stopped'"
fi
package_purge >"$W/purge.log" 2>&1; rc=$?
if [ $rc -eq 0 ] && [ ! -e /var/lib/fabrico11y ] && [ ! -e /etc/fabrico11y ]; then
  if [ "$PACKAGE_FAMILY" = fedora ]; then
    pass A13b "explicit guest cleanup after RPM-preserved removal: state/config removed"
  else pass A13b "purge: /var/lib/fabrico11y and /etc/fabrico11y removed"; fi
else
  fail A13b "rc=$rc"
fi

else
  echo "DEFERRED A13 removal to registered upgrade/reboot lifecycle phases"
fi

if [ "$PACKAGE_FAMILY" = fedora ]; then
  [ "$(getenforce)" = Enforcing ] && pass F2 "SELinux remained enforcing through delivery/restart/removal" || fail F2 "SELinux changed mode"
  audit_status=$(auditctl -s 2>&1); audit_rc=$?
  if [ "$audit_rc" -ne 0 ] || ! echo "$audit_status" | grep -Eq '^enabled [12]$'; then
    fail F3 "audit unavailable/disabled: $audit_status"
  fi
  ausearch -m AVC,USER_AVC -ts boot > "$W/selinux-avc.log" 2>&1; audit_search_rc=$?
  [ "$audit_search_rc" -le 1 ] || fail F3 "audit search failed: $audit_search_rc"
  grep -E 'comm="(fabric-server|fabric-node|fabricctl)"' "$W/selinux-avc.log"; avc_rc=$?
  case "$avc_rc" in
    0) fail F3 "Fabric process SELinux denial; retain AVC evidence" ;;
    1) pass F3 "no Fabric process AVC in guest audit" ;;
    *) fail F3 "audit read error rc=$avc_rc; retain audit evidence" ;;
  esac
fi
echo "RESULT fails=$FAILS not_run=$NOTRUN"
[ $FAILS -gt 0 ] && exit 1
[ $NOTRUN -gt 0 ] && exit 3
exit 0
⚠️ **GitHub.com Fallback** ⚠️