Experiment formal source prerequisite green plan 01 - kmosoti/FabricO11y GitHub Wiki
Prospective only. No new green or native acceptance inferred. Existing red logs and receipts remain retained.
SHA256 d98621757427284d23170676af8496372b7d3dede043ac4efa8eb48203d58d85.
# Finite source prerequisites, prospective green verification
No candidate VM, browser, native clock binding, owner constructor or acceptance
workload runs under this plan. One serial Olo executor, fresh strict census and
full 1 GiB reservation per command, default16/20GiB/noSwap/1800s,95GB unchanged.
Read #59 before and after each command; only advance on an actual stopped
receipt. Neither source correction nor a finite passing test is qualification.
META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
Clock clean head a74ab66bae81197d8f7a22f61ccba1a686b7a79e:
```sh
python3 -B META/run-step-console-clock-green.py console-clock-controls-green-01 1073741824 env PYTHONPATH=tools python3 -B -m unittest discover -s tools/ui -p test_session_clock.py -vThe four unchanged lifecycle methods previously produced eight real assertion failures at d33fb33, exit1/stopped. Two earlier import failures are separate setup failures. Frozen image hash/dev/inode/map witnesses and private process group shutdown/diagnostic retention were corrected; native cleanup failures now retain staging and capture existing ledger/frames regardless close error. Tests76f937331e2ebe6e2da9ee121a39a0411c823f38bd788c3c4a36107cd3dcec49, oracle4817a0d09c1c920a67242637cbbb3a6473d6bea42ce716d07a73e8d7122d1728, protocoldea94fa0109a4e757ac2c022b2829e2ec4073678feb077b1472b32a9ed6cdab3 are unchanged. Productionf183ce8caf2f73fe16e11f9de3342b4368a5990a17964e75861cecb4c4e68a34, net+67 within the extra85-line scope. Independent Fenn static review followed by its small archive correction; actual full controls still not run.
Installation checker clean head db5b9d86ff70795a4b7ecec046b60d07d8e4bb1a:
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-controls-02 1073741824 python3 -B tools/qualification/install/test_run_qemu.py
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-shell-syntax-02 1073741824 bash -n tools/qualification/install/acceptance.sh tools/qualification/install/run.sh tools/qualification/install/lifecycle.sh
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-docs-check-02 1073741824 env PATH=/run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0:/usr/local/bin:/usr/bin:/bin BUN_INSTALL_CACHE_DIR=/run/media/kmosoti/data/FabricO11y/tools/docs-bun-cache /run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0/bun tools/docs/check.mjs
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-docs-tests-02 1073741824 env PATH=/run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0:/usr/local/bin:/usr/bin:/bin BUN_INSTALL_CACHE_DIR=/run/media/kmosoti/data/FabricO11y/tools/docs-bun-cache /run/media/kmosoti/data/FabricO11y/tools/bun-1.4.0/bun tools/docs/check.test.mjs
python3 -B META/run-step-installation-checker-final-02.py installation-checker-final-docs-hooks-02 1073741824 python3 -B tools/docs/test_hooks.pyThe final F3 control is literal readable positive versus grep-error2 on the real acceptance block, without errexit masking. At42e45f5 one intended failing subcase accepted grep2 as PASS (exit1/stopped); f437f0e now distinguishes rc0/1/ other. Unchanged test35bbbc144065a22bd37390aed7325ded8f56b3bdfbee672944fc19292cfba744; acceptancec287ad707ea97ea5e25033a6d78d425b4a324016b69144458255b07d95b30b73. Rev5 is prospective, result prose migrated to wiki and CURRENT describes behavior only. CX-INSTALLATION-BASELINE-AUDIT-READ registered separately. Full suite has ten methods. These actual greens precede ONE final PR66 push; Pip delta review and exact final-head hosted CI precede owner merge. Historical remote fcd57 hosted fast38333544753/extended38333544797 both succeeded, and belong only to that revision. No local cold Cargo checks in this plan.
Docs use the already locked DATA dependency directory through a real ignored node_modules directory with entry symlinks. No install/cache write is requested; post-move dependency view has not yet been checked. Existing lock/package hashes remain exact. All source-result data is wiki-only. Every command label is unique.
## run-step-installation-checker-final-02.py
SHA256 `84114964a3b9c595aa04ea4afb68798c29f740830973b33bbb197089e85761a1`.
```python
import datetime, hashlib, json, os, re, signal, subprocess, sys, time
from pathlib import Path
ROOT = Path('/run/media/kmosoti/data/FabricO11y')
REPO = ROOT / 'scratch/installation-checker-repair'
OUT = ROOT / 'results/service-opening-candidate'
STORE = ROOT / 'scratch/service-opening-census'
FROZEN = 'db5b9d86ff70795a4b7ecec046b60d07d8e4bb1a'
def require(value, reason):
if not value:
raise ValueError(reason)
label, reserve_text, *command = sys.argv[1:]
reserve = int(reserve_text)
delegated = command[:1] == ['--delegate']
if delegated: command = command[1:]
require(re.fullmatch(r'[a-z0-9-]{1,80}', label) and reserve > 0 and command, 'ordinary workload admission guard')
record_path = OUT / (label + '.json')
require(not record_path.exists(), 'unique run label required')
def census():
# Rootless container files can belong to subordinate UIDs. Inspect all of
# them in the same user namespace; never skip unreadable directories.
require(not STORE.is_symlink(), 'ordinary workload admission guard')
require((STORE / '.owned-census-store').read_text() == 'service-opening-census-v1\n', 'ordinary workload admission guard')
args = ['podman', '--root', str(STORE/'graph'), '--runroot', str(STORE/'run'),
'--tmpdir', str(STORE/'tmp'), '--storage-driver', 'vfs',
'unshare', 'python3', '-B', str(OUT/'census.py')]
env = dict(os.environ, TMPDIR=str(STORE/'tmp'), TMP=str(STORE/'tmp'), TEMP=str(STORE/'tmp'))
result = subprocess.run(args, env=env, text=True, capture_output=True, timeout=25)
if result.returncode:
raise RuntimeError(f'strict namespace census failed ({result.returncode}): {result.stderr}')
observed = json.loads(result.stdout)
require(observed['uid'] == 0 and observed['gid'] == 0, 'ordinary workload admission guard')
require(type(observed['data_bytes']) is int and observed['data_bytes'] >= 0, 'ordinary workload admission guard')
return observed
def git(*args):
return subprocess.check_output(['git', *args], cwd=REPO, text=True).strip()
require(git('rev-parse', 'HEAD') == FROZEN and not git('status', '--porcelain'), 'clean frozen source required')
require(not subprocess.check_output(['systemctl', '--user', 'list-units', 'fabric-work-*', '--state=running', '--plain', '--no-legend'], text=True).strip(), 'another project workload is active')
initial = census()
used = initial['data_bytes']
record = {'label': label, 'command': command, 'delegated': delegated, 'frozen': None, 'scope': 'harness controls only; no candidate acceptance evidence', 'harness_revision': FROZEN, 'source_status_before': git('status', '--porcelain'), 'storage_root': str(ROOT), 'admission_bytes': used, 'admission_observation': initial, 'reserved_bytes': reserve, 'observations': [], 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat()}
if used + reserve > 95_000_000_000:
record.update(state='not_run', reason='95 GB storage admission threshold')
record_path.write_text(json.dumps(record, indent=2)+'\n')
raise SystemExit('storage admission denied; project command not launched')
receipts = REPO / 'target/resource-containment/runs'
before = set(receipts.glob('*.json'))
env = dict(os.environ, FABRIC_STORAGE_ROOT=str(ROOT), CARGO_HOME=str(ROOT/'toolchain-cache/cargo'), RUSTUP_HOME=str(ROOT/'toolchain-cache/rustup'), CARGO_BUILD_JOBS='2')
env['PATH'] = str(ROOT/'tools/candidate-dpkg/bin') + ':' + env['PATH']
log_path = OUT / (label + '.log')
record.update(state='running', log=str(log_path))
record_path.write_text(json.dumps(record, indent=2)+'\n')
def find_unit():
with log_path.open() as stream:
prefix = stream.read(65536)
match = re.search(r'Running as unit: (fabric-work-[0-9a-f]+)\.service', prefix)
return match.group(1) if match else None
def stop_owned(process):
unit = find_unit()
if unit:
stopped = subprocess.run(['systemctl', '--user', 'stop', unit],
text=True, capture_output=True, timeout=25)
record['stop'] = {'unit': unit, 'exit': stopped.returncode, 'stderr': stopped.stderr}
if stopped.returncode:
# Interrupt the launcher too; its finally block independently
# confirms that this exact service has no surviving descendants.
process.send_signal(signal.SIGINT)
else:
process.send_signal(signal.SIGINT)
record['stop'] = {'unit': None, 'launcher_interrupted': True}
with log_path.open('wb') as log:
launcher = ['python3', '-B', 'tools/resource_group.py', *(['--delegate'] if delegated else []), '--', *command]
process = subprocess.Popen(launcher, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT)
next_census = time.monotonic() + 30
try:
while process.poll() is None:
time.sleep(1)
if time.monotonic() >= next_census:
observed = census()
record['observations'].append({'unix':time.time(), **observed})
record_path.write_text(json.dumps(record, indent=2)+'\n')
if observed['data_bytes'] >= 95_000_000_000:
record['interruption'] = '95 GB threshold reached'
break
next_census = time.monotonic() + 30
except BaseException as error:
record['interruption'] = f'{type(error).__name__}: {error}'
finally:
if record.get('interruption') and process.poll() is None:
try:
stop_owned(process)
except BaseException as error:
record['stop_error'] = f'{type(error).__name__}: {error}'
process.send_signal(signal.SIGINT)
status = process.wait()
record['launcher_exit'] = status
if record.get('interruption'): status = 2
with log_path.open('rb') as stream:
log_digest = hashlib.file_digest(stream, 'sha256').hexdigest()
record.update(state='interrupted' if record.get('interruption') else 'completed', exit=status, source_status_after=git('status', '--porcelain'), finished_utc=datetime.datetime.now(datetime.timezone.utc).isoformat(), log_sha256=log_digest, resource_receipts=[])
archive = OUT/'resource-containment'; archive.mkdir(exist_ok=True)
for p in sorted(set(receipts.glob('*.json')) - before):
value=json.loads(p.read_text())
if value['command'] != command: continue
q=archive/p.name; q.write_bytes(p.read_bytes())
record['resource_receipts'].append({'path':str(q),'sha256':hashlib.sha256(q.read_bytes()).hexdigest(),'exit':value['exit'],'stop_confirmed':value['stop_confirmed'],'retained_failure_evidence':value['retained_failure_evidence']})
if not record['resource_receipts'] or any(not r['stop_confirmed'] or r['exit'] != status for r in record['resource_receipts']):
record['receipt_error'] = 'missing or inconsistent outer resource receipt'
status = 2
record['exit'] = status
record_path.write_text(json.dumps(record, indent=2)+'\n')
print(json.dumps(record, indent=2))
if status: print(log_path.read_text()[-8000:])
raise SystemExit(status)
SHA256 7e2e1038a4262a3bcdbfac9c6f19c5ab658666757ed1b1cce9a3292ac4a58dcf.
import datetime, hashlib, json, os, re, signal, subprocess, sys, time
from pathlib import Path
ROOT = Path('/run/media/kmosoti/data/FabricO11y')
REPO = ROOT / 'scratch/console-access-acceptance'
OUT = ROOT / 'results/service-opening-candidate'
STORE = ROOT / 'scratch/service-opening-census'
FROZEN = 'a74ab66bae81197d8f7a22f61ccba1a686b7a79e'
def require(value, reason):
if not value:
raise ValueError(reason)
label, reserve_text, *command = sys.argv[1:]
reserve = int(reserve_text)
delegated = command[:1] == ['--delegate']
if delegated: command = command[1:]
require(re.fullmatch(r'[a-z0-9-]{1,80}', label) and reserve > 0 and command, 'ordinary workload admission guard')
record_path = OUT / (label + '.json')
require(not record_path.exists(), 'unique run label required')
def census():
# Rootless container files can belong to subordinate UIDs. Inspect all of
# them in the same user namespace; never skip unreadable directories.
require(not STORE.is_symlink(), 'ordinary workload admission guard')
require((STORE / '.owned-census-store').read_text() == 'service-opening-census-v1\n', 'ordinary workload admission guard')
args = ['podman', '--root', str(STORE/'graph'), '--runroot', str(STORE/'run'),
'--tmpdir', str(STORE/'tmp'), '--storage-driver', 'vfs',
'unshare', 'python3', '-B', str(OUT/'census.py')]
env = dict(os.environ, TMPDIR=str(STORE/'tmp'), TMP=str(STORE/'tmp'), TEMP=str(STORE/'tmp'))
result = subprocess.run(args, env=env, text=True, capture_output=True, timeout=25)
if result.returncode:
raise RuntimeError(f'strict namespace census failed ({result.returncode}): {result.stderr}')
observed = json.loads(result.stdout)
require(observed['uid'] == 0 and observed['gid'] == 0, 'ordinary workload admission guard')
require(type(observed['data_bytes']) is int and observed['data_bytes'] >= 0, 'ordinary workload admission guard')
return observed
def git(*args):
return subprocess.check_output(['git', *args], cwd=REPO, text=True).strip()
require(git('rev-parse', 'HEAD') == FROZEN and not git('status', '--porcelain'), 'clean frozen source required')
require(not subprocess.check_output(['systemctl', '--user', 'list-units', 'fabric-work-*', '--state=running', '--plain', '--no-legend'], text=True).strip(), 'another project workload is active')
initial = census()
used = initial['data_bytes']
record = {'label': label, 'command': command, 'delegated': delegated, 'frozen': None, 'scope': 'harness controls only; no candidate acceptance evidence', 'harness_revision': FROZEN, 'source_status_before': git('status', '--porcelain'), 'storage_root': str(ROOT), 'admission_bytes': used, 'admission_observation': initial, 'reserved_bytes': reserve, 'observations': [], 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat()}
if used + reserve > 95_000_000_000:
record.update(state='not_run', reason='95 GB storage admission threshold')
record_path.write_text(json.dumps(record, indent=2)+'\n')
raise SystemExit('storage admission denied; project command not launched')
receipts = REPO / 'target/resource-containment/runs'
before = set(receipts.glob('*.json'))
env = dict(os.environ, FABRIC_STORAGE_ROOT=str(ROOT), CARGO_HOME=str(ROOT/'toolchain-cache/cargo'), RUSTUP_HOME=str(ROOT/'toolchain-cache/rustup'), CARGO_BUILD_JOBS='2')
env['PATH'] = str(ROOT/'tools/candidate-dpkg/bin') + ':' + env['PATH']
log_path = OUT / (label + '.log')
record.update(state='running', log=str(log_path))
record_path.write_text(json.dumps(record, indent=2)+'\n')
def find_unit():
with log_path.open() as stream:
prefix = stream.read(65536)
match = re.search(r'Running as unit: (fabric-work-[0-9a-f]+)\.service', prefix)
return match.group(1) if match else None
def stop_owned(process):
unit = find_unit()
if unit:
stopped = subprocess.run(['systemctl', '--user', 'stop', unit],
text=True, capture_output=True, timeout=25)
record['stop'] = {'unit': unit, 'exit': stopped.returncode, 'stderr': stopped.stderr}
if stopped.returncode:
# Interrupt the launcher too; its finally block independently
# confirms that this exact service has no surviving descendants.
process.send_signal(signal.SIGINT)
else:
process.send_signal(signal.SIGINT)
record['stop'] = {'unit': None, 'launcher_interrupted': True}
with log_path.open('wb') as log:
launcher = ['python3', '-B', 'tools/resource_group.py', *(['--delegate'] if delegated else []), '--', *command]
process = subprocess.Popen(launcher, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT)
next_census = time.monotonic() + 30
try:
while process.poll() is None:
time.sleep(1)
if time.monotonic() >= next_census:
observed = census()
record['observations'].append({'unix':time.time(), **observed})
record_path.write_text(json.dumps(record, indent=2)+'\n')
if observed['data_bytes'] >= 95_000_000_000:
record['interruption'] = '95 GB threshold reached'
break
next_census = time.monotonic() + 30
except BaseException as error:
record['interruption'] = f'{type(error).__name__}: {error}'
finally:
if record.get('interruption') and process.poll() is None:
try:
stop_owned(process)
except BaseException as error:
record['stop_error'] = f'{type(error).__name__}: {error}'
process.send_signal(signal.SIGINT)
status = process.wait()
record['launcher_exit'] = status
if record.get('interruption'): status = 2
with log_path.open('rb') as stream:
log_digest = hashlib.file_digest(stream, 'sha256').hexdigest()
record.update(state='interrupted' if record.get('interruption') else 'completed', exit=status, source_status_after=git('status', '--porcelain'), finished_utc=datetime.datetime.now(datetime.timezone.utc).isoformat(), log_sha256=log_digest, resource_receipts=[])
archive = OUT/'resource-containment'; archive.mkdir(exist_ok=True)
for p in sorted(set(receipts.glob('*.json')) - before):
value=json.loads(p.read_text())
if value['command'] != command: continue
q=archive/p.name; q.write_bytes(p.read_bytes())
record['resource_receipts'].append({'path':str(q),'sha256':hashlib.sha256(q.read_bytes()).hexdigest(),'exit':value['exit'],'stop_confirmed':value['stop_confirmed'],'retained_failure_evidence':value['retained_failure_evidence']})
if not record['resource_receipts'] or any(not r['stop_confirmed'] or r['exit'] != status for r in record['resource_receipts']):
record['receipt_error'] = 'missing or inconsistent outer resource receipt'
status = 2
record['exit'] = status
record_path.write_text(json.dumps(record, indent=2)+'\n')
print(json.dumps(record, indent=2))
if status: print(log_path.read_text()[-8000:])
raise SystemExit(status)SHA256 f183ce8caf2f73fe16e11f9de3342b4368a5990a17964e75861cecb4c4e68a34.
"""Disposable target-only clock fixture and contained native binding admission."""
import argparse
import ctypes
import hashlib
import json
import os
from pathlib import Path
import platform
import select
import shutil
import signal
import socket
import subprocess
import sys
import tempfile
import threading
import time
import session_clock_oracle as oracle
HERE=Path(__file__).resolve().parent
sys.path.insert(0,str(HERE.parent))
from resource_group import STORAGE, require_limits
def digest(path):
with Path(path).open('rb') as stream:return hashlib.file_digest(stream,'sha256').hexdigest()
class KernelTime(ctypes.Structure):_fields_=[('sec',ctypes.c_long),('nsec',ctypes.c_long)]
def kernel_time():
value=KernelTime();oracle.require(ctypes.CDLL(None,use_errno=True).syscall(228,0,ctypes.byref(value))==0,'fixture-raw-clock');return value.sec*1000000000+value.nsec
def freeze_images(executable,helper):
frozen={}
for name,path in (('executable',executable),('helper',helper)):
before=path.stat();frozen[name+'_sha256']=digest(path);after=path.stat()
oracle.require((before.st_dev,before.st_ino,before.st_size,before.st_mtime_ns,before.st_ctime_ns)==(after.st_dev,after.st_ino,after.st_size,after.st_mtime_ns,after.st_ctime_ns),'isolation-'+('target' if name=='executable' else 'helper')+'-frozen')
frozen[name+'_identity']={'device':before.st_dev,'inode':before.st_ino}
return frozen
def process_binding(pid, executable, helper, intercepted=True, *, expected=None):
proc=Path(f'/proc/{pid}');actual=proc/'exe'
if expected is not None:
for name,paths,message in (('executable',(actual,executable),'isolation-target-frozen'),('helper',(helper,),'isolation-helper-frozen')):
for path in paths:
stat=path.stat()
oracle.require({'device':stat.st_dev,'inode':stat.st_ino}==expected[name+'_identity'] and digest(path)==expected[name+'_sha256'],message)
oracle.require(actual.resolve()==executable.resolve() and digest(actual)==digest(executable),'isolation-target')
rows=[line.split(maxsplit=5) for line in (proc/'maps').read_text().splitlines()]
mappings={(row[3],row[4],row[5]) for row in rows if len(row)==6 and row[5]==str(helper)};stat=helper.stat()
if expected is not None and mappings:
frozen=expected['helper_identity']
oracle.require(mappings=={(f'{os.major(frozen["device"]):02x}:{os.minor(frozen["device"]):02x}',str(frozen['inode']),str(helper))},'isolation-helper-frozen')
oracle.require(mappings==({(f'{os.major(stat.st_dev):02x}:{os.minor(stat.st_dev):02x}',str(stat.st_ino),str(helper))} if intercepted else set()),'isolation-interception')
libc=next(Path(row[5]) for row in rows if len(row)==6 and row[5].endswith('/libc.so.6'))
return {'pid':pid,'executable_sha256':digest(actual),'helper_sha256':digest(helper) if mappings else None,'libc':str(libc),'libc_sha256':digest(libc),
'start_ticks':int((proc/'stat').read_text().rsplit(')',1)[1].split()[19]),'cgroup':(proc/'cgroup').read_text(),'helper_maps':sorted(mappings)}
def stop_group(process):
try:os.killpg(process.pid,signal.SIGTERM)
except ProcessLookupError:process.wait(timeout=3);return
try:process.wait(timeout=3)
except subprocess.TimeoutExpired:pass
try:os.killpg(process.pid,signal.SIGKILL)
except ProcessLookupError:pass
process.wait(timeout=3);deadline=time.monotonic()+3
while True:
try:
while os.waitpid(-process.pid,os.WNOHANG)[0]:pass
except ChildProcessError:pass
try:os.killpg(process.pid,0)
except ProcessLookupError:return
oracle.require(time.monotonic()<deadline,'fixture-process-group-stop');time.sleep(.01)
class SessionClock:
def __init__(self, executable, helper, directory, deadline, initial=None, socket_type=socket.SOCK_SEQPACKET, preload=True, invalid_fd=False):
self.deadline=deadline;self.frames=[];self.rows=[];self.bytes=0;self.error=None;self.buffer=b''
env=dict(os.environ);oracle.require(not any(name in env for name in ('LD_PRELOAD','LD_AUDIT','FABRIC_CLOCK_FD')),'fixture-environment')
self.parent,self.child=socket.socketpair(socket.AF_UNIX,socket_type|socket.SOCK_NONBLOCK|socket.SOCK_CLOEXEC)
self.fd=self.child.fileno();self.socket_identity=os.readlink(f'/proc/self/fd/{self.fd}')
self.parent.send(initial if initial is not None else self.frame(0));self.stderr=directory.open('xb')
if preload:env.update(LD_PRELOAD=str(helper),FABRIC_CLOCK_FD=str(1048575 if invalid_fd else self.fd))
self.argv=[str(executable),'interactive',str(self.fd),str(helper),self.socket_identity]
try:self.process=subprocess.Popen(self.argv,env=env,pass_fds=(self.fd,),stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=self.stderr,start_new_session=True)
except BaseException:self.parent.close();self.child.close();self.stderr.close();raise
self.child.close();self.receiver=threading.Thread(target=self.receive,daemon=True);self.receiver.start()
def frame(self,offset):
frame=b'FCLKv1\0\0'+len(self.frames).to_bytes(8,'little')+offset.to_bytes(8,'little')
oracle.check_frames(self.frames+[frame]);self.frames.append(frame);return frame
def advance(self,offset):oracle.require(self.parent.send(self.frame(offset))==24,'fixture-control-send')
def receive(self):
try:
while True:
if not select.select([self.parent],[],[],.1)[0]:continue
raw=self.parent.recv(257)
if not raw:return
self.bytes+=len(raw);oracle.require(len(raw)<=256 and self.bytes<=1048576,'fixture-ledger-bound')
self.rows.append(json.loads(raw))
except BaseException as error:
self.error=type(error).__name__+': '+str(error)
try:os.killpg(self.process.pid,signal.SIGTERM)
except ProcessLookupError:pass
def line(self):
while b'\n' not in self.buffer:
oracle.require(self.error is None,'fixture-ledger-error')
remaining=self.deadline-time.monotonic();oracle.require(remaining>0,'fixture-deadline')
oracle.require(select.select([self.process.stdout],[],[],remaining)[0],'fixture-deadline')
raw=os.read(self.process.stdout.fileno(),4096);oracle.require(bool(raw),'fixture-probe-eof')
self.buffer+=raw;oracle.require(len(self.buffer)<=65536,'fixture-probe-bound')
raw,self.buffer=self.buffer.split(b'\n',1);return json.loads(raw)
def command(self,name):self.process.stdin.write((name+'\n').encode());self.process.stdin.flush()
def close(self):
try:stop_group(self.process)
finally:
self.receiver.join(timeout=1);self.parent.close();self.process.stdin.close();self.process.stdout.close();self.stderr.close()
oracle.require(not self.receiver.is_alive(),'fixture-receiver-stop')
try:os.killpg(self.process.pid,0)
except ProcessLookupError:return
raise ValueError('fixture-process-group-stop')
def native_admission(executable,helper,output,deadline,report):
def close(current,evidence):
try:current.close()
except BaseException as error:
report['native_cleanup_error']=type(error).__name__+': '+str(error);raise
finally:
evidence['ledger']=list(current.rows);evidence['frames']=[frame.hex() for frame in current.frames]
target=SessionClock(executable,helper,output/'positive.stderr',deadline);report['clocks']=[];report['system_time']=[]
try:
ready=target.line();report['ready']=ready
oracle.require(ready=={'kind':'ready','pid':target.process.pid,'env_clear':True,'cloexec':True,'socket_type':5} and os.readlink(f'/proc/{target.process.pid}/fd/{target.fd}')==target.socket_identity and int(Path(f'/proc/{target.process.pid}/fdinfo/{target.fd}').read_text().split('flags:',1)[1].split()[0],8)&os.O_CLOEXEC,'isolation-control')
report['target']=process_binding(target.process.pid,executable,helper,expected=report['frozen_images']);report['controller']=process_binding(os.getpid(),Path(sys.executable),helper,False);report['host_clocks']=[]
report['control_fd']={'identity':target.socket_identity,'fdinfo':Path(f'/proc/{target.process.pid}/fdinfo/{target.fd}').read_text()};report['probe_argv']=target.argv
for sequence,offset in enumerate((0,1,1799,1800,28799,28800)):
if sequence:target.advance(offset)
if sequence==2:
target.command('fork');fork,child,done=(target.line() for _ in range(3));report['fork']=[fork,child,done]
row=fork['row'];oracle.require(row['pid']==child['pid']==done['fork_pid']!=target.process.pid and done['status']==0,'isolation-fork-pid')
oracle.require(row['offset_s']==1 and row['sequence']==done['sequence_before']==1 and row['returned_ns']==row['raw_ns'],'isolation-fork-consumption')
oracle.require(row['syscall_before_ns']<=row['raw_ns']<=row['syscall_after_ns'] and row['syscall_after_ns']-row['syscall_before_ns']<=1000000000,'isolation-host-clock')
oracle.require(child['kind']=='exec' and child['env_clear'] and not child['helper_mapped'] and not child['control_fd_present'],'isolation-exec-escape')
oracle.require(child['syscall_before_ns']<=child['clock_ns']<=child['syscall_after_ns'] and child['syscall_after_ns']-child['syscall_before_ns']<=1000000000,'isolation-host-clock')
target.command('clock')
for _ in range(3):
value=target.line();oracle.require(value['kind']=='clock','clock-binding');report['clocks'].append(value['row'])
system=target.line();oracle.require(system['kind']=='system','clock-binding');report['system_time'].append(system['row'])
before=kernel_time();clock=time.time_ns();after=kernel_time();oracle.require(before<=clock<=after and after-before<=1000000000 and (not report['host_clocks'] or before>=report['host_clocks'][-1]['syscall_after_ns']),'isolation-host-clock');report['host_clocks'].append({'sequence':sequence,'syscall_before_ns':before,'clock_ns':clock,'syscall_after_ns':after})
target.command('sleep');sleep=target.line();oracle.require(sleep['kind']=='sleep','sleep-fields');report['sleep']=sleep['row']
oracle.check_frames(target.frames);oracle.check_probe_clocks(report['clocks'],target.process.pid);oracle.check_sleep(report['sleep'])
for row,clock in zip(report['system_time'],report['clocks'][::3]):
oracle.require((row['pid'],row['sequence'],row['offset_s'],row['clock_id'])==(clock['pid'],clock['sequence'],clock['offset_s'],0),'clock-binding')
oracle.require(row['syscall_before_ns']<=row['raw_ns']<=row['syscall_after_ns'] and row['syscall_after_ns']-row['syscall_before_ns']<=1000000000,'clock-raw-bracket')
oracle.require(row['returned_ns']==row['raw_ns']+row['offset_s']*1000000000,'clock-realtime-only')
target.command('quit');oracle.require(target.process.wait(timeout=max(.001,deadline-time.monotonic()))==0,'fixture-probe-exit')
finally:
close(target,report)
oracle.require(target.error is None,'fixture-ledger-error')
for row in report['clocks']+report['system_time']:
oracle.require({key:value for key,value in row.items() if not key.startswith('syscall_')} in report['ledger'],'clock-ledger-binding')
report['controls']=[]
initial=b'FCLKv1\0\0'+bytes(16)
controls=[('short',initial[:-1],'FCLK-FRAME'),('oversize',initial+b'x','FCLK-FRAME'),('version',b'bad-v001'+bytes(16),'FCLK-FRAME'),
('initial-sequence',initial[:8]+(1).to_bytes(8,'little')+bytes(8),'FCLK-SEQUENCE'),('initial-offset',initial[:16]+(1).to_bytes(8,'little'),'FCLK-OFFSET'),
('skip',None,'FCLK-SEQUENCE'),('duplicate',None,'FCLK-SEQUENCE'),('decrease',None,'FCLK-OFFSET'),('cap',None,'FCLK-OFFSET'),('overflow',None,'FCLK-OFFSET'),
('count',None,'FCLK-COUNT'),('death',None,'FCLK-EOF'),('closed',None,'FCLK-RECV'),('descriptor',None,'FCLK-FD'),('type',None,'FCLK-TYPE'),('unpreloaded',None,'isolation-interception')]
for name,packet,expected in controls:
error_path=output/(name+'.stderr');current=SessionClock(executable,helper,error_path,deadline,initial=packet,socket_type=socket.SOCK_STREAM if name=='type' else socket.SOCK_SEQPACKET,preload=name!='unpreloaded',invalid_fd=name=='descriptor')
evidence={'case':name,'refusal':expected,'exit':None,'stderr':None};report['controls'].append(evidence)
try:
if name=='unpreloaded':
oracle.require(current.line()=={'kind':'absent','pid':current.process.pid},'fixture-unpreloaded')
try:process_binding(current.process.pid,executable,helper,expected=report['frozen_images'])
except ValueError as error:oracle.require(str(error)==expected,'fixture-wrong-refusal')
else:raise ValueError('fixture-unpreloaded-admitted')
else:
if packet is None and name not in ('type','descriptor'):
current.line()
if name=='death':current.parent.shutdown(socket.SHUT_WR)
elif name=='decrease':current.parent.send(initial[:8]+(1).to_bytes(8,'little')+(1).to_bytes(8,'little'));current.parent.send(initial[:8]+(2).to_bytes(8,'little')+bytes(8))
else:
values=[(i,0) for i in range(1,65)] if name=='count' else [(2 if name=='skip' else 0 if name=='duplicate' else 1,28861 if name=='cap' else 2**64-1 if name=='overflow' else 0)]
if name!='closed':
for seq,value in values:current.parent.send(initial[:8]+seq.to_bytes(8,'little')+value.to_bytes(8,'little'))
current.command('close' if name=='closed' else 'clock')
oracle.require(current.process.wait(timeout=max(.001,deadline-time.monotonic()))==78,'fixture-control-exit')
finally:close(current,evidence)
stderr=error_path.read_text();oracle.require(name=='unpreloaded' or stderr.strip()==expected,'fixture-wrong-refusal')
evidence.update(exit=current.process.returncode,stderr=stderr)
def main():
parser=argparse.ArgumentParser(description=__doc__);parser.add_argument('--output',type=Path,required=True);parser.add_argument('--rustc',required=True);parser.add_argument('--cc',default='gcc');args=parser.parse_args()
require_limits();oracle.require(platform.system()=='Linux' and platform.machine()=='x86_64','fixture-platform')
output=args.output;oracle.require(output.is_absolute() and output.resolve()==output and output.is_relative_to(STORAGE),'fixture-output');output.mkdir(mode=0o700)
scratch=Path(os.environ['FABRIC_SCRATCH_ROOT']);oracle.require(scratch.is_dir() and scratch.is_relative_to(STORAGE),'fixture-scratch')
directory=Path(tempfile.mkdtemp(prefix='session-clock-',dir=scratch));deadline=time.monotonic()+240;report={'scope':'native binding only; actual packaged browser session checks pending','state':'failed','exit':1,'commands':[]}
def run(command):
report['commands'].append(command);path=output/('tool-'+str(len(report['commands']))+'.log');timeout=max(.001,deadline-time.monotonic());failure=None
with path.open('xb') as stream:
process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT,start_new_session=True)
try:
while process.poll() is None:
if time.monotonic()>=deadline:raise subprocess.TimeoutExpired(command,timeout)
oracle.require(path.stat().st_size<=1048576,'fixture-tool-output');time.sleep(.01)
except BaseException as error:failure=error;raise
finally:
try:stop_group(process)
except BaseException as error:
report['tool_cleanup_error']=type(error).__name__+': '+str(error)
if failure is None:raise
finally:
with path.open('rb') as captured:raw=captured.read(1048576)
report.setdefault('tool_outputs',[]).append({'command':command,'exit':process.returncode,'raw_log':str(path),'output':raw.decode(errors='replace')})
if isinstance(failure,subprocess.TimeoutExpired):failure.output=raw
oracle.require(path.stat().st_size<=1048576,'fixture-tool-output');oracle.require(process.returncode==0,'fixture-tool-exit');return raw.decode()
try:
compiler=Path(shutil.which(args.cc)).resolve();rustc=Path(shutil.which(args.rustc)).resolve();version=run([str(rustc),'-vV'])
oracle.require('release: 1.99.0' in version and 'commit-hash: b940084d7' in version,'fixture-toolchain')
run([str(compiler),'--version']);helper=directory/'session-clock.so';executable=directory/'session-clock-probe'
run([str(compiler),'-shared','-fPIC','-pthread','-O2','-Wall','-Wextra','-Werror','-o',str(helper),str(HERE/'session_clock.c'),'-ldl'])
run([str(rustc),'--edition=2021','-C','opt-level=1','-o',str(executable),str(HERE/'session_clock_probe.rs')])
dynamic=run(['readelf','--dyn-syms','--wide',str(executable)]);oracle.require('clock_gettime@' in dynamic,'fixture-dynamic-clock-import')
interpreter=run(['readelf','-lW',str(executable)]).split('Requesting program interpreter: ',1)[1].split(']',1)[0]
report['frozen_images']=freeze_images(executable,helper)
report['hashes']={str(path):digest(path) for path in (compiler,rustc,helper,executable,Path(interpreter),HERE/'session_clock.py',HERE/'session_clock.c',HERE/'session_clock_probe.rs')}
oracle.require(all(report['hashes'][str(path)]==report['frozen_images'][name+'_sha256'] for name,path in (('executable',executable),('helper',helper))),'fixture-image-hash')
oracle.require(sum(path.stat().st_size for path in directory.rglob('*') if path.is_file())<=512*1024**2,'fixture-storage-bound')
native_admission(executable,helper,output,deadline,report)
report['build_id_notes']={str(path):run(['readelf','-nW',str(path)]) for path in (helper,executable,Path(interpreter),Path(report['target']['libc']))};report.update(state='completed',exit=0)
except BaseException as error:report['error']=type(error).__name__+': '+str(error);raise
finally:
try:
if report['state']=='completed':shutil.rmtree(directory)
elif any(name in report for name in ('tool_cleanup_error','native_cleanup_error')):report['failure_evidence']=str(directory)
else:
retained=output/'failed-fixture';shutil.move(str(directory),str(retained));report['failure_evidence']=str(retained)
except BaseException as error:
report['cleanup_error']=type(error).__name__+': '+str(error);report['failure_evidence']=str(directory)
if 'error' not in report:report.update(state='failed',exit=1);raise
finally:
report['cleanup']={'owned_fixture':str(directory),'absent':not directory.exists(),'failure_evidence':report.get('failure_evidence')}
(output/'receipt.json').write_text(json.dumps(report,indent=2)+'\n');print(str(output/'receipt.json'))
return report['exit']
if __name__=='__main__':raise SystemExit(main())SHA256 35bbbc144065a22bd37390aed7325ded8f56b3bdfbee672944fc19292cfba744.
"""Named rejection must survive provenance and environment checks."""
import ast
import importlib.util
import os
from pathlib import Path
import unittest
import subprocess
import tempfile
from types import SimpleNamespace
from unittest.mock import call, patch
SPEC = importlib.util.spec_from_file_location('install_qemu', Path(__file__).with_name('run-qemu.py'))
MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(MODULE)
class MutationGradingTests(unittest.TestCase):
def test_expected_rpm_mutation_cleans_only_confirmed_stopped_scratch(self):
main = next(node for node in ast.parse(Path(MODULE.__file__).read_text()).body
if isinstance(node, ast.FunctionDef) and node.name == 'main')
final = next(node.finalbody for node in main.body if isinstance(node, ast.Try))
index = next(i for i, node in enumerate(final) if isinstance(node, ast.Assign)
and any(isinstance(t, ast.Name) and t.id == 'completed_mutation' for t in node.targets))
code = compile(ast.Module(body=final[index:index + 2], type_ignores=[]), '<actual-cleanup>', 'exec')
for output, stopped, error, expected in (
('ACCEPT A6 FAIL injected', True, None, True),
('ACCEPT A12 FAIL unrelated', True, None, False),
('ACCEPT A6 FAIL injected', False, None, False),
('ACCEPT A6 FAIL injected', True, 'helper hash mismatch', False)):
with self.subTest(output=output, stopped=stopped, error=error), \
tempfile.TemporaryDirectory(dir=os.environ['FABRIC_SCRATCH_ROOT']) as root:
namespace = dict(args=SimpleNamespace(mutate=None, expected_mutation='root-user'),
grading_mutation='root-user', mutation_package_archive=None,
mutation_rejected=MODULE.mutation_rejected, rc=1, acceptance_log=output,
error=error, cleanup_ok=stopped, process_cleanup_confirmed=stopped,
work_dir=Path(root), result_dir=Path(root), shutil=MODULE.shutil, json=MODULE.json,
guest_acceptance_exit=1, mutation_package_sha256=None)
with patch.object(MODULE.shutil, 'rmtree') as remove:
exec(code, namespace)
self.assertEqual(remove.call_args_list, [call(Path(root))] if expected else [])
def test_reboot_witness_runs_after_privilege_transition(self):
# Origin: Fedora upgrade03 really rebooted, but SSH flattened bash -c
# argv, expanding protected paths too early and leaving a bare test.
def fake_ssh(args, *, input_text=None, **kwargs):
shell = '''sudo() { privileged=yes; "$@"; }
cat() {
[ "$privileged" = yes ] || return 1
case "$1" in
/proc/sys/kernel/random/boot_id) echo current ;;
/root/accept/lifecycle-boot-id) echo "$saved" ;;
*) return 1 ;;
esac
}
export -f cat
export privileged saved
'''
return subprocess.run(['bash', '-c', shell + ' '.join(args[1:])],
input=input_text, text=True, capture_output=True,
env={'PATH': '/usr/bin:/bin', 'saved': saved})
with patch.object(MODULE, 'run', fake_ssh):
saved = 'before'
self.assertEqual(MODULE.changed_boot(['fake-ssh']).returncode, 0)
saved = 'current'
self.assertNotEqual(MODULE.changed_boot(['fake-ssh']).returncode, 0)
saved = 'before'
old = fake_ssh(['fake-ssh', 'sudo', 'bash', '-c',
'test "$(cat /proc/sys/kernel/random/boot_id)" != "$(cat /root/accept/lifecycle-boot-id)"'])
self.assertNotEqual(old.returncode, 0)
def test_all_registered_named_defects_are_required(self):
for mutation, checks in MODULE.MUTATION_EXPECTED_FAILURES.items():
output = '\n'.join(f'ACCEPT {check} FAIL injected' for check in sorted(checks))
self.assertTrue(MODULE.mutation_rejected(mutation, 1, output))
for check in checks:
with self.subTest(mutation=mutation, missing=check):
self.assertFalse(MODULE.mutation_rejected(
mutation, 1, output.replace(f'ACCEPT {check} FAIL', f'ACCEPT {check} PASS')))
def test_unrelated_failure_is_not_detection(self):
self.assertFalse(MODULE.mutation_rejected('root-user', 1, 'ACCEPT A12 FAIL timeout'))
def test_bad_provenance_or_environment_cannot_pass(self):
output = 'ACCEPT A6 FAIL injected'
self.assertFalse(MODULE.mutation_rejected('root-user', 1, output, 'helper hash mismatch'))
self.assertFalse(MODULE.mutation_rejected('root-user', 1, output + '\nACCEPT A8h NOT-RUN unavailable'))
for code in (None, 0, 2, 3, -9):
self.assertFalse(MODULE.mutation_rejected('root-user', code, output))
class InstallationShellControls(unittest.TestCase):
"""Origin: C-035 / C-aud-7; real shell decisions, isolated paths and command stubs."""
def shell(self, program, family='fedora', *, unit='', state='inactive', avc='',
retained=True, purge_keeps_state=False, systemctl_mode='normal', grep_error=False):
with tempfile.TemporaryDirectory(dir=os.environ['FABRIC_SCRATCH_ROOT']) as directory:
root = Path(directory)
paths = {old: root / name for old, name in (
('/root/accept', 'accept'), ('/var/lib/fabrico11y', 'state'),
('/etc/fabrico11y', 'config'), ('/usr/bin/fabric-server', 'absent-binary'))}
work, data, config = (paths[p] for p in ('/root/accept', '/var/lib/fabrico11y', '/etc/fabrico11y'))
work.mkdir(); config.mkdir()
if retained: (data / 'server').mkdir(parents=True)
account = 'fabricolly:x:995:995:Fabric O11y service::/usr/sbin/nologin\n'
(work / 'lifecycle-account').write_text(account)
(config / 'server.conf').write_text('synthetic configuration\n')
for old, new in paths.items(): program = program.replace(old, str(new))
stubs = '''
rpm() { [ "$*" = '-e fabrico11y' ]; }
dpkg() {
if [ "$1" = -P ]; then
if [ "$PURGE_KEEP_STATE" != yes ]; then rm -rf "$DATA"; fi
rm -rf "$CONFIG"
fi
}
rm() {
[ "$1" = -rf ] || return 97; shift
for path; do
case "$path" in "$DATA"|"$CONFIG") command rm -rf -- "$path" ;; *) return 97 ;; esac
done
}
package_remove() { return 0; }
getent() { printf '%s' "$ACCOUNT"; }
getenforce() { echo Enforcing; }
auditctl() { echo 'enabled 1'; }
ausearch() { printf '%s\\n' "$AVC"; }
grep() {
if [ "$GREP_ERROR" = yes ] && [ "$1" = -E ]; then
echo 'synthetic audit read error' >&2; return 2
fi
command grep "$@"
}
systemctl() {
case "$SYSTEMCTL_MODE" in
empty) return 3 ;;
incomplete) echo inactive; return 3 ;;
error) printf 'inactive\\ninactive\\n'; echo 'synthetic systemctl query error' >&2; return 1 ;;
esac
local rc=3 value unit
for unit; do
case "$unit" in fabrico11y-*.service)
value=inactive; if [ "$unit" = "$BAD_UNIT" ]; then value=$BAD_STATE; fi
case " $* " in *' --quiet '*) ;; *) printf '%s\\n' "$value" ;; esac
if [ "$value" = active ]; then rc=0; fi
esac
done
return "$rc"
}
'''
env = dict(os.environ, ACCOUNT=account, DATA=str(data), CONFIG=str(config),
BAD_UNIT=unit, BAD_STATE=state, AVC=avc,
PURGE_KEEP_STATE='yes' if purge_keeps_state else 'no',
SYSTEMCTL_MODE=systemctl_mode, GREP_ERROR='yes' if grep_error else 'no')
return subprocess.run(['bash', '-s', '--', 'remove', family, 'synthetic-package'],
input=stubs + program, env=env, text=True, capture_output=True, timeout=10)
def test_lifecycle_remove_rejects_active_services_and_fabric_avc(self):
program = Path(__file__).with_name('lifecycle.sh').read_text()
for unit in ('fabrico11y-server.service', 'fabrico11y-node.service'):
for state in ('active', 'activating', 'deactivating', 'reloading'):
with self.subTest(unit=unit, state=state):
result = self.shell(program, unit=unit, state=state)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn('ACCEPT A13a PASS', result.stdout)
for comm in ('fabric-server', 'fabric-node', 'fabricctl'):
with self.subTest(comm=comm):
result = self.shell(program, avc=f'type=AVC msg=audit(1.0:1): avc: denied {{ read }} comm="{comm}"')
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn('ACCEPT F4 PASS', result.stdout)
for family in ('debian', 'fedora'):
with self.subTest(clean=family):
result = self.shell(program, family)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertIn('ACCEPT A13a PASS', result.stdout)
self.assertIn('ACCEPT A13b PASS', result.stdout)
def test_lifecycle_remove_rejects_missing_state_and_incomplete_purge(self):
program = Path(__file__).with_name('lifecycle.sh').read_text()
for kwargs, forbidden in (({'retained': False}, 'ACCEPT A13a PASS'),
({'purge_keeps_state': True}, 'ACCEPT A13b PASS')):
with self.subTest(kwargs=kwargs):
result = self.shell(program, 'debian', **kwargs)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn(forbidden, result.stdout)
def test_lifecycle_remove_rejects_unobserved_services_and_unreadable_audit(self):
program = Path(__file__).with_name('lifecycle.sh').read_text()
for mode in ('empty', 'incomplete', 'error'):
with self.subTest(systemctl_mode=mode):
result = self.shell(program, systemctl_mode=mode)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn('ACCEPT A13a PASS', result.stdout)
result = self.shell(program, grep_error=True)
self.assertIn('synthetic audit read error', result.stderr)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn('ACCEPT F4 PASS', result.stdout)
def test_acceptance_remove_rejects_either_live_unit(self):
source = Path(__file__).with_name('acceptance.sh').read_text()
functions = '\n'.join(line for line in source.splitlines() if line.startswith(('pass()', 'fail()')))
start = source.index('package_remove >')
block = source[start:source.index('package_purge >', start)]
for decision in ('stopped=$(systemctl is-active ', 'pass A13a ', 'fail A13a '):
self.assertIn(decision, block, 'actual nonempty A13a decision required')
program = 'set -u\nFAILS=0\nW=/root/accept\n' + functions + '\n' + block + '\nexit "$FAILS"\n'
for unit in ('fabrico11y-server.service', 'fabrico11y-node.service'):
for state in ('active', 'activating', 'deactivating', 'reloading', 'inactive'):
with self.subTest(unit=unit, state=state):
result = self.shell(program, unit=unit, state=state)
self.assertEqual(result.returncode, 0 if state == 'inactive' else 1,
result.stdout + result.stderr)
self.assertIn('ACCEPT A13a ' + ('PASS' if state == 'inactive' else 'FAIL'), result.stdout)
for mode in ('empty', 'incomplete', 'error'):
with self.subTest(systemctl_mode=mode):
result = self.shell(program, systemctl_mode=mode)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn('ACCEPT A13a PASS', result.stdout)
def test_acceptance_fedora_rejects_unreadable_audit(self):
source = Path(__file__).with_name('acceptance.sh').read_text()
functions = '\n'.join(line for line in source.splitlines() if line.startswith(('pass()', 'fail()')))
start = source.index(' audit_status=$(auditctl -s 2>&1)')
block = source[start:source.index('\nfi\necho "RESULT', start)]
for decision in ('ausearch -m AVC,USER_AVC ', 'selinux-avc.log', 'pass F3 ', 'fail F3 '):
self.assertIn(decision, block, 'actual nonempty F3 audit decision required')
program = 'set -u\nFAILS=0\nW=/root/accept\n' + functions + '\n' + block + '\nexit "$FAILS"\n'
for grep_error in (False, True):
with self.subTest(grep_error=grep_error):
result = self.shell(program, grep_error=grep_error)
if grep_error:
self.assertIn('synthetic audit read error', result.stderr)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertNotIn('ACCEPT F3 PASS', result.stdout)
else:
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertIn('ACCEPT F3 PASS', result.stdout)
if __name__ == '__main__':
unittest.main()SHA256 c287ad707ea97ea5e25033a6d78d425b4a324016b69144458255b07d95b30b73.
#!/bin/bash
# Running-installation acceptance, executed as root INSIDE a disposable
# Debian 13 systemd container or VM (see run.sh and run-qemu.py). Never run it
# on a real host: it
# creates and deletes the fabricolly account, installs and purges the package
# and starts services. Protocol:
# docs/experiments/formal/installation-acceptance-protocol.md.
#
# Usage: acceptance.sh <DEB> [--self-spindle-ca <CA-PATH>]
# [--memory-stressor tail|parallel]
# Prints one line per check, "ACCEPT <ID> PASS|FAIL|NOT-RUN <detail>". Exits 1
# if any check failed, 3 if none failed but one could not run in this
# environment, and 0 only when every check ran and passed.
set -u
DEB=${1:?usage: acceptance.sh <DEB> [--self-spindle-ca <CA-PATH>] [--memory-stressor tail|parallel]}
shift
SELF_SPINDLE_CA=
A12_MEMORY_STRESSOR=tail
PACKAGE_FAMILY=debian
DEFER_REMOVAL=0
LEGACY_SERVER=0
while [ "$#" -gt 0 ]; do
case "$1" in
--legacy-server) LEGACY_SERVER=1; shift ;;
--defer-removal) DEFER_REMOVAL=1; shift ;;
--package-family)
[ "$#" -ge 2 ] || exit 2
PACKAGE_FAMILY=$2; shift 2
case "$PACKAGE_FAMILY" in debian|fedora) ;; *) exit 2 ;; esac
;;
--self-spindle-ca)
[ "$#" -ge 2 ] && [ -n "$2" ] || { echo "missing self-spindle CA path" >&2; exit 2; }
SELF_SPINDLE_CA=$2; shift 2
case "$SELF_SPINDLE_CA" in
/*) ;;
*) echo "self-spindle CA path must be absolute" >&2; exit 2 ;;
esac
case "$SELF_SPINDLE_CA" in
*[!A-Za-z0-9_./-]*) echo "self-spindle CA path contains unsupported characters" >&2; exit 2 ;;
esac
;;
--memory-stressor)
[ "$#" -ge 2 ] || { echo "missing A12 memory stressor" >&2; exit 2; }
A12_MEMORY_STRESSOR=$2; shift 2
case "$A12_MEMORY_STRESSOR" in
tail|parallel) ;;
*) echo "unsupported A12 memory stressor: $A12_MEMORY_STRESSOR" >&2; exit 2 ;;
esac
;;
*) echo "usage: acceptance.sh <DEB> [--self-spindle-ca <CA-PATH>] [--memory-stressor tail|parallel]" >&2; exit 2 ;;
esac
done
FAILS=0
NOTRUN=0
UNIFIED=0; [ "$(stat -fc %T /sys/fs/cgroup)" = cgroup2fs ] && UNIFIED=1
W=/root/accept
mkdir -p "$W"
LOGDIR=/var/log/fabric-accept
SCRIPT_DIR=$(cd -- "$(dirname -- "$0")" && pwd)
pass() { echo "ACCEPT $1 PASS ${*:2}"; }
fail() { echo "ACCEPT $1 FAIL ${*:2}"; FAILS=$((FAILS + 1)); }
notrun() { echo "ACCEPT $1 NOT-RUN ${*:2}"; NOTRUN=$((NOTRUN + 1)); }
wait_for() { # wait_for <seconds> <command...>
local limit=$1; shift
local end=$((SECONDS + limit))
until "$@" >/dev/null 2>&1; do
[ $SECONDS -ge $end ] && return 1
sleep 1
done
}
admin() { fabricctl admin "$W/admin.conf" "$@"; }
query() { admin query "$1"; }
show() { systemctl show -P "$2" "$1"; }
package_install() {
if [ "$PACKAGE_FAMILY" = fedora ]; then rpm -Uvh --replacepkgs "$DEB"; else dpkg -i "$DEB"; fi
}
package_remove() {
if [ "$PACKAGE_FAMILY" = fedora ]; then rpm -e fabrico11y; else dpkg -r fabrico11y; fi
}
package_purge() {
if [ "$PACKAGE_FAMILY" = fedora ]; then
rpm -e fabrico11y >/dev/null 2>&1 || :
rm -rf /var/lib/fabrico11y /etc/fabrico11y
else dpkg -P fabrico11y; fi
}
package_extract() {
if [ "$PACKAGE_FAMILY" = fedora ]; then
mkdir -p "$x"
(cd "$x" && rpm2cpio "$DEB" | cpio -idm --quiet)
else dpkg-deb -x "$DEB" "$x"; fi
}
echo "ENV family=$PACKAGE_FAMILY systemd $(systemctl --version | head -1 | cut -d' ' -f2) kernel $(uname -r)"
cat /etc/os-release
if [ "$PACKAGE_FAMILY" = fedora ]; then
[ "$(getenforce)" = Enforcing ] && pass F1 "SELinux enforcing" || fail F1 "SELinux must remain enforcing"
fi
echo "ENV cgroup $(stat -fc %T /sys/fs/cgroup) controllers: $(cat /sys/fs/cgroup/cgroup.controllers 2>/dev/null)"
# A1 sysusers: dry run and a temporary root, applied twice.
x=$W/pkg; rm -rf "$x"; package_extract
conf=$x/usr/lib/sysusers.d/fabrico11y.conf
r=$W/sysroot; rm -rf "$r"; mkdir -p "$r/etc"
dry=$(systemd-sysusers --dry-run --root="$r" "$conf" 2>&1); dry_rc=$?
dry_clean=0; [ -e "$r/etc/passwd" ] && dry_clean=1
systemd-sysusers --root="$r" "$conf" >/dev/null 2>&1 && first=$(grep '^fabricolly:' "$r/etc/passwd")
systemd-sysusers --root="$r" "$conf" >/dev/null 2>&1 && second=$(grep '^fabricolly:' "$r/etc/passwd")
if [ $dry_rc -eq 0 ] && echo "$dry" | grep -q fabricolly && [ $dry_clean -eq 0 ] \
&& [ -n "${first:-}" ] && [ "$first" = "${second:-}" ] \
&& echo "$first" | grep -Eq '^fabricolly:x:[0-9]+:[0-9]+:Fabric O11y service:/:/usr/sbin/nologin$'; then
pass A1 "temporary root: $first"
else
fail A1 "dry_rc=$dry_rc first='${first:-}' second='${second:-}'"
fi
# A2 collision refusal: a human account, then a non-system group, named fabricolly.
useradd -m -u 1500 -s /bin/bash fabricolly
out=$(package_install 2>&1); rc=$?
if [ $rc -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
pass A2a "human account uid 1500 refused, nothing unpacked"
else
fail A2a "rc=$rc"
fi
package_purge >/dev/null 2>&1; userdel -r fabricolly 2>/dev/null
groupadd -g 1600 fabricolly
out=$(package_install 2>&1); rc=$?
if [ $rc -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
pass A2b "non-system group gid 1600 refused, nothing unpacked"
else
fail A2b "rc=$rc"
fi
package_purge >/dev/null 2>&1; groupdel fabricolly 2>/dev/null
# Candidate-only root-ID collision checks. The historical predecessor's old
# preinstall is preserved in migration fixtures and is not credited with these.
if [ "$LEGACY_SERVER" = 1 ]; then
root_before=$(getent passwd root); root_group_before=$(getent group root)
groupadd --system fabricolly
useradd -M -o -u 0 -g fabricolly -s /usr/sbin/nologin fabricolly
out=$(package_install 2>&1); rc=$?
if [ "$rc" -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
pass A2c "duplicate root UID refused before unpack"
else fail A2c "root UID rc=$rc"; fi
package_purge >/dev/null 2>&1
userdel -f fabricolly >/dev/null 2>&1
groupdel fabricolly >/dev/null 2>&1
groupadd -o -g 0 fabricolly
out=$(package_install 2>&1); rc=$?
if [ "$rc" -ne 0 ] && echo "$out" | grep -q "refusing to install" && [ ! -e /usr/bin/fabric-server ]; then
pass A2d "duplicate root GID refused before unpack"
else fail A2d "root GID rc=$rc"; fi
package_purge >/dev/null 2>&1
groupdel -f fabricolly >/dev/null 2>&1
[ "$(getent passwd root)" = "$root_before" ] && [ "$(getent group root)" = "$root_group_before" ] || fail SETUP "duplicate-ID fixture altered root identity"
fi
# A3 install, then install again: same identity, exit 0 both times.
package_install >"$W/install1.log" 2>&1; rc1=$?
id1=$(getent passwd fabricolly)
package_install >"$W/install2.log" 2>&1; rc2=$?
id2=$(getent passwd fabricolly)
uid=$(id -u fabricolly 2>/dev/null); gid=$(id -g fabricolly 2>/dev/null); pg=$(id -gn fabricolly 2>/dev/null)
if [ $rc1 -eq 0 ] && [ $rc2 -eq 0 ] && [ "$id1" = "$id2" ] && [ "$uid" -lt 1000 ] && [ "$uid" -gt 0 ] \
&& [ "$pg" = fabricolly ] && [ "$(getent passwd fabricolly | cut -d: -f7)" = /usr/sbin/nologin ]; then
pass A3 "installed twice; $id1"
else
fail A3 "rc1=$rc1 rc2=$rc2 id1='$id1' id2='$id2'"
fi
etc_mode=$(stat -c '%a %U:%G' /etc/fabrico11y)
[ "$etc_mode" = "750 root:fabricolly" ] && pass A3b "/etc/fabrico11y $etc_mode" || fail A3b "/etc/fabrico11y $etc_mode"
# A4 systemd-analyze verify on the installed units and slice.
out=$(systemd-analyze verify /usr/lib/systemd/system/fabrico11y-node.service \
/usr/lib/systemd/system/fabrico11y-server.service /usr/lib/systemd/system/system-fabrico11y.slice 2>&1); rc=$?
[ $rc -eq 0 ] && pass A4 "exit 0" || fail A4 "rc=$rc $out"
# Configuration: a private CA, the server certificate for 127.0.0.1, the admin
# token, and one authorized and one denied log.
cd "$W"
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -keyout ca.key -out ca.pem -days 2 \
-subj "/CN=fabric accept CA" -addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign" 2>/dev/null
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -keyout server.key -out server.csr \
-subj "/CN=127.0.0.1" 2>/dev/null
printf 'subjectAltName=IP:127.0.0.1\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=serverAuth\n' > san.ext
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.pem -days 2 -extfile san.ext 2>/dev/null
install -m 0644 -o root -g root ca.pem server.pem /etc/fabrico11y/
install -m 0640 -o root -g fabricolly server.key /etc/fabrico11y/
(umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > admin-token)
install -m 0640 -o root -g fabricolly admin-token /etc/fabrico11y/admin-token
cat > /etc/fabrico11y/server.conf <<EOF
listen=127.0.0.1:7443
tls_cert=/etc/fabrico11y/server.pem
tls_key=/etc/fabrico11y/server.key
admin_token_file=/etc/fabrico11y/admin-token
state_dir=/var/lib/fabrico11y/server
journal_bytes=21474836480
retention_s=86400
retention_bytes=21474836480
EOF
if [ -n "$SELF_SPINDLE_CA" ]; then
printf 'self_spindle_ca=%s\n' "$SELF_SPINDLE_CA" >> /etc/fabrico11y/server.conf
fi
chmod 0644 /etc/fabrico11y/server.conf
if [ "$PACKAGE_FAMILY" = fedora ]; then
restorecon -R /etc/fabrico11y
fi
printf 'server_url=https://127.0.0.1:7443\nserver_ca=/etc/fabrico11y/ca.pem\nadmin_token_file=%s/admin-token\n' "$W" > admin.conf
mkdir -p $LOGDIR && chmod 0755 $LOGDIR
install -m 0600 /dev/null $LOGDIR/allowed.log
install -m 0600 /dev/null $LOGDIR/denied.log
setfacl -m u:fabricolly:r $LOGDIR/allowed.log
if [ "$LEGACY_SERVER" = 1 ]; then
mkdir -p /etc/systemd/system/fabrico11y-server.service.d
printf '[Service]\nExecStart=\nExecStart=/usr/bin/fabric-server serve-legacy /etc/fabrico11y/server.conf\n' > /etc/systemd/system/fabrico11y-server.service.d/legacy-fixture.conf
systemctl daemon-reload
fi
systemctl enable --now fabrico11y-server.service >/dev/null 2>&1
wait_for 30 curl -fsS --cacert /etc/fabrico11y/ca.pem -H "authorization: Bearer $(cat admin-token)" \
https://127.0.0.1:7443/v1/admin/nodes || fail SETUP "server did not answer"
enroll=$(admin node add spindle-1 --log $LOGDIR/allowed.log --log $LOGDIR/denied.log --interval 15)
token=$(echo "$enroll" | grep -o '"token":"[^"]*"' | cut -d'"' -f4)
(umask 077; printf '%s\n' "$token" > node-token)
install -m 0640 -o root -g fabricolly node-token /etc/fabrico11y/node-token
cat > /etc/fabrico11y/node.conf <<EOF
spool_dir=/var/lib/fabrico11y/node/spool
metric_interval_s=15
spool_bytes=268435456
log=$LOGDIR/allowed.log
log=$LOGDIR/denied.log
server_url=https://127.0.0.1:7443
server_ca=/etc/fabrico11y/ca.pem
token_file=/etc/fabrico11y/node-token
EOF
chmod 0644 /etc/fabrico11y/node.conf
echo "accept-line-before-start" >> $LOGDIR/allowed.log
echo "secret-line-never-read" >> $LOGDIR/denied.log
systemctl enable --now fabrico11y-node.service >/dev/null 2>&1
# A5 TLS: the admin API refuses an unverified client and answers a verified one;
# the node delivers over TLS (rows below).
curl -sS -o /dev/null https://127.0.0.1:7443/v1/admin/nodes 2>/dev/null; untrusted=$?
code=$(curl -sS -o /dev/null -w '%{http_code}' --cacert /etc/fabrico11y/ca.pem \
-H "authorization: Bearer $(cat admin-token)" https://127.0.0.1:7443/v1/admin/nodes)
[ $untrusted -eq 60 ] && [ "$code" = 200 ] && pass A5 "untrusted client exit 60; CA-verified 200" \
|| fail A5 "untrusted=$untrusted verified=$code"
logs_query() { query "{\"kind\":\"logs\",\"node\":\"spindle-1\",\"from_ns\":0,\"to_ns\":9223372036854775807,\"limit\":1000}"; }
has_line() { logs_query | grep -q "$1"; }
# A6 identity of the running services: non-root UID and primary GID, no capabilities.
wait_for 60 has_line accept-line-before-start
ok=1; detail=""
for unit in fabrico11y-node fabrico11y-server; do
pid=$(show $unit.service MainPID)
u=$(awk '/^Uid:/{print $2}' /proc/$pid/status); g=$(awk '/^Gid:/{print $2}' /proc/$pid/status)
cap=$(awk '/^CapEff:/{print $2}' /proc/$pid/status); nnp=$(awk '/^NoNewPrivs:/{print $2}' /proc/$pid/status)
detail="$detail $unit:uid=$u,gid=$g,CapEff=$cap,NoNewPrivs=$nnp"
[ "$u" = "$uid" ] && [ "$g" = "$gid" ] && [ "$u" != 0 ] && [ "$cap" = 0000000000000000 ] && [ "$nnp" = 1 ] || ok=0
done
[ $ok = 1 ] && pass A6 "$detail" || fail A6 "$detail"
# A7 managed directories: owner fabricolly:fabricolly, mode 0700.
ok=1; detail=""
for d in /var/lib/fabrico11y/node /var/lib/fabrico11y/server /run/fabrico11y/node /run/fabrico11y/server; do
m=$(stat -c '%a %U:%G' "$d" 2>/dev/null || echo missing)
detail="$detail $d=$m"
[ "$m" = "700 fabricolly:fabricolly" ] || ok=0
done
[ $ok = 1 ] && pass A7 "$detail" || fail A7 "$detail"
# A8 cgroup placement, effective limits and accounting. On the unified
# hierarchy the files are memory.max/memory.high/pids.max; on the legacy
# hierarchy memory.limit_in_bytes and pids.max, and MemoryHigh has no kernel
# counterpart, so its enforcement (A8h) is not run there.
ok=1; detail=""
cgroup_of() { # pid -> the service's cgroup path in the systemd tree
if [ "$UNIFIED" = 1 ]; then sed -n 's/^0:://p' /proc/$1/cgroup; else sed -n 's/^[0-9]*:name=systemd://p' /proc/$1/cgroup; fi
}
mem_max_file() { [ "$UNIFIED" = 1 ] && echo "/sys/fs/cgroup$1/memory.max" || echo "/sys/fs/cgroup/memory$1/memory.limit_in_bytes"; }
pids_max_file() { [ "$UNIFIED" = 1 ] && echo "/sys/fs/cgroup$1/pids.max" || echo "/sys/fs/cgroup/pids$1/pids.max"; }
expect() { # unit high max tasks
local unit=$1 pid cg
pid=$(show "$unit" MainPID); cg=$(cgroup_of "$pid")
detail="$detail $unit:cgroup=$cg"
[ "$cg" = "/system.slice/system-fabrico11y.slice/$unit" ] || ok=0
for kv in "EffectiveMemoryHigh=$2" "EffectiveMemoryMax=$3" "EffectiveTasksMax=$4"; do
local got; got=$(show "$unit" "${kv%%=*}")
detail="$detail ${kv%%=*}=$got"
[ "$got" = "${kv#*=}" ] || ok=0
done
for acct in MemoryCurrent TasksCurrent; do
local got; got=$(show "$unit" $acct)
detail="$detail $acct=$got"
case "$got" in ''|'[not set]'|18446744073709551615) ok=0 ;; esac
done
local got_max got_pids
got_max=$(cat "$(mem_max_file "$cg")" 2>/dev/null); got_pids=$(cat "$(pids_max_file "$cg")" 2>/dev/null)
detail="$detail kernel-memory-max=$got_max kernel-pids-max=$got_pids"
[ "$got_max" = "$3" ] && [ "$got_pids" = "$4" ] || ok=0
}
expect fabrico11y-node.service $((128 << 20)) $((256 << 20)) 128
expect fabrico11y-server.service $((2560 << 20)) $((3072 << 20)) 512
slice=/system.slice/system-fabrico11y.slice
s_max=$(cat "$(mem_max_file $slice)" 2>/dev/null); s_pids=$(cat "$(pids_max_file $slice)" 2>/dev/null)
detail="$detail slice kernel-memory-max=$s_max kernel-pids-max=$s_pids"
[ "$s_max" = $((3328 << 20)) ] && [ "$s_pids" = 640 ] || ok=0
[ $ok = 1 ] && pass A8 "$detail" || fail A8 "$detail"
if [ "$UNIFIED" = 1 ]; then
n_high=$(cat "/sys/fs/cgroup$(cgroup_of "$(show fabrico11y-node.service MainPID)")/memory.high")
s_high=$(cat /sys/fs/cgroup$slice/memory.high)
[ "$n_high" = $((128 << 20)) ] && [ "$s_high" = $((2816 << 20)) ] \
&& pass A8h "node memory.high=$n_high slice memory.high=$s_high" || fail A8h "node=$n_high slice=$s_high"
echo "REPORT controllers slice=[$(cat /sys/fs/cgroup$slice/cgroup.controllers)]"
else
notrun A8h "legacy cgroup hierarchy: MemoryHigh has no kernel counterpart (systemd reports the value only)"
fi
echo "REPORT IOReadBytes node=$(show fabrico11y-node.service IOReadBytes) (I/O accounting and weights are reported, never counted as enforcement)"
# A9 host metrics and the authorized log, read inside the sandbox.
metric_query='{"kind":"metrics","node":"spindle-1","name":"system.memory.available","from_ns":0,"to_ns":9223372036854775807,"limit":10}'
wait_for 60 sh -c "fabricctl admin $W/admin.conf query '$metric_query' | grep -q '\"time_ns\"'"
m=$(query "$metric_query")
if echo "$m" | grep -q '"time_ns"' && has_line accept-line-before-start; then
pass A9 "system.memory.available points and the authorized log line delivered"
else
fail A9 "metrics: $(echo "$m" | head -c 300)"
fi
# A10 the denied log is a visible gap and its content is never delivered.
wait_for 60 sh -c "fabricctl admin $W/admin.conf query '{\"kind\":\"logs\",\"node\":\"spindle-1\",\"from_ns\":0,\"to_ns\":9223372036854775807,\"limit\":1000}' | grep -q 'denied.log'"
answer=$(logs_query)
gap=$(echo "$answer" | grep -o 'log source unavailable [^"]*denied.log[^"]*' | head -1)
if [ -n "$gap" ] && ! echo "$answer" | grep -q secret-line-never-read; then
pass A10 "gap: $gap"
else
fail A10 "no denied-log gap or secret delivered"
fi
# A11 graceful shutdown and restart with a retained Spool.
systemctl stop fabrico11y-server.service
server_stop=$(show fabrico11y-server.service Result)
echo "accept-line-during-outage" >> $LOGDIR/allowed.log
sleep 20
systemctl restart fabrico11y-node.service
node_stop=$(show fabrico11y-node.service Result)
spool_files=$(find /var/lib/fabrico11y/node/spool -type f | wc -l)
systemctl start fabrico11y-server.service
wait_for 90 has_line accept-line-during-outage; delivered=$?
has_line accept-line-before-start; kept=$?
if [ "$server_stop" = success ] && [ "$node_stop" = success ] && [ "$spool_files" -gt 0 ] \
&& [ $delivered -eq 0 ] && [ $kept -eq 0 ]; then
pass A11 "server stop Result=$server_stop; node restart Result=$node_stop; spool files=$spool_files; outage line delivered after restart; earlier line retained"
else
fail A11 "server=$server_stop node=$node_stop spool=$spool_files delivered=$delivered kept=$kept"
fi
# A12 memory and task pressure are contained in the slice; the services survive.
if [ "$A12_MEMORY_STRESSOR" = parallel ] && [ "$UNIFIED" -eq 0 ]; then
echo "FIXTURE A12 memory-stressor=parallel helper_sha256=unavailable reason=requires-cgroup-v2"
notrun A12 "parallel memory stressor requires unified cgroup v2; use the historical tail default on legacy hierarchies"
else
n_before=$(show fabrico11y-node.service NRestarts); s_before=$(show fabrico11y-server.service NRestarts)
# hog <unit> <limit bytes> <worker bytes> <require kernel OOM> <properties...>:
# runs the selected fixture and samples MemoryCurrent until it ends. Contained means
# the peak stayed within the limit and the kernel killed it: Result=oom-kill
# on the unified hierarchy; on the legacy one systemd cannot see OOM events,
# so a SIGKILL (Result=signal, status 9) before RuntimeMaxSec is required.
hog() {
local unit=$1 limit=$2 worker_bytes=$3 require_kernel_oom=$4 expected_high=$5 expected_tasks=$6
shift 6
local before_max=0 before_group=0 after_max=0 after_group=0
if [ "$require_kernel_oom" = 1 ] && [ "$UNIFIED" = 1 ]; then
local slice_events=/sys/fs/cgroup/system.slice/system-fabrico11y.slice/memory.events
while read -r event value; do
case "$event" in max) before_max=$value ;; oom_group_kill) before_group=$value ;; esac
done < "$slice_events"
fi
if [ "$A12_MEMORY_STRESSOR" = parallel ]; then
systemd-run --quiet --unit="$unit" --slice=system-fabrico11y.slice \
-p RuntimeMaxSec=300 -p OOMPolicy=kill -p MemorySwapMax=0 "$@" \
python3 "$SCRIPT_DIR/a12-memory-pressure.py" --parallel-bytes "$worker_bytes" \
--memory-high-bytes "$expected_high" --memory-max-bytes "$limit" --tasks-max "$expected_tasks"
else
systemd-run --quiet --unit="$unit" --slice=system-fabrico11y.slice -p RuntimeMaxSec=300 "$@" tail /dev/zero
fi
local peak=0 cur
while [ "$(systemctl is-active "$unit" 2>/dev/null)" = active ]; do
cur=$(show "$unit" MemoryCurrent); case "$cur" in ''|'[not set]') ;; *) [ "$cur" -gt "$peak" ] && peak=$cur ;; esac
sleep 0.2
done
local result status max_events=0 oom_group_events=0 events_state=not-checked
result=$(show "$unit" Result); status=$(show "$unit" ExecMainStatus)
if [ "$require_kernel_oom" = 1 ] && [ "$UNIFIED" = 1 ]; then
local slice_events=/sys/fs/cgroup/system.slice/system-fabrico11y.slice/memory.events event value
while read -r event value; do
case "$event" in max) after_max=$value ;; oom_group_kill) after_group=$value ;; esac
done < "$slice_events"
max_events=$((after_max - before_max))
oom_group_events=$((after_group - before_group))
events_state="slice-delta max=$max_events oom_group_kill=$oom_group_events"
fi
systemctl reset-failed "$unit" 2>/dev/null
local contained=0
local event_check=1
if [ "$require_kernel_oom" = 1 ] && [ "$UNIFIED" = 1 ] \
&& { [ "$max_events" -le 0 ] || [ "$oom_group_events" -le 0 ]; }; then event_check=0; fi
if [ "$peak" -le "$limit" ] && [ "$event_check" = 1 ] \
&& { [ "$result" = oom-kill ] || { [ "$UNIFIED" = 0 ] && [ "$result" = signal ] && [ "$status" = 9 ]; }; }; then
contained=1
fi
echo "$contained $result/$status peak=$peak limit=$limit events=$events_state"
}
positive_control='not-used'; positive_control_ok=1
if [ "$A12_MEMORY_STRESSOR" = parallel ]; then
helper=$SCRIPT_DIR/a12-memory-pressure.py
helper_sha=$(sha256sum "$helper" | awk '{print $1}')
page_bytes=$(getconf PAGESIZE)
node_worker_bytes=$(((((2 * (256 << 20) + 95) / 96 + page_bytes - 1) / page_bytes) * page_bytes))
slice_worker_bytes=$(((((2 * (3328 << 20) + 95) / 96 + page_bytes - 1) / page_bytes) * page_bytes))
echo "FIXTURE A12 memory-stressor=parallel helper_sha256=$helper_sha workers=96 node_worker_bytes=$node_worker_bytes slice_worker_bytes=$slice_worker_bytes"
if command -v python3 >/dev/null 2>&1 && [ -f "$helper" ]; then
systemd-run --quiet --unit=fabric-accept-mem-control --slice=system-fabrico11y.slice \
-p Type=exec -p RemainAfterExit=yes \
-p RuntimeMaxSec=15 -p MemoryHigh=128M -p MemoryMax=256M -p MemorySwapMax=0 -p TasksMax=128 \
python3 "$helper" --control --memory-high-bytes $((128 << 20)) \
--memory-max-bytes $((256 << 20)) --tasks-max 128
control_peak=0; cur=0; control_end=$((SECONDS + 20))
while [ "$(show fabric-accept-mem-control SubState)" = running ] && [ "$SECONDS" -lt "$control_end" ]; do
cur=$(show fabric-accept-mem-control MemoryCurrent)
case "$cur" in ''|'[not set]') ;; *) [ "$cur" -gt "$control_peak" ] && control_peak=$cur ;; esac
sleep 0.2
done
control_result=$(show fabric-accept-mem-control Result)
control_status=$(show fabric-accept-mem-control ExecMainStatus)
control_state=$(show fabric-accept-mem-control SubState)
systemctl stop fabric-accept-mem-control 2>/dev/null
systemctl reset-failed fabric-accept-mem-control 2>/dev/null
positive_control_ok=0
if [ "$control_peak" -ge $((24 << 20)) ] && [ "$control_peak" -lt $((128 << 20)) ] \
&& [ "$control_state" = exited ] && [ "$control_result" = success ] \
&& [ "$control_status" = 0 ]; then positive_control_ok=1; fi
positive_control="$control_result/$control_status state=$control_state peak=$control_peak high=$((128 << 20))"
else
positive_control_ok=0; positive_control='python3-or-fixture-unavailable'
fi
else
echo "FIXTURE A12 memory-stressor=tail acceptance_sha256=$(sha256sum "$0" | awk '{print $1}')"
fi
if [ "$A12_MEMORY_STRESSOR" = parallel ] && [ "$positive_control_ok" -ne 1 ]; then
echo "A12 parallel positive control failed: $positive_control" >&2
fi
if [ "$A12_MEMORY_STRESSOR" = parallel ]; then
mem=$(hog fabric-accept-mem $((256 << 20)) "$node_worker_bytes" 1 $((128 << 20)) 128 \
-p MemoryHigh=128M -p MemoryMax=256M -p TasksMax=128)
slice_hog=$(hog fabric-accept-slice $((3328 << 20)) "$slice_worker_bytes" 1 $((2816 << 20)) 640)
else
mem=$(hog fabric-accept-mem $((256 << 20)) 0 0 0 0 -p MemoryHigh=128M -p MemoryMax=256M -p TasksMax=128)
slice_hog=$(hog fabric-accept-slice $((3328 << 20)) 0 0 0 0)
fi
# A process tree that tries to exceed the Spindle's task limit.
systemd-run --quiet --unit=fabric-accept-tasks --slice=system-fabrico11y.slice -p TasksMax=128 \
bash -c 'for i in $(seq 300); do sleep 120 & done; wait' >/dev/null 2>&1
tasks_now=0; end=$((SECONDS + 20))
while [ $SECONDS -lt $end ]; do
cur=$(show fabric-accept-tasks.service TasksCurrent 2>/dev/null)
case "$cur" in ''|*[!0-9]*) ;; *) [ "$cur" -gt "$tasks_now" ] && tasks_now=$cur ;; esac
sleep 0.2
done
fork_errors=$(journalctl -q -u fabric-accept-tasks.service --no-pager 2>/dev/null | grep -ci "fork")
systemctl stop fabric-accept-tasks.service 2>/dev/null; systemctl reset-failed fabric-accept-tasks.service 2>/dev/null
n_after=$(show fabrico11y-node.service NRestarts); s_after=$(show fabrico11y-server.service NRestarts)
alive=$(systemctl is-active fabrico11y-node.service fabrico11y-server.service | tr '\n' ' ')
echo "accept-line-after-pressure" >> $LOGDIR/allowed.log
wait_for 60 has_line accept-line-after-pressure; after=$?
if [ "${mem%% *}" = 1 ] && [ "${slice_hog%% *}" = 1 ] && [ "$positive_control_ok" = 1 ] \
&& [ "$tasks_now" -le 128 ] && [ "$tasks_now" -ge 100 ] \
&& [ "$fork_errors" -gt 0 ] && [ "$n_before" = "$n_after" ] && [ "$s_before" = "$s_after" ] \
&& [ "$alive" = "active active " ] && [ $after -eq 0 ]; then
pass A12 "memory-stressor=$A12_MEMORY_STRESSOR; positive control=$positive_control; service-limit hog: ${mem#* }; slice-limit hog: ${slice_hog#* }; task tree peaked at TasksCurrent=$tasks_now with $fork_errors fork-failure lines; services active, NRestarts unchanged, delivery continued"
else
fail A12 "memory-stressor=$A12_MEMORY_STRESSOR positive-control=[$positive_control]; mem=[$mem] slice=[$slice_hog] tasks=$tasks_now forks=$fork_errors restarts node $n_before->$n_after server $s_before->$s_after alive='$alive' after=$after"
fi
fi
if [ "$DEFER_REMOVAL" = 0 ]; then
# A13 remove keeps data, configuration and the account; purge removes data and
# configuration.
package_remove >"$W/remove.log" 2>&1; rc=$?
stopped_rc=0
stopped=$(systemctl is-active fabrico11y-node.service fabrico11y-server.service) || stopped_rc=$?
stopped_ok=0
case "$stopped_rc:$stopped" in
$'3:inactive\ninactive'|$'3:inactive\nfailed'|$'3:failed\ninactive'|$'3:failed\nfailed') stopped_ok=1 ;; esac
if [ $rc -eq 0 ] && [ ! -e /usr/bin/fabric-server ] && [ -d /var/lib/fabrico11y/server ] \
&& [ -f /etc/fabrico11y/server.conf ] && getent passwd fabricolly >/dev/null && [ "$stopped_ok" -eq 1 ]; then
pass A13a "remove: binaries gone, services '$stopped', state, configuration and account kept"
else
fail A13a "rc=$rc stopped_rc=$stopped_rc stopped='$stopped'"
fi
package_purge >"$W/purge.log" 2>&1; rc=$?
if [ $rc -eq 0 ] && [ ! -e /var/lib/fabrico11y ] && [ ! -e /etc/fabrico11y ]; then
if [ "$PACKAGE_FAMILY" = fedora ]; then
pass A13b "explicit guest cleanup after RPM-preserved removal: state/config removed"
else pass A13b "purge: /var/lib/fabrico11y and /etc/fabrico11y removed"; fi
else
fail A13b "rc=$rc"
fi
else
echo "DEFERRED A13 removal to registered upgrade/reboot lifecycle phases"
fi
if [ "$PACKAGE_FAMILY" = fedora ]; then
[ "$(getenforce)" = Enforcing ] && pass F2 "SELinux remained enforcing through delivery/restart/removal" || fail F2 "SELinux changed mode"
audit_status=$(auditctl -s 2>&1); audit_rc=$?
if [ "$audit_rc" -ne 0 ] || ! echo "$audit_status" | grep -Eq '^enabled [12]$'; then
fail F3 "audit unavailable/disabled: $audit_status"
fi
ausearch -m AVC,USER_AVC -ts boot > "$W/selinux-avc.log" 2>&1; audit_search_rc=$?
[ "$audit_search_rc" -le 1 ] || fail F3 "audit search failed: $audit_search_rc"
grep -E 'comm="(fabric-server|fabric-node|fabricctl)"' "$W/selinux-avc.log"; avc_rc=$?
case "$avc_rc" in
0) fail F3 "Fabric process SELinux denial; retain AVC evidence" ;;
1) pass F3 "no Fabric process AVC in guest audit" ;;
*) fail F3 "audit read error rc=$avc_rc; retain audit evidence" ;;
esac
fi
echo "RESULT fails=$FAILS not_run=$NOTRUN"
[ $FAILS -gt 0 ] && exit 1
[ $NOTRUN -gt 0 ] && exit 3
exit 0