Experiment formal simulator helper build preflight 01 - kmosoti/FabricO11y GitHub Wiki
Prepared coordination preflight — separate simulator helper
Prepared pre-flight, 2026-10-15; build NOT RUN. No release/tag publication is proposed.
Frozen product candidate remains 8fb6f413179b094288316e25845c137440ba23fa. Canonical DEB/RPM and all three original helpers were hash-revalidated and are unchanged. Main is 881e90ae5fc6a14c6606fecac82c5373b50a4dd5; PRs 63–67 are merged. The separate helper build uses the frozen source authority below; only examples/spindle_sim.rs differs among the full 463 compilation input paths.
Observed prerequisite evidence:
- Native sealed helper: four real raw Batches/ACKs, native exit 0, registered missing-begin fixture exit 1; expected red preserved.
- 33 source simulator controls: exit 0, no skips.
- Exact 108-version Debian sysroot preparation: exit 0; immutable OCI and fingerprint retained.
- Fresh source freeze and whole-toolchain census: exit 0.
Next command, after attaching this actual board comment URL to the admission JSON and freezing its new SHA-256:
python3 -B DATA/results/candidate-build-fresh-8fb6f413179b-01/run-step-simulator-documentation-01.py simulator-separate-helper-build-01 9663676416 python3 -B DATA/results/candidate-build-fresh-8fb6f413179b-01/build-simulator-helper-proposal-01.py --id native-witness-01 --admission DATA/results/candidate-build-fresh-8fb6f413179b-01/simulator-helper-build-admission-01.json --admission-sha256 SHA256_OF_FINAL_ADMISSION
DATA=/run/media/kmosoti/data/FabricO11y. Before execution recheck board/main/source/artifacts and exact admission; no simulated approval/URL. Build controller requires complete canonical path equality except the helper file, source commit/tree/bundle, toolchain tree fingerprint, exact registry inventory, pinned container tools and prepared immutable sysroot. Outer default 1800 seconds; inner build 1500; actual 7/8 GiB high/max, no swap; two build jobs; rootless container, read-only root/source/toolchain, no network; new empty target and private registry. Full-tree census plus 9 GiB reservation must remain below 95e9, owned/result growth guards are 8/1 GiB, absolute project ceiling 100e9. One heavy job only. No package/UI/dump-helper build. Retain all failures, raw commands and stop/cleanup receipts. Separate helper ELF goes in DATA/results/spindle-cold-native-witness-01/.
Expected outcome: compile exit 0 and retained ELF/GLIBC-floor receipt; any compiler/setup/guard failure is preserved and never counts as native witness or acceptance evidence. Native positive, negative, fault and syscall-order controls require their subsequent exact plans. No release, tag or package refreeze is proposed.
Exact prepared inputs:
{
"schema": "separate-spindle-cold-build-v1",
"candidate": "8fb6f413179b094288316e25845c137440ba23fa",
"command": [
"cargo",
"build",
"--offline",
"--locked",
"--release",
"--workspace",
"--example",
"spindle_sim"
],
"source_date_epoch": 1791841113,
"preflight_url": null,
"source_head": "af3f40c5bda62fe8454917077dae4960cb56d406",
"source": {
"commit": "1a923e70bed65b899382c7aad06590ae11d46855",
"tree": "5b49e6b9502c53adddf0d1ec6fef797266d67a30",
"manifest": "/run/media/kmosoti/data/FabricO11y/results/simulator-native-source-01/provenance/source-manifest.json",
"manifest_sha256": "0398b2b7220990ece3812167508054897949b18b89c6fc73d81a581ceefd232a",
"bundle": "/run/media/kmosoti/data/FabricO11y/results/simulator-native-source-01/provenance/frozen-source.bundle",
"bundle_sha256": "4c2e8be5b1495be3c02ac9ba5dfd3ffb5ca3ca4d1c7487e0b8bdefb3c0c8a3da"
},
"prepared_image": {
"archive": "/run/media/kmosoti/data/FabricO11y/results/spindle-sysroot-prepare-01/prepared.oci.tar",
"archive_sha256": "eaf3e44ac1ee1e544f8cf702219260cc5ead7eaa299d93c93455837ad0a9df34",
"image_id": "3d2067dbc9e4867194dc450f8ee3c10fec1ae3dbe168c5f4035992726ee91bfc",
"rootfs_diff_ids": [
"sha256:e58989146fcec76699871403b9897b5c014af1ccd85f5a9300a215e7975525d7",
"sha256:fa731dba66c1b00343f2e35ba18bee34af8f671bf6c7bbd0e1ee6d16144b6dd3"
],
"sysroot_summary_sha256": "39c360e023cd296b08a4d58b9f59218a03eb0ebc21d0fe1b7dd82d103059d0ac",
"sysroot_summary": "/run/media/kmosoti/data/FabricO11y/results/spindle-sysroot-prepare-01/sysroot-prepared/summary.json"
},
"toolchain_tree_sha256": "fcb07970ad99acfe964448e221eab95376eb86ea9dfe2975aa4c76b499933559",
"fingerprint_script_sha256": "0edcd3be23dbf30e4826879f36cd05a6d677cca3c3981d87ffa06f0251abacd2",
"growth_script_sha256": "e0ebea0b7635732ec0ed785c07954889e7d7349f1a04a88e5087d22ae86972ae"
}
Exact controller sources
These are the complete controllers; their bytes are part of this pre-flight.
build-simulator-helper-proposal-01.py
SHA-256 747506ea37d172c2957823410b37bf7bbe02dba3a28fd067c60853b54cd5b86d.
#!/usr/bin/env python3
"""PROSPECTIVE helper-only build; refuse missing external source/image authority.
No preparation, downloads, package/UI/WASM/dump-helper build or shared target.
Root must review/hash this file and admit its exact command before invocation.
"""
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import shutil
import signal
import subprocess
import sys
import threading
import time
DATA = Path('/run/media/kmosoti/data/FabricO11y')
META = DATA / 'results/candidate-build-fresh-8fb6f413179b-01'
FROZEN = '8fb6f413179b094288316e25845c137440ba23fa'
CANONICAL = DATA / 'results/installation-package-build/fresh-8fb6f413179b-01-deb-a/provenance/source-manifest.json'
CANONICAL_SHA = '534b601f904ce80f50cc545be8b8b40f47f8940975fd4cbf2e2e8c8cb54f2cff'
TC_HOME = DATA / 'toolchain-cache/rustup'
TC = TC_HOME / 'toolchains/1.99.0-x86_64-unknown-linux-gnu'
REGISTRY = DATA / 'toolchain-cache/cargo/registry'
REGISTRY_SHA = '47646ea10bb1e5fe179fc11ce36e0fe11acb6f5dacd9aa7d09c0c9a6d18577ce'
FINGERPRINT = META / 'simulator-helper-fingerprint-proposal-01.py'
GROWTH = META / 'simulator-helper-growth-census-proposal-01.py'
HISTORICAL_PINS = META / 'simulator-helper-historical-apt-pins-01.md'
HISTORICAL_SHA = '323ef7b4e9f247625ee79760a2d59a5a0eb76fcf8a697bc2ab0f34ca9b9ad3bc'
RESERVE, OWNED_MAX, RESULT_MAX = 9 * 1024**3, 8 * 1024**3, 1024**3
COMMAND = ['cargo', 'build', '--offline', '--locked', '--release', '--workspace', '--example', 'spindle_sim']
def need(ok, message):
if not ok:
raise RuntimeError(message)
def sha(path):
with open(path, 'rb') as stream:
return hashlib.file_digest(stream, 'sha256').hexdigest()
def read(path, expected=None):
need(not path.is_symlink() and path.is_file() and path.stat().st_size <= 2**20, 'bounded regular authority file')
need(expected is None or sha(path) == expected, 'external authority bytes drifted')
return json.loads(path.read_text())
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--id', required=True)
parser.add_argument('--admission', type=Path, required=True)
parser.add_argument('--admission-sha256', required=True)
args = parser.parse_args()
need(re.fullmatch('[a-z0-9-]{1,64}', args.id) is not None, 'owned run identifier')
authority = read(args.admission, args.admission_sha256)
need(authority['schema'] == 'separate-spindle-cold-build-v1' and authority['candidate'] == FROZEN
and authority['command'] == COMMAND and authority['source_date_epoch'] == 1791841113,
'admitted candidate/command/epoch')
need(re.fullmatch('https://github.com/kmosoti/FabricO11y/issues/59#issuecomment-[0-9]+',
authority['preflight_url']) is not None, 'external preparation/build preflight reference')
need(sha(FINGERPRINT) == authority['fingerprint_script_sha256']
and sha(GROWTH) == authority['growth_script_sha256'], 'controller scripts drifted')
need(not HISTORICAL_PINS.is_symlink() and sha(HISTORICAL_PINS) == HISTORICAL_SHA,
'historical partial package-version record drifted')
need(sha(Path('/usr/bin/podman')) == 'a844579b16612e6d0fd513bac59e27d82bb1220a34820f1e1e1e4c683b2f1454'
and sha(Path('/usr/bin/fuse-overlayfs')) == '63e93359cc0524a0f618e2bb228ebf45a0780c37b3f0d44c681b4ac6b739be69', 'host container tools drifted')
sys.path.insert(0, str(DATA / 'scratch/candidate-build-fresh-8fb6f413179b-01-a/tools'))
from resource_group import require_limits, require_storage, STORAGE
need(STORAGE == DATA and os.environ.get('FABRIC_RESOURCE_RUNTIME_SECONDS') == '1800', 'exact DATA/default deadline launcher')
group = require_limits(); require_storage()
subprocess.run(['systemctl', '--user', 'set-property', '--runtime', group.name,
'MemoryHigh=7G', 'MemoryMax=8G'], check=True, timeout=30)
need((group / 'memory.high').read_text().strip() == str(7 * 1024**3)
and (group / 'memory.max').read_text().strip() == str(8 * 1024**3)
and (group / 'memory.swap.max').read_text().strip() == '0', '7/8GiB no-swap enforcement')
need(shutil.disk_usage(DATA).free >= RESERVE, 'physical free space below full9GiB allowance')
need(sha(CANONICAL) == CANONICAL_SHA, 'canonical source authority drifted')
source_authority, prepared = authority['source'], authority['prepared_image']
manifest = read(Path(source_authority['manifest']), source_authority['manifest_sha256'])
original = read(CANONICAL)['source_files']; members = manifest['source_files']
need(manifest['original_head'] == authority['source_head'] and manifest['source_commit'] == source_authority['commit']
and manifest['source_tree'] == source_authority['tree'] and members.keys() == original.keys(), 'complete frozen source set')
helper = 'examples/spindle_sim.rs'
need(members[helper]['sha256'] != original[helper]['sha256'] and all(members[name] == original[name]
for name in original if name != helper), 'sole helper compilation-input change')
need(sum(record['bytes'] for record in members.values()) <= 1024**3, 'bounded complete source copy')
for item in (source_authority['bundle'], prepared['archive']):
need(Path(item).resolve(strict=True).is_relative_to(DATA) and not Path(item).is_symlink(), 'DATA input only')
need(sha(Path(source_authority['bundle'])) == source_authority['bundle_sha256']
and manifest['source_bundle_sha256'] == source_authority['bundle_sha256'], 'source bundle binding')
need(sha(Path(prepared['archive'])) == prepared['archive_sha256'], 'prepared OCI archive binding')
sysroot = read(Path(prepared['sysroot_summary']), prepared['sysroot_summary_sha256'])
need(sha(TC / 'bin/rustc') == 'f3834d26669b03f6855fa54bd2381443123e860167bc0c7a8dd137e5cf6e5e4f'
and sha(TC / 'bin/cargo') == 'e951141cc55a6cd7b9876d187bd30a3720e6086b99413af95e3b8de1cdd72f14', 'pinned compiler executables')
scratch = Path(os.environ['FABRIC_SCRATCH_ROOT']).resolve(strict=True)
need(scratch.is_relative_to(DATA / 'scratch'), 'DATA scratch root')
owned, result = scratch / ('spindle-cold-' + args.id), DATA / 'results' / ('spindle-cold-' + args.id)
need(not owned.exists() and not result.exists() and not owned.is_symlink() and not result.is_symlink(), 'fresh owned roots')
owned.mkdir(mode=0o700); result.mkdir(mode=0o700)
marker = owned / '.owned-spindle-helper'; marker.write_text(args.id + '\n')
store = owned / 'podman'
for name in ('graph', 'run', 'tmp'):
(store / name).mkdir(parents=True)
for name in ('target', 'cargo', 'out', 'scratch', 'logs'):
(owned / name).mkdir()
podman = ['/usr/bin/podman', '--root', str(store / 'graph'), '--runroot', str(store / 'run'),
'--tmpdir', str(store / 'tmp'), '--storage-driver', 'overlay',
'--storage-opt', 'overlay.mount_program=/usr/bin/fuse-overlayfs']
environment = dict(os.environ, TMPDIR=str(store / 'tmp'), TMP=str(store / 'tmp'), TEMP=str(store / 'tmp'),
GIT_CONFIG_NOSYSTEM='1', GIT_CONFIG_GLOBAL='/dev/null')
receipt = dict(state='not_built', classification='separate native helper; no package/qualification',
command=COMMAND, source=source_authority, prepared_image=prepared,
admission_sha256=args.admission_sha256, owned=str(owned), result=str(result),
cgroup=str(group), reserved_bytes=RESERVE, commands=[], samples=[])
running = None; failures = []; stop_meter = threading.Event(); meter = None
container_name = 'spindle-helper-' + args.id
def command(label, argv, timeout=60):
need(not failures, 'growth guard already failed')
with open(owned / 'logs' / (label + '.stdout'), 'wb') as out, open(owned / 'logs' / (label + '.stderr'), 'wb') as err:
completed = subprocess.run(argv, env=environment, stdout=out, stderr=err, timeout=timeout)
receipt['commands'].append(dict(label=label, command=argv, exit=completed.returncode))
need(completed.returncode == 0, label + ' failed')
need(not failures, 'growth guard failed during command')
metadata = owned / 'logs' / (label + '.stdout')
need(metadata.stat().st_size <= 2**20, 'bounded command metadata before reading')
return metadata.read_text()
def observe():
completed = subprocess.run([*podman, 'unshare', 'python3', '-B', str(GROWTH),
'--owned', str(owned), '--result', str(result)], env=environment, text=True,
capture_output=True, timeout=25)
need(completed.returncode == 0 and len(completed.stdout) <= 16384, 'strict owned census failed')
value = json.loads(completed.stdout); receipt['samples'].append(value)
need(value['owned']['logical_bytes'] < OWNED_MAX and value['result']['logical_bytes'] < RESULT_MAX
and value['free_bytes'] >= RESULT_MAX, 'owned/output/free-space growth guard')
return value
def sample():
while not stop_meter.wait(5):
try:
observe()
except BaseException as error:
failures.append(str(error))
if running is not None and running.poll() is None:
running.send_signal(signal.SIGTERM)
break
try:
info = command('podman-info', [*podman, 'info', '--format',
'{{.Host.Security.Rootless}}|{{.Store.GraphRoot}}|{{.Store.RunRoot}}|{{.Store.ImageCopyTmpDir}}'])
need(info.strip().split('|') == ['true', str(store / 'graph'), str(store / 'run'), str(store / 'tmp')], 'rootless owned storage')
observe(); meter = threading.Thread(target=sample); meter.start()
command('toolchain-census-before', ['python3', '-B', str(FINGERPRINT), 'tree', '--root', str(TC_HOME),
'--out', str(result / 'toolchain-before')])
need(sha(result / 'toolchain-before/tree.jsonl') == authority['toolchain_tree_sha256'], 'complete toolchain fingerprint')
command('source-clone', ['git', '-c', 'core.hooksPath=/dev/null', 'clone', '--no-checkout',
str(Path(source_authority['bundle'])), str(owned / 'source')])
command('source-checkout', ['git', '-C', str(owned / 'source'), '-c', 'core.hooksPath=/dev/null',
'checkout', '--detach', source_authority['commit']])
need(command('source-tree', ['git', '-C', str(owned / 'source'), 'rev-parse', 'HEAD^{tree}']).strip()
== source_authority['tree'], 'actual private source tree')
listed = command('source-paths', ['git', '-C', str(owned / 'source'), 'ls-files']).splitlines()
need(set(listed) == set(members), 'actual source path set')
for name, record in members.items():
path = owned / 'source' / name
need(not path.is_symlink() and path.is_file() and path.stat().st_size == record['bytes']
and sha(path) == record['sha256'], 'actual frozen source bytes')
shutil.copytree(REGISTRY, owned / 'cargo/registry', symlinks=True)
entries, copied = [], 0
for base, dirs, files in os.walk(owned / 'cargo/registry', followlinks=False):
dirs.sort()
need(not any((Path(base) / name).is_symlink() for name in dirs + files), 'registry symlink refused')
for name in sorted(files):
path = Path(base) / name; size = path.stat().st_size
need(path.is_file(), 'regular registry files only')
copied += size; entries.append((path.relative_to(owned / 'cargo/registry').as_posix(), size, sha(path)))
need(copied == 438223080 and hashlib.sha256(json.dumps(entries, separators=(',', ':')).encode()).hexdigest()
== REGISTRY_SHA, 'exact copied registry inventory')
need(not any((owned / 'target').iterdir()), 'cold empty target')
command('image-load', [*podman, 'load', '-i', prepared['archive']], timeout=300)
actual = json.loads(command('image-inspect', [*podman, 'image', 'inspect', prepared['image_id']]))[0]
need(actual['Id'] == prepared['image_id'] and actual['RootFS']['Layers'] == prepared['rootfs_diff_ids'], 'immutable prepared image identity')
need(actual['Architecture'] == 'amd64' and actual['Os'] == 'linux', 'native Debian architecture')
relative = '/' + str(group.relative_to(Path('/sys/fs/cgroup')))
guest = r'''set -euo pipefail
actual=$(awk -F: '$1 == "0" {print $3}' /proc/self/cgroup)
[ "$actual" = "$FABRIC_EXPECTED_CGROUP" ]
[ "$(cat /sys/fs/cgroup$actual/memory.high)" = 7516192768 ]
[ "$(cat /sys/fs/cgroup$actual/memory.max)" = 8589934592 ]
[ "$(cat /sys/fs/cgroup$actual/memory.swap.max)" = 0 ]
python3 -B /control/fingerprint.py sysroot --out /out/sysroot-checked --historical-pins /control/historical-pins.md
python3 -B -c 'import hashlib,os; p="/out/sysroot-checked/summary.json"; assert hashlib.sha256(open(p,"rb").read()).hexdigest()==os.environ["FABRIC_EXPECTED_SYSROOT_SUMMARY_SHA256"]'
export PATH=/opt/rustup/toolchains/1.99.0-x86_64-unknown-linux-gnu/bin:/usr/bin:/bin
export RUSTUP_HOME=/opt/rustup CARGO_HOME=/cargo CARGO_TARGET_DIR=/target
export CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=2 CARGO_NET_OFFLINE=true
export TMPDIR=/scratch TMP=/scratch TEMP=/scratch SOURCE_DATE_EPOCH=1791841113
unset RUSTFLAGS CARGO_ENCODED_RUSTFLAGS RUSTC_WRAPPER RUSTC_WORKSPACE_WRAPPER LD_PRELOAD LD_AUDIT LD_LIBRARY_PATH
rustc --version --verbose > /out/rustc-version.txt
grep -Fx 'commit-hash: b940084d7eb6a299eb4bfeb8e34901bc051e7ac4' /out/rustc-version.txt
grep -Fx 'host: x86_64-unknown-linux-gnu' /out/rustc-version.txt
grep -Fx 'release: 1.99.0' /out/rustc-version.txt
grep -Fx 'LLVM version: 23.1.1' /out/rustc-version.txt
cd /src
git diff --quiet
cargo build --offline --locked --release --workspace --example spindle_sim
cp /target/release/examples/spindle_sim /out/spindle_sim
MAX_GLIBC=2.34 bash /src/packaging/check-glibc.sh /out/spindle_sim > /out/glibc-floor.txt
'''
invocation = [*podman, 'run', '--name', container_name, '--pull=never', '--network=none',
'--read-only', '--read-only-tmpfs=false', '--cgroups=disabled', '--cgroupns=host', '--systemd=false',
'--security-opt=label=disable', '--env', 'FABRIC_EXPECTED_CGROUP=' + relative,
'--env', 'FABRIC_EXPECTED_SYSROOT_SUMMARY_SHA256=' + prepared['sysroot_summary_sha256']]
for source, target, mode in ((owned / 'source', '/src', 'ro'), (TC_HOME, '/opt/rustup', 'ro'),
(owned / 'cargo', '/cargo', 'rw'), (owned / 'target', '/target', 'rw'),
(owned / 'scratch', '/scratch', 'rw'), (owned / 'out', '/out', 'rw'),
(FINGERPRINT, '/control/fingerprint.py', 'ro'),
(HISTORICAL_PINS, '/control/historical-pins.md', 'ro')):
invocation += ['--volume', str(source) + ':' + target + ':' + mode]
invocation += [prepared['image_id'], 'bash', '-euc', guest]
need(not failures, 'growth guard failed before compile')
with open(owned / 'logs/build.stdout', 'wb') as out, open(owned / 'logs/build.stderr', 'wb') as err:
running = subprocess.Popen(invocation, env=environment, stdout=out, stderr=err)
status = running.wait(timeout=1500)
receipt['commands'].append(dict(label='helper-build', command=invocation, exit=status))
need(status == 0 and not failures, 'native build failed/interrupted')
need(read(owned / 'out/sysroot-checked/summary.json') == sysroot, 'actual immutable sysroot fingerprint')
receipt['sysroot_checked_summary_sha256'] = sha(owned / 'out/sysroot-checked/summary.json')
command('toolchain-census-after', ['python3', '-B', str(FINGERPRINT), 'tree', '--root', str(TC_HOME),
'--out', str(result / 'toolchain-after')])
need(sha(result / 'toolchain-after/tree.jsonl') == authority['toolchain_tree_sha256'], 'toolchain changed during build')
observe()
shutil.copy2(owned / 'out/spindle_sim', result / 'spindle_sim')
for name in ('rustc-version.txt', 'glibc-floor.txt'):
shutil.copy2(owned / 'out' / name, result / name)
receipt.update(state='built', exit=0, binary_sha256=sha(result / 'spindle_sim'),
registry_bytes=copied, registry_sha256=REGISTRY_SHA)
except BaseException as error:
receipt.update(state='failed', exit=2, error=f'{type(error).__name__}: {error}')
finally:
stop_meter.set()
if meter is not None:
meter.join(timeout=30)
receipt['growth_guard_stopped'] = meter is None or not meter.is_alive()
# Only exact owned container/store; unconfirmed stop forbids scratch deletion.
active = None; client_stopped = running is None
try:
removed = subprocess.run([*podman, 'rm', '--force', container_name], env=environment,
capture_output=True, text=True, timeout=30)
active = subprocess.run([*podman, 'ps', '-q'], env=environment, capture_output=True, text=True, timeout=30)
receipt['container_cleanup'] = dict(exit=removed.returncode, remaining=active.stdout.strip())
except BaseException as error:
receipt['container_cleanup'] = dict(error=f'{type(error).__name__}: {error}')
try:
if running is not None:
try:
running.wait(timeout=10)
except subprocess.TimeoutExpired:
running.kill(); running.wait(timeout=10)
client_stopped = running.poll() is not None
except BaseException as error:
receipt['client_stop_error'] = f'{type(error).__name__}: {error}'
receipt['stop_confirmed'] = (active is not None and active.returncode == 0 and not active.stdout.strip()
and receipt['growth_guard_stopped'] and client_stopped)
try:
snapshot = observe()
logs = list((owned / 'logs').iterdir())
need(all(not path.is_symlink() and path.is_file() for path in logs), 'regular owned logs only')
need(sum(path.stat().st_size for path in logs) + snapshot['result']['logical_bytes'] + 2**20 < RESULT_MAX,
'raw-log archive exceeds output allowance')
shutil.copytree(owned / 'logs', result / 'logs')
observe()
except BaseException as error:
receipt.update(state='failed', exit=2, archive_error=f'{type(error).__name__}: {error}')
try:
receipt['memory'] = {name: (group / name).read_text().strip() for name in ('memory.peak', 'memory.events', 'memory.high', 'memory.max', 'memory.swap.max')}
except BaseException as error:
receipt.update(state='failed', exit=2, memory_receipt_error=f'{type(error).__name__}: {error}')
receipt['cleanup'] = 'owned scratch retained for root/Pip archive-and-cleanup review; no shared/frozen paths touched'
if not receipt['stop_confirmed']:
receipt.update(state='failed', exit=2, error='unconfirmed container/guard stop')
receipt_text = json.dumps(receipt, indent=2) + '\n'
need(len(receipt_text.encode()) <= 2**20, 'bounded final receipt within reserved output headroom')
(result / 'receipt.json').write_text(receipt_text)
return receipt['exit']
if __name__ == '__main__':
sys.exit(main())
simulator-helper-fingerprint-proposal-01.py
SHA-256 0edcd3be23dbf30e4826879f36cd05a6d677cca3c3981d87ffa06f0251abacd2.
#!/usr/bin/env python3
"""PROSPECTIVE ONLY: complete streamed inventories, never a build or admission.
Root must admit an exact command before invocation. `sysroot` may optionally
download installed-package archives; that option belongs only to the separately
admitted, disposable image-preparation cell. `tree` is read-only collection.
"""
import argparse
import hashlib
import json
import os
from pathlib import Path
import stat
import subprocess
REQUESTED = {'build-essential': '12.9', 'binutils': '2.40-2',
'ca-certificates': '20250419~deb12u1', 'dpkg-dev': '1.21.23',
'git': '1:2.39.5-0+deb12u3', 'libssl-dev': '3.0.22-1~deb12u1',
'pkg-config': '1.8.1-1', 'python3': '3.11.2-1+b1', 'rpm': '4.18.0+dfsg-1+deb12u1'}
HISTORICAL_SHA = '323ef7b4e9f247625ee79760a2d59a5a0eb76fcf8a697bc2ab0f34ca9b9ad3bc'
EXCLUDED = ('/proc', '/sys', '/dev', '/run', '/out', '/control', '/src', '/cargo',
'/target', '/scratch', '/opt/rustup', '/etc/hosts', '/etc/hostname', '/etc/resolv.conf')
def need(ok, message):
if not ok:
raise RuntimeError(message)
def sha(path):
with open(path, 'rb') as stream:
return hashlib.file_digest(stream, 'sha256').hexdigest()
def emit(stream, record):
stream.write(json.dumps(record, sort_keys=True, separators=(',', ':')) + '\n')
def inventory(root, destination, excluded=()):
root = root.resolve(strict=True)
need(not destination.exists(), 'fresh inventory file required')
count = total = 0
with destination.open('x') as stream:
emit(stream, {'schema': 'semantic-tree-v1', 'excluded': list(excluded)})
root_stat = root.stat()
emit(stream, dict(path='.', kind='directory', mode=stat.S_IMODE(root_stat.st_mode), uid=root_stat.st_uid, gid=root_stat.st_gid))
count += 1
for base, dirs, files in os.walk(root, followlinks=False,
onerror=lambda error: (_ for _ in ()).throw(error)):
dirs.sort(); files.sort()
for name in dirs[:]:
path = Path(base) / name
if str(path) in excluded or path.is_symlink():
dirs.remove(name)
for name in sorted(set(dirs + files + [name for name in os.listdir(base)
if (Path(base) / name).is_symlink()])):
path = Path(base) / name
if str(path) in excluded:
continue
before = path.lstat()
record = dict(path=path.relative_to(root).as_posix(), mode=stat.S_IMODE(before.st_mode),
uid=before.st_uid, gid=before.st_gid)
if stat.S_ISREG(before.st_mode):
record.update(kind='file', bytes=before.st_size, sha256=sha(path))
total += before.st_size
elif stat.S_ISDIR(before.st_mode):
record.update(kind='directory')
elif stat.S_ISLNK(before.st_mode):
record.update(kind='symlink', target=os.readlink(path))
else:
raise RuntimeError(f'unexpected rootfs object: {path}')
after = path.lstat()
need((before.st_dev, before.st_ino, before.st_mode, before.st_size, before.st_mtime_ns)
== (after.st_dev, after.st_ino, after.st_mode, after.st_size, after.st_mtime_ns),
f'inventory input changed: {path}')
emit(stream, record); count += 1
return dict(sha256=sha(destination), entries=count, regular_bytes=total)
def installed():
result = subprocess.run(['dpkg-query', '-W', '-f',
'${Package}\t${Architecture}\t${Version}\t${db:Status-Status}\n'],
check=True, text=True, capture_output=True, timeout=60)
need(len(result.stdout) <= 2**20, 'finite package inventory')
packages = sorted(tuple(line.split('\t')[:3]) for line in result.stdout.splitlines()
if line.split('\t')[-1] == 'installed')
need(packages and len(packages) == len(set(packages)), 'complete unique installed packages')
actual = {name: version for name, arch, version in packages}
need(all(actual.get(name) == version for name, version in REQUESTED.items()), 'historical requested versions drifted')
return packages
def sysroot(out, download, historical_pins):
relative = next(line[3:] for line in Path('/proc/self/cgroup').read_text().splitlines() if line.startswith('0::'))
need(relative == os.environ.get('FABRIC_EXPECTED_CGROUP'), 'guest must share the admitted parent cgroup')
group = Path('/sys/fs/cgroup') / relative.lstrip('/')
need((group / 'memory.high').read_text().strip() == str(7 * 1024**3)
and (group / 'memory.max').read_text().strip() == str(8 * 1024**3)
and (group / 'memory.swap.max').read_text().strip() == '0', 'guest 7/8GiB no-swap enforcement')
packages = installed()
need(historical_pins is not None and not historical_pins.is_symlink()
and historical_pins.is_file() and historical_pins.stat().st_size <= 2**20
and sha(historical_pins) == HISTORICAL_SHA, 'exact historical partial-version record')
known = []
for line in historical_pins.read_text().splitlines():
if line.startswith('| ') and not line.startswith(('| Package |', '| --- |')):
fields = tuple(item.strip() for item in line.split('|')[1:-1])
need(len(fields) == 3, 'three historical package fields')
known.append(fields)
need(len(known) == 108 and len(set(known)) == 108
and set(known).issubset(set(packages)), 'historical transitive versions drifted or missing')
cache = Path('/var/cache/apt/archives')
if download:
arguments = [name + (':' + arch if arch != 'all' else '') + '=' + version
for name, arch, version in packages]
# Exact installed versions, including base packages; never upgrade or substitute.
subprocess.run(['apt-get', 'download', *arguments], cwd=cache, check=True, timeout=900)
expected, seen = set(packages), set()
with (out / 'packages.jsonl').open('x') as stream:
for name, arch, version in packages:
emit(stream, dict(package=name, architecture=arch, version=version))
with (out / 'archives.jsonl').open('x') as stream:
for path in sorted(cache.glob('*.deb')):
need(not path.is_symlink() and path.is_file(), 'regular archive only')
result = subprocess.run(['dpkg-deb', '-f', str(path), 'Package', 'Architecture', 'Version'],
check=True, capture_output=True, text=True, timeout=30)
fields = dict(line.split(': ', 1) for line in result.stdout.splitlines())
key = fields['Package'], fields['Architecture'], fields['Version']
need(key in expected and key not in seen, 'archive set differs from installed package set')
seen.add(key)
emit(stream, dict(package=key[0], architecture=key[1], version=key[2],
file=path.name, bytes=path.stat().st_size, sha256=sha(path)))
need(seen == expected, 'every installed package needs its exact archive; no base-package omission')
rootfs = inventory(Path('/'), out / 'rootfs.jsonl', EXCLUDED)
return dict(schema='prepared-helper-sysroot-v1', requested=REQUESTED, installed_packages=len(packages),
historical_partial_versions_sha256=HISTORICAL_SHA, historical_partial_versions=108,
packages_sha256=sha(out / 'packages.jsonl'), archives_sha256=sha(out / 'archives.jsonl'),
rootfs=rootfs, exclusions=list(EXCLUDED))
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('mode', choices=('tree', 'sysroot'))
parser.add_argument('--root', type=Path)
parser.add_argument('--out', type=Path, required=True)
parser.add_argument('--download-all-installed', action='store_true')
parser.add_argument('--historical-pins', type=Path)
args = parser.parse_args()
need(not args.out.exists(), 'fresh owned output required')
args.out.mkdir(mode=0o700, parents=False)
if args.mode == 'tree':
need(args.root is not None and not args.download_all_installed and args.historical_pins is None,
'tree collection has no network or package input')
summary = inventory(args.root, args.out / 'tree.jsonl')
else:
summary = sysroot(args.out, args.download_all_installed, args.historical_pins)
(args.out / 'summary.json').write_text(json.dumps(summary, sort_keys=True, indent=2) + '\n')
if __name__ == '__main__':
main()
simulator-helper-growth-census-proposal-01.py
SHA-256 e0ebea0b7635732ec0ed785c07954889e7d7349f1a04a88e5087d22ae86972ae.
#!/usr/bin/env python3
"""PROSPECTIVE sampled census payload, invoked only inside rootless userns.
No deletion, admission or stop authority. Builder/outer driver act on the
observations. Logical pathname bytes are not filesystem quota or transient max.
"""
import argparse
import json
import os
from pathlib import Path
import time
DATA = Path('/run/media/kmosoti/data/FabricO11y')
def census(root):
total = count = 0
for base, dirs, files in os.walk(root, followlinks=False,
onerror=lambda error: (_ for _ in ()).throw(error)):
for name in dirs[:]:
path = Path(base) / name
if path.is_symlink():
dirs.remove(name)
total += path.lstat().st_size; count += 1
for name in files:
total += (Path(base) / name).lstat().st_size; count += 1
return dict(logical_bytes=total, paths=count)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--owned', type=Path, required=True)
parser.add_argument('--result', type=Path, required=True)
args = parser.parse_args()
if os.getuid() != 0 or os.getgid() != 0:
raise RuntimeError('must inspect subordinate-UID files from Podman user namespace')
for path in (args.owned, args.result):
if not path.resolve(strict=True).is_relative_to(DATA) or path.is_symlink():
raise RuntimeError('owned DATA roots only')
started = time.monotonic_ns()
for attempt in range(3):
try:
observation = dict(owned=census(args.owned), result=census(args.result),
free_bytes=os.statvfs(DATA).f_bavail * os.statvfs(DATA).f_frsize)
break
except FileNotFoundError:
if attempt == 2:
raise
time.sleep(0.1)
observation.update(started_monotonic_ns=started, finished_monotonic_ns=time.monotonic_ns(),
unix_ns=time.time_ns(), uid=os.getuid(), gid=os.getgid(), attempt=attempt + 1)
print(json.dumps(observation, sort_keys=True))
if __name__ == '__main__':
main()
run-step-simulator-documentation-01.py
SHA-256 39333a5c90b16f2f75a69a030c69c9272a62c44fbb6b43fec10d27111cdfcea1.
import datetime, hashlib, json, os, re, signal, subprocess, sys, time
from pathlib import Path
ROOT = Path('/run/media/kmosoti/data/FabricO11y')
REPO = ROOT / 'scratch/simulator-exposure-witness'
OUT = ROOT / 'results/service-opening-candidate'
STORE = ROOT / 'scratch/service-opening-census'
FROZEN = '6bd5250a6223fe3663b254c25627ce4ff2b4c9e0'
def require(value, reason):
if not value:
raise ValueError(reason)
label, reserve_text, *command = sys.argv[1:]
reserve = int(reserve_text)
delegated = command[:1] == ['--delegate']
if delegated: command = command[1:]
require(re.fullmatch(r'[a-z0-9-]{1,80}', label) and reserve > 0 and command, 'ordinary workload admission guard')
record_path = OUT / (label + '.json')
require(not record_path.exists(), 'unique run label required')
def census():
# Rootless container files can belong to subordinate UIDs. Inspect all of
# them in the same user namespace; never skip unreadable directories.
require(not STORE.is_symlink(), 'ordinary workload admission guard')
require((STORE / '.owned-census-store').read_text() == 'service-opening-census-v1\n', 'ordinary workload admission guard')
args = ['podman', '--root', str(STORE/'graph'), '--runroot', str(STORE/'run'),
'--tmpdir', str(STORE/'tmp'), '--storage-driver', 'vfs',
'unshare', 'python3', '-B', str(OUT/'census.py')]
env = dict(os.environ, TMPDIR=str(STORE/'tmp'), TMP=str(STORE/'tmp'), TEMP=str(STORE/'tmp'))
result = subprocess.run(args, env=env, text=True, capture_output=True, timeout=25)
if result.returncode:
raise RuntimeError(f'strict namespace census failed ({result.returncode}): {result.stderr}')
observed = json.loads(result.stdout)
require(observed['uid'] == 0 and observed['gid'] == 0, 'ordinary workload admission guard')
require(type(observed['data_bytes']) is int and observed['data_bytes'] >= 0, 'ordinary workload admission guard')
return observed
def git(*args):
return subprocess.check_output(['git', *args], cwd=REPO, text=True).strip()
require(git('rev-parse', 'HEAD') == FROZEN and not git('status', '--porcelain'), 'clean frozen source required')
require(not subprocess.check_output(['systemctl', '--user', 'list-units', 'fabric-work-*', '--state=running', '--plain', '--no-legend'], text=True).strip(), 'another project workload is active')
initial = census()
used = initial['data_bytes']
record = {'label': label, 'command': command, 'delegated': delegated, 'frozen': None, 'scope': 'harness controls only; no candidate acceptance evidence', 'harness_revision': FROZEN, 'source_status_before': git('status', '--porcelain'), 'storage_root': str(ROOT), 'admission_bytes': used, 'admission_observation': initial, 'reserved_bytes': reserve, 'observations': [], 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat()}
if used + reserve > 95_000_000_000:
record.update(state='not_run', reason='95 GB storage admission threshold')
record_path.write_text(json.dumps(record, indent=2)+'\n')
raise SystemExit('storage admission denied; project command not launched')
receipts = REPO / 'target/resource-containment/runs'
before = set(receipts.glob('*.json'))
env = dict(os.environ, FABRIC_STORAGE_ROOT=str(ROOT), CARGO_HOME=str(ROOT/'toolchain-cache/cargo'), RUSTUP_HOME=str(ROOT/'toolchain-cache/rustup'), CARGO_BUILD_JOBS='2')
env['PATH'] = str(ROOT/'tools/candidate-dpkg/bin') + ':' + env['PATH']
log_path = OUT / (label + '.log')
record.update(state='running', log=str(log_path))
record_path.write_text(json.dumps(record, indent=2)+'\n')
def find_unit():
with log_path.open() as stream:
prefix = stream.read(65536)
match = re.search(r'Running as unit: (fabric-work-[0-9a-f]+)\.service', prefix)
return match.group(1) if match else None
def stop_owned(process):
unit = find_unit()
if unit:
stopped = subprocess.run(['systemctl', '--user', 'stop', unit],
text=True, capture_output=True, timeout=25)
record['stop'] = {'unit': unit, 'exit': stopped.returncode, 'stderr': stopped.stderr}
if stopped.returncode:
# Interrupt the launcher too; its finally block independently
# confirms that this exact service has no surviving descendants.
process.send_signal(signal.SIGINT)
else:
process.send_signal(signal.SIGINT)
record['stop'] = {'unit': None, 'launcher_interrupted': True}
with log_path.open('wb') as log:
launcher = ['python3', '-B', 'tools/resource_group.py', *(['--delegate'] if delegated else []), '--', *command]
process = subprocess.Popen(launcher, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT)
next_census = time.monotonic() + 30
try:
while process.poll() is None:
time.sleep(1)
if time.monotonic() >= next_census:
observed = census()
record['observations'].append({'unix':time.time(), **observed})
record_path.write_text(json.dumps(record, indent=2)+'\n')
if observed['data_bytes'] >= 95_000_000_000:
record['interruption'] = '95 GB threshold reached'
break
next_census = time.monotonic() + 30
except BaseException as error:
record['interruption'] = f'{type(error).__name__}: {error}'
finally:
if record.get('interruption') and process.poll() is None:
try:
stop_owned(process)
except BaseException as error:
record['stop_error'] = f'{type(error).__name__}: {error}'
process.send_signal(signal.SIGINT)
status = process.wait()
record['launcher_exit'] = status
if record.get('interruption'): status = 2
with log_path.open('rb') as stream:
log_digest = hashlib.file_digest(stream, 'sha256').hexdigest()
record.update(state='interrupted' if record.get('interruption') else 'completed', exit=status, source_status_after=git('status', '--porcelain'), finished_utc=datetime.datetime.now(datetime.timezone.utc).isoformat(), log_sha256=log_digest, resource_receipts=[])
archive = OUT/'resource-containment'; archive.mkdir(exist_ok=True)
for p in sorted(set(receipts.glob('*.json')) - before):
value=json.loads(p.read_text())
if value['command'] != command: continue
q=archive/p.name; q.write_bytes(p.read_bytes())
record['resource_receipts'].append({'path':str(q),'sha256':hashlib.sha256(q.read_bytes()).hexdigest(),'exit':value['exit'],'stop_confirmed':value['stop_confirmed'],'retained_failure_evidence':value['retained_failure_evidence']})
if not record['resource_receipts'] or any(not r['stop_confirmed'] or r['exit'] != status for r in record['resource_receipts']):
record['receipt_error'] = 'missing or inconsistent outer resource receipt'
status = 2
record['exit'] = status
record_path.write_text(json.dumps(record, indent=2)+'\n')
print(json.dumps(record, indent=2))
if status: print(log_path.read_text()[-8000:])
raise SystemExit(status)