Experiment formal custody native registration source plan 02 - kmosoti/FabricO11y GitHub Wiki
NOT RUN. Plan01 and its43-method prediction are superseded without an Olo execution. Wren's independent static-review probe is retained on Pip's side and is not an owner-contained receipt. This new request admits only the exact source command below at clean df13db3e975879b96d24c75fc2dffb70d00afd29, tree 79ff7dd51666fcf3c370069165010859ad217c33. No native VM, producer, browser, package, compile or candidate acceptance is requested. Source is held until review and the actual red is archived.
C-101 F1 is addressed by complete base→HEAD ancestry, both original bf parent/tree and all five changed paths/modes/bytes, including CURRENT and the existing release protocol. F2–F6 literal controls and native-shaped diagnostic seq1/before/both-cursor fixtures are registered before correction. The independent query oracle has an explicit immutable SHA control; no oracle caller scope is widened. Reader correction ceiling is separately reduced to+80/497total; production remains byte-identical to77d32ec. Later producer/caller duties move to the separately pinned prospective card and remain mandatory/unrun.
Static prediction:60methods,35positive,8failing methods with15 assertion/subtest failures,17explicit dependent SKIP, exit1. The first native-root error must be exactly native reader shape unavailable (qualification harness, not product): unexpected or failed Spool input. Three container, four schema-URL, two old-query identity, four common Resource-key/Scope-name and one gap-visibility refusal failures account for the other14entries. Import/setup/errors/timeouts do not establish those registered reds. Actual counts and messages must be recorded rather than inferred from this prediction. No later production fix may precede the published actual red.
The unchanged guarded driver explicitly sets and receipts DATA storage root before tools/resource_group.py. Fresh strict whole-DATA namespace census +1GiB full source allowance must fit95e9 admission under100GB ceiling; source clean/frozen and no other project service required. Default16GiBhigh/20GiBmax/noSwap/1800s; serial only. Preserve actual inner/log/outer command/resource/exit/confirmed stop/temporary cleanup receipts and all meaningful failures. Read#59 before/after. No source green authenticates real guest ownership/stop or completes QUAL-INSTALL/QUAL-H; first genuine native mapping is queued separately.
python3 -B /run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01/run-step-custody-native-registration-02.py custody-native-registration-red-02 1073741824 env PYTHONPATH=tools/qualification/install:tools/qualification:tools python3 -B -m unittest discover -v -s tools/qualification/install -p 'test_*.py'SHA-256 87c033a0738ffbc15726fe58226fca71ec7e4a925ac3b7051fa29b88eda39f1e, 1789 bytes.
{
"head": "df13db3e975879b96d24c75fc2dffb70d00afd29",
"tree": "79ff7dd51666fcf3c370069165010859ad217c33",
"status": "",
"all_paths_root_observed": [
{
"path": "docs/CURRENT.md",
"sha256": "e9fdcff8e0c0b6327297a386bfb9160971b49a733bc34ccad55612c537abe158",
"bytes": 21390,
"mode": "100644"
},
{
"path": "docs/experiments/formal/installation-native-custody-caller-card.md",
"sha256": "ac08f57cbb9535a201490ccc8bf6baa2bd6b535ed9f47dd68cdfbe2cf8116ad3",
"bytes": 7648,
"mode": "100644"
},
{
"path": "docs/experiments/formal/installation-native-custody-protocol.md",
"sha256": "fa7d55c1981edae3bbdd41e7d0babbfcdcbfc36f1e0e8d57d45521da26b7a2c4",
"bytes": 17322,
"mode": "100644"
},
{
"path": "docs/experiments/formal/installation-release-protocol.md",
"sha256": "59a028727c20d518f0075e1dcea334f1ff5542d8c46e2a73e6d1c4649b4f147d",
"bytes": 25271,
"mode": "100644"
},
{
"path": "tools/qualification/install/test_custody_native.py",
"sha256": "db7c96fc572b3825e9e327e6b8cf86d74e058b4d3cb6a4d1d1ae292f76db037f",
"bytes": 31873,
"mode": "100644"
}
],
"helper_metadata_sha256": "bc1b35fec2caed9a6cffb5efa4ca48c766cfa0cfe3c9a136f9e95e13bacb0040",
"previous_bf_parent_and_tree": [
"77d32ece01646f39d97e98b942fa7c776463a19c",
"20a4964343232ae44c4ecac86a2765fcb7064709"
],
"complete_base_diff_name_status": [
"M\tdocs/CURRENT.md",
"A\tdocs/experiments/formal/installation-native-custody-caller-card.md",
"A\tdocs/experiments/formal/installation-native-custody-protocol.md",
"M\tdocs/experiments/formal/installation-release-protocol.md",
"A\ttools/qualification/install/test_custody_native.py"
]
}
SHA-256 bc1b35fec2caed9a6cffb5efa4ca48c766cfa0cfe3c9a136f9e95e13bacb0040, 9300 bytes.
{
"kind": "prospective-source-metadata-not-execution",
"head": "df13db3e975879b96d24c75fc2dffb70d00afd29",
"tree": "79ff7dd51666fcf3c370069165010859ad217c33",
"base": "77d32ece01646f39d97e98b942fa7c776463a19c",
"ancestry": [
{
"commit": "df13db3e975879b96d24c75fc2dffb70d00afd29",
"parents": "a273fd21cbea2bf07f8b1e623b7d5491655ff4a3"
},
{
"commit": "a273fd21cbea2bf07f8b1e623b7d5491655ff4a3",
"parents": "bf4320efcf0da09cea494cf0a09a23b74ba228f0"
},
{
"commit": "bf4320efcf0da09cea494cf0a09a23b74ba228f0",
"parents": "77d32ece01646f39d97e98b942fa7c776463a19c"
},
{
"commit": "77d32ece01646f39d97e98b942fa7c776463a19c",
"parents": "158a5ec1129e6a9789b3d4271cea0020fecd6ff6"
}
],
"changed_paths": [
{
"path": "docs/CURRENT.md",
"status": "M",
"mode": "100644",
"bytes": 21390,
"sha256": "e9fdcff8e0c0b6327297a386bfb9160971b49a733bc34ccad55612c537abe158"
},
{
"path": "docs/experiments/formal/installation-native-custody-caller-card.md",
"status": "A",
"mode": "100644",
"bytes": 7648,
"sha256": "ac08f57cbb9535a201490ccc8bf6baa2bd6b535ed9f47dd68cdfbe2cf8116ad3"
},
{
"path": "docs/experiments/formal/installation-native-custody-protocol.md",
"status": "A",
"mode": "100644",
"bytes": 17322,
"sha256": "fa7d55c1981edae3bbdd41e7d0babbfcdcbfc36f1e0e8d57d45521da26b7a2c4"
},
{
"path": "docs/experiments/formal/installation-release-protocol.md",
"status": "M",
"mode": "100644",
"bytes": 25271,
"sha256": "59a028727c20d518f0075e1dcea334f1ff5542d8c46e2a73e6d1c4649b4f147d"
},
{
"path": "tools/qualification/install/test_custody_native.py",
"status": "A",
"mode": "100644",
"bytes": 31873,
"sha256": "db7c96fc572b3825e9e327e6b8cf86d74e058b4d3cb6a4d1d1ae292f76db037f"
}
],
"static_method_definitions": {
"tools/qualification/install/test_a12_memory_pressure.py": {
"EffectiveLimitTests": [
"test_missing_root_controls_are_unlimited_and_slice_caps_inherit",
"test_tighter_leaf_limit_is_not_hidden_by_slice_limit"
]
},
"tools/qualification/install/test_continue_qemu.py": {
"ContinuationProvenanceTests": [
"test_only_bound_reboot_observer_failure_is_eligible"
]
},
"tools/qualification/install/test_custody_native.py": {
"NativeFixture": [],
"NativeShapeAvailability": [
"test_unfiltered_native_root_and_explicit_fixture_projection_are_supported"
],
"LiteralNativeRecords": [
"test_mixed_record_indices_and_original_payload_remain_literal",
"test_query_oracle_stays_byte_identical",
"test_collection_gap_remains_visible_in_decoded_batch",
"test_malformed_diagnostic_row_cannot_hide_behind_fixture_selection",
"test_resource_metrics_resource_must_be_strictly_decoded",
"test_resource_spans_resource_must_be_strictly_decoded",
"test_scope_metrics_scope_must_be_strictly_decoded",
"test_schema_url_utf8_is_required_at_every_common_placement",
"test_common_resource_keys_and_scope_names_are_utf8",
"test_legacy_mode_still_refuses_a_diagnostics_directory",
"test_extra_allowed_path_row_in_pending_batch_is_refused",
"test_predecessor_query_rows_cannot_substitute_label_or_strand"
],
"NativeShapeControls": [
"test_crc_valid_malformed_diagnostic_frame_is_rejected_before_projection",
"test_append_and_uncollected_stop_tail_preserve_full_authority",
"test_source_path_not_body_selects_the_five_fixture_rows",
"test_one_observed_rotation_preserves_prior_archived_inventory",
"test_rewritten_diagnostic_prefix_is_refused",
"test_unjustified_or_unobserved_rotation_cannot_replace_prior_prefix",
"test_unchanged_old_diagnostic_file_is_preserved_without_rotation",
"test_every_pending_raw_log_row_is_in_both_complete_query_answers",
"test_extra_allowed_path_row_in_native_pending_batch_is_refused",
"test_fixture_source_gap_in_pending_or_unread_batch_is_refused",
"test_legacy_frame_and_failure_markers_are_also_refused_in_native_mode",
"test_native_root_requires_private_diagnostic_directory_and_active_file",
"test_diagnostic_archive_must_match_actual_in_root_bytes_and_identity",
"test_diagnostic_only_pending_batch_still_needs_original_byte_ack",
"test_fixture_prefix_gap_and_extra_fixture_query_row_are_refused",
"test_invalid_or_substituted_fixture_path_registration_is_refused",
"test_unexpected_aliased_or_nonprivate_diagnostic_input_is_refused"
]
},
"tools/qualification/install/test_custody_witness.py": {
"MechanismAvailability": [
"test_stopped_snapshot_and_conservation_are_available"
],
"CustodyWitnessControls": [
"test_clean_raw_snapshot_and_two_line_conservation",
"test_archived_leading_lost_line_cannot_escape_source_conservation",
"test_complete_frame_and_envelope_refusals",
"test_nested_otlp_structural_refusals",
"test_generic_reader_positive_forms",
"test_generic_reader_structural_refusals",
"test_cursor_inherits_batch_recursion_budget",
"test_btrfs_inherits_cursor_recursion_budget",
"test_packed_numeric_reader_forms_and_refusals",
"test_logs_common_schema_and_utf8_refusals",
"test_logs_common_and_message_depth_boundaries",
"test_metrics_traces_container_scope",
"test_original_payload_log_projection_binding",
"test_full_cursor_prefix_identity_and_sequence_order",
"test_stop_and_path_admission_refusals",
"test_install_ack_query_and_reboot_conservation_refusals"
]
},
"tools/qualification/install/test_run_qemu.py": {
"MutationGradingTests": [
"test_expected_rpm_mutation_cleans_only_confirmed_stopped_scratch",
"test_reboot_witness_runs_after_privilege_transition",
"test_all_registered_named_defects_are_required",
"test_unrelated_failure_is_not_detection",
"test_bad_provenance_or_environment_cannot_pass"
],
"InstallationShellControls": [
"test_lifecycle_remove_rejects_active_services_and_fabric_avc",
"test_lifecycle_remove_rejects_missing_state_and_incomplete_purge",
"test_lifecycle_remove_rejects_unobserved_services_and_unreadable_audit",
"test_acceptance_remove_rejects_either_live_unit",
"test_acceptance_fedora_rejects_unreadable_audit"
]
}
},
"protected_paths": [
{
"path": "tools/qualification/install/test_a12_memory_pressure.py",
"sha256": "4270ee8854978058ec1acfa677cc2e64638f6e677779cd746181abfda670f63d",
"byte_equal_base": true
},
{
"path": "tools/qualification/install/test_continue_qemu.py",
"sha256": "9abfe581c764bf6434ff86e89bf50b08d48eb62f3a27b377ba7091d026a59e32",
"byte_equal_base": true
},
{
"path": "tools/qualification/install/test_custody_witness.py",
"sha256": "fa4782a9ff25e72d91546207681db8e9f96c4b771472bbbcacb60b41f85c4a61",
"byte_equal_base": true
},
{
"path": "tools/qualification/install/test_run_qemu.py",
"sha256": "35bbbc144065a22bd37390aed7325ded8f56b3bdfbee672944fc19292cfba744",
"byte_equal_base": true
},
{
"path": "tools/qualification/install/custody_witness.py",
"sha256": "2c500203cc26a474366d4b9daf13d63332fb6454fc94649a00f8dd43eb796ebf",
"byte_equal_base": true
},
{
"path": "tools/qualification/query_oracle.py",
"sha256": "fc696ee1173093c258ff4451c4a796dd8e5832761f9e0ff49f89a922b411d23a",
"byte_equal_base": true
},
{
"path": "tools/qualification/delivery_oracle.py",
"sha256": "09034939ed48e3c75937adc6087add8964c4971f2c2a3b32b565b0ecefca2fcc",
"byte_equal_base": true
},
{
"path": "tools/qualification/rate_oracle.py",
"sha256": "afb3a535fe95dc99cba444a401730a8d79a69f2aabf1c9b7bf3e5c2724c603c6",
"byte_equal_base": true
}
],
"production_net_lines": 0,
"future_reader_ceiling_net": 80,
"future_reader_max_total": 497,
"predicted_initial": {
"methods": 60,
"positive_methods": 35,
"failing_methods": 8,
"failed_assertion_or_subtest_entries": 15,
"dependent_skips": 17,
"exit": 1,
"status": "NOT RUN",
"failure_entries": {
"native_root": 1,
"common_container": 3,
"schema_url": 4,
"old_query_identity": 2,
"common_resource_key_or_scope_name_utf8": 4,
"collection_gap_visibility": 1
}
},
"inside_command": [
"env",
"PYTHONPATH=tools/qualification/install:tools/qualification:tools",
"python3",
"-B",
"-m",
"unittest",
"discover",
"-v",
"-s",
"tools/qualification/install",
"-p",
"test_*.py"
],
"expected_first_native_root_error": "native reader shape unavailable (qualification harness, not product): unexpected or failed Spool input",
"execution": "NOT RUN: root alone executes admitted contained cells; no project imports/tests/build/native/push by helper"
}
SHA-256 c05ef517ebe075c870d62610e6f64dffefc1d250df8534dd6ad7e972506262f9, 7042 bytes.
import datetime, hashlib, json, os, re, signal, subprocess, sys, time
from pathlib import Path
ROOT = Path('/run/media/kmosoti/data/FabricO11y')
REPO = ROOT / 'scratch/installation-custody-native'
OUT = ROOT / 'results/service-opening-candidate'
STORE = ROOT / 'scratch/service-opening-census'
FROZEN = 'df13db3e975879b96d24c75fc2dffb70d00afd29'
def require(value, reason):
if not value:
raise ValueError(reason)
label, reserve_text, *command = sys.argv[1:]
reserve = int(reserve_text)
delegated = command[:1] == ['--delegate']
if delegated: command = command[1:]
require(re.fullmatch(r'[a-z0-9-]{1,80}', label) and reserve > 0 and command, 'ordinary workload admission guard')
record_path = OUT / (label + '.json')
require(not record_path.exists(), 'unique run label required')
def census():
# Rootless container files can belong to subordinate UIDs. Inspect all of
# them in the same user namespace; never skip unreadable directories.
require(not STORE.is_symlink(), 'ordinary workload admission guard')
require((STORE / '.owned-census-store').read_text() == 'service-opening-census-v1\n', 'ordinary workload admission guard')
args = ['podman', '--root', str(STORE/'graph'), '--runroot', str(STORE/'run'),
'--tmpdir', str(STORE/'tmp'), '--storage-driver', 'vfs',
'unshare', 'python3', '-B', str(OUT/'census.py')]
env = dict(os.environ, TMPDIR=str(STORE/'tmp'), TMP=str(STORE/'tmp'), TEMP=str(STORE/'tmp'))
result = subprocess.run(args, env=env, text=True, capture_output=True, timeout=25)
if result.returncode:
raise RuntimeError(f'strict namespace census failed ({result.returncode}): {result.stderr}')
observed = json.loads(result.stdout)
require(observed['uid'] == 0 and observed['gid'] == 0, 'ordinary workload admission guard')
require(type(observed['data_bytes']) is int and observed['data_bytes'] >= 0, 'ordinary workload admission guard')
return observed
def git(*args):
return subprocess.check_output(['git', *args], cwd=REPO, text=True).strip()
require(git('rev-parse', 'HEAD') == FROZEN and not git('status', '--porcelain'), 'clean frozen source required')
require(not subprocess.check_output(['systemctl', '--user', 'list-units', 'fabric-work-*', '--state=running', '--plain', '--no-legend'], text=True).strip(), 'another project workload is active')
initial = census()
used = initial['data_bytes']
record = {'label': label, 'command': command, 'delegated': delegated, 'frozen': None, 'scope': 'harness controls only; no candidate acceptance evidence', 'harness_revision': FROZEN, 'source_status_before': git('status', '--porcelain'), 'storage_root': str(ROOT), 'admission_bytes': used, 'admission_observation': initial, 'reserved_bytes': reserve, 'observations': [], 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat()}
if used + reserve > 95_000_000_000:
record.update(state='not_run', reason='95 GB storage admission threshold')
record_path.write_text(json.dumps(record, indent=2)+'\n')
raise SystemExit('storage admission denied; project command not launched')
receipts = REPO / 'target/resource-containment/runs'
before = set(receipts.glob('*.json'))
env = dict(os.environ, FABRIC_STORAGE_ROOT=str(ROOT), CARGO_HOME=str(ROOT/'toolchain-cache/cargo'), RUSTUP_HOME=str(ROOT/'toolchain-cache/rustup'), CARGO_BUILD_JOBS='2')
env['PATH'] = str(ROOT/'tools/candidate-dpkg/bin') + ':' + env['PATH']
log_path = OUT / (label + '.log')
record.update(state='running', log=str(log_path))
record_path.write_text(json.dumps(record, indent=2)+'\n')
def find_unit():
with log_path.open() as stream:
prefix = stream.read(65536)
match = re.search(r'Running as unit: (fabric-work-[0-9a-f]+)\.service', prefix)
return match.group(1) if match else None
def stop_owned(process):
unit = find_unit()
if unit:
stopped = subprocess.run(['systemctl', '--user', 'stop', unit],
text=True, capture_output=True, timeout=25)
record['stop'] = {'unit': unit, 'exit': stopped.returncode, 'stderr': stopped.stderr}
if stopped.returncode:
# Interrupt the launcher too; its finally block independently
# confirms that this exact service has no surviving descendants.
process.send_signal(signal.SIGINT)
else:
process.send_signal(signal.SIGINT)
record['stop'] = {'unit': None, 'launcher_interrupted': True}
with log_path.open('wb') as log:
launcher = ['python3', '-B', 'tools/resource_group.py', *(['--delegate'] if delegated else []), '--', *command]
process = subprocess.Popen(launcher, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT)
next_census = time.monotonic() + 30
try:
while process.poll() is None:
time.sleep(1)
if time.monotonic() >= next_census:
observed = census()
record['observations'].append({'unix':time.time(), **observed})
record_path.write_text(json.dumps(record, indent=2)+'\n')
if observed['data_bytes'] >= 95_000_000_000:
record['interruption'] = '95 GB threshold reached'
break
next_census = time.monotonic() + 30
except BaseException as error:
record['interruption'] = f'{type(error).__name__}: {error}'
finally:
if record.get('interruption') and process.poll() is None:
try:
stop_owned(process)
except BaseException as error:
record['stop_error'] = f'{type(error).__name__}: {error}'
process.send_signal(signal.SIGINT)
status = process.wait()
record['launcher_exit'] = status
if record.get('interruption'): status = 2
with log_path.open('rb') as stream:
log_digest = hashlib.file_digest(stream, 'sha256').hexdigest()
record.update(state='interrupted' if record.get('interruption') else 'completed', exit=status, source_status_after=git('status', '--porcelain'), finished_utc=datetime.datetime.now(datetime.timezone.utc).isoformat(), log_sha256=log_digest, resource_receipts=[])
archive = OUT/'resource-containment'; archive.mkdir(exist_ok=True)
for p in sorted(set(receipts.glob('*.json')) - before):
value=json.loads(p.read_text())
if value['command'] != command: continue
q=archive/p.name; q.write_bytes(p.read_bytes())
record['resource_receipts'].append({'path':str(q),'sha256':hashlib.sha256(q.read_bytes()).hexdigest(),'exit':value['exit'],'stop_confirmed':value['stop_confirmed'],'retained_failure_evidence':value['retained_failure_evidence']})
if not record['resource_receipts'] or any(not r['stop_confirmed'] or r['exit'] != status for r in record['resource_receipts']):
record['receipt_error'] = 'missing or inconsistent outer resource receipt'
status = 2
record['exit'] = status
record_path.write_text(json.dumps(record, indent=2)+'\n')
print(json.dumps(record, indent=2))
if status: print(log_path.read_text()[-8000:])
raise SystemExit(status)SHA-256 e9fdcff8e0c0b6327297a386bfb9160971b49a733bc34ccad55612c537abe158, 21390 bytes.
# Current project state
## Implemented
FabricO11y collects, durably forwards, retains and queries Linux telemetry. It is
usable for bounded development trials; no current deployment profile is qualified
and no release tag has been produced.
The [product contract](PRODUCT-CONTRACT.md) defines the promises and the
[verification matrix](formal/verification-matrix.md) maps each invariant to checks
and remaining limits.
| Component | Current behavior |
| --- | --- |
| Spindle (`fabric-node`) | Host metrics, selected log files and loopback OTLP/HTTP trace intake; durable FAB1 Spool, exact retry, visible collection gaps, output-rate cap and last-valid remote configuration. |
| Delivery | Verified TLS (the configured CA and the dialed name only), per-Spindle credentials, ordered Batches, deduplication and ACK after grouped two-sync journal commit. A quiet sender closes its group after 2 ms. One rule interprets every server answer; an ACK beyond the Spool's durable sequence is refused. |
| Storage | Bounded external-merge sealing to Zstd Parquet, manifest publication before checkpoint/reclaim, retention by age and bytes. The normal writer uses aligned sorted groups and disk-backed completed pages; estimated input targets admit a single oversized row. No whole-server memory bound is established. A journal group the stream table cannot absorb quarantines the commit thread (in memory; a restart replays). |
| Query | Logs, metric points/rates and spans across journal and Segments, snapshot pagination, freshness and completeness. Legacy queries default to Scan. Console queries (owner and reader, all telemetry, at most 1,000 rows and 24 hours) use Walk with storage-owned source maps; the now-redundant scoped-filtering code remains. Unpaginated rates stay outside the console API. |
| Console and access | Leptos/WASM PWA with a local owner passkey, expiring read-only reader tokens for scripts and AI clients, and Spindle tokens: three fixed roles ([ADR-0029](decisions/ADR-0029-use-three-fixed-roles-for-the-first-release.md)) at one route guard. Normal `serve` requires an HTTPS origin and RP ID, needs no administrator token and refuses `/v1/admin/*`; `serve-legacy` is migration only. Enrollment requires resident credentials and user verification; finite Chrome checks include independent credential inventory. The exact-package and physical-device release matrix remains open. |
| Hardening | Console peer admission (including an IPv6 /48 cap), 503 on a poisoned lock, absolute OTLP deadlines, no-follow descriptor-relative log opening, one credential-file policy, audit denials capped apart from other events and eight added unit sandbox directives; see [security hardening](architecture/security-hardening.md). |
| Self-observation | Every production server CLI launches and supervises a dedicated local Spindle. Both emit bounded diagnostics through ordinary Spool/TLS/ACK delivery. Edge Spindles retain their configured destination. |
| Packaging | Debian and RPM artifacts include the console, Apache-2.0 license and dependency notices, with static systemd services and a shared slice. Exact frozen `a5dbda7a` packages passed fresh Debian/Fedora baseline and upgrade/reboot/removal cells, with six named mutation controls detected; broader release acceptance remains open. These cells establish normal service operation under the added unit directives; their individual adversarial enforcement remains unverified. |
| Architecture | Pure `no_std` semantic core, effect ports, application transitions and adapters; executable layer and purity gates. `ports` and `app` may link no external crate, and a requirement satisfied by `[patch]` counts as the member edge. |
| Repository | Historical tooling, experiment data and the FOL2 demonstration live in the [research archive](https://github.com/kmosoti/FabricO11y-research); dead Rust and experiment selectors are removed. The launcher's storage root is `FABRIC_STORAGE_ROOT` with host-scaled memory caps. Workflow actions are pinned by SHA, the fuzz nightly by date, and `tools/ci/check-vendor.py` compares vendored crates with crates.io. |
See the [system view](architecture/system.md), [operations guide](operations.md)
and [architecture index](architecture/README.md) for behavior and configuration.
## Release scope and open work
The release epic ([#33](https://github.com/kmosoti/FabricO11y/issues/33)) tracks the
remaining work. Owner decisions ([#34](https://github.com/kmosoti/FabricO11y/issues/34)):
tag `v0.1.0-alpha.1`; Debian 13 and Fedora 44 (pinned at the freeze on 2026-10-12, before Fedora 45
shipped); physical-device and mobile passkey checks are deferred as a known
issue; reader queries are not individually audited. The
[release plan](milestones/release-readiness.md) covers Debian/Fedora central and edge
deployment, 100 GB retained telemetry with separate working space, local passkeys,
owner and reader access, and the PWA; Apache-2.0 licensing, dependency notices and a
[private security-reporting route](../SECURITY.md) are included. No new
qualification is claimed here: repository CI receipts are integration evidence, not
package, device or deployment acceptance.
Known limits and open checks:
- The [fixed-query caller prerequisites](experiments/formal/query-caller-run-01.md) retain an unpaced-page HTTP 429 and the obsolete fixture-only oracle corpus. The repaired disposable query smoke exited 0 with all eight gates; four full query successors remain pending. Six main cells retain their original `8484493` results under prospectively selected `clearing-v2`; older sampled-v1 edge diagnostics failed in five cells. No broader qualification follows.
- An attacker spread over many IPv6 /48s or IPv4 addresses can still fill the 1,024-entry peer table; a denial at a new second costs up to two access-state syncs (clock watermark and denial record), though at most one denial is stored per second ([#47](https://github.com/kmosoti/FabricO11y/issues/47)).
- Exact shipped units ran successfully in fresh Debian 13/Fedora 44 baseline and lifecycle cells ([#41](https://github.com/kmosoti/FabricO11y/issues/41)); independent adversarial probes of every added hardening directive remain outside that finite installation result.
- The [installation verifier controls](experiments/formal/installation-release-protocol.md#prospective-removal-checker-counterexample-controls) require both observed removal states to be inactive or failed with systemctl exit 3 and reject unreadable Fedora audit evidence. Revision-bound source-only execution evidence belongs to the [checker results wiki](https://github.com/kmosoti/FabricO11y/wiki/Experiment-formal-installation-checker-prerequisite-results-01); guest acceptance remains separate.
- The [stopped-custody upgrade witness](experiments/formal/installation-release-protocol.md#prospective-stopped-custody-upgrade-witness) registers independent framing/Cursor and exact pending/unread-source conservation controls. Its raw snapshot/conservation adapter is implemented; verification and native integration remain pending. No new package lifecycle outcome is recorded.
- The prospective [native custody input amendment](experiments/formal/installation-native-custody-protocol.md) registers unfiltered diagnostic-root, append/rotation and explicit path-projection controls plus bounded producer/caller cards. The reader correction and all native cells are NOT RUN; synthetic controls cannot substitute for owner-bound guest observations.
- The browser checks added for the role matrix (`live_browser_check`, bridge smoke, packaged console) are CI-only and have no recorded result.
- In cloud containers the `xtask` tests `check_logs` (needs the launcher) and `evidence` (needs full git history) fail, and `tests/control.rs` fails under umask 022; these are environmental.
## Results that shape the design
Measurements below belong to their linked workloads and revisions. Heap, RSS and
cgroup memory are different quantities; isolated gains cannot be multiplied into
a service-capacity claim.
| Investigation | Result | Consequence |
| --- | --- | --- |
| [Bounded sealer](experiments/benchmarks/ingestion-memory-run-01.md), [encoded-page correction](experiments/formal/encoded-page-memory-run-01.md), [readiness continuation](experiments/formal/readiness-continuation-results.md) | Combined aligned-input/disk-PageStore opt-in passed eight cells with three pairs each: 37.3 MiB steady256 heap and 39.5 MiB bigrows64; high-entropy probe 40.07 MiB. Supplemental row/page scratch observations completed. Seven cells were 1.7–11.9% slower; bigrows64 was 2.7% faster on the shared host. | Adopted as defaults, with unflagged recovery tests, loaded-binary equivalence and all 17 fast checks passing. The companion-inclusive full soak passed all ten gates. |
| [Walk](experiments/benchmarks/query-walk-run-01.md), [tail blocks](experiments/benchmarks/block-tail-run-01.md), [text filters](experiments/benchmarks/text-filter-run-01.md) | Large selective-query wins, including a 614→28 ms tail shape. | Keep hierarchical pruning and delayed payload work. Scan remains default pending broader acceptance. |
| [Fixed-demand plans](experiments/benchmarks/query-plan-run-01.md) | Three small-profile pairs favored Walk: median 42.2% lower server CPU and 64.1% lower sampled phase-peak RSS. | Workload-specific evidence; no universal plan winner. |
| [Service/recovery](experiments/benchmarks/service-recovery-research-findings.md) | Two 20-Spindle cases each recovered 60,000 exact logs; Scan/Walk peak RSS 165.7/92.9 MiB. Eighteen selected storage fault/recovery cases passed. | Short bounded service behavior is supported; long-term stability is still open. |
| [Pressure and remote edges](experiments/benchmarks/hammer-reference-findings.md) | Four repaired cells each recovered 1.26 million exact logs. Final Walk peak RSS 285.61 MiB. Real remote collectors and the eight-worker simulator delivered exactly. | Journal identity races were fixed. One missing outer receipt prevents strict campaign closeout; the two-worker simulator retained backlog and misreported timing. |
| [Synthetic RCA](experiments/benchmarks/rca-journal-findings.md) | Fourteen journal/restart cells, 350 exact complete chains and 30 rejected corruptions. | Logs, metrics and traces support tested investigations; published-storage RCA, real application exporters and causal interpretation remain unmeasured. |
| [Self-observation](experiments/formal/server-self-observation-findings.md) | Native TLS delivery, restart identity, source pinning, refusal cases and process cleanup checked under a 4 GB combined cap. Tiny debug fixture: 19.9/8.5 MiB server/Spindle peak RSS. | Dedicated Spindle is implemented. Diagnostic files before Spool commit remain best-effort; colocation cannot observe total host loss. |
| [Cross-system trials](experiments/benchmarks/cross-system-sweep-findings.md), [continuation](experiments/benchmarks/cross-system-continuation-findings.md), [native candidates](experiments/benchmarks/native-frontier-findings.md) | Smaller sort runs and borrowed rows reduced allocations in selected fixtures. Larger-input models, timing guards and one RSS guard failed. | Retain the mechanisms and counterexamples; no storage-engine migration, allocator tuning or rejected candidate became a default. |
The [experiment index](experiments/README.md) retains protocols and failures.
The [research synthesis](research/cross-system-source-synthesis.md) and
[coupled model review](research/coupled-performance-review.md) separate upstream ideas, measurements and hypotheses.
## Invariants and reliability
Delivery, Spool, control, history/query, architecture, sealer, observation encoding
and query semantics are tracked in the [verification matrix](formal/verification-matrix.md).
The independent Python delivery/query/rate oracles remain the semantic reference.
Properties, corpus replay, simulation and negative controls supplement them.
The continuation records eleven bounded Kani proofs, named mutants, transport
TLC and four 60-second fuzz targets with 11,370,454 executions without a crash;
finite checks do not prove every execution. The [invariant audit](experiments/formal/invariant-consolidation.md)
retains minimized launcher, control-publication and Spool-exhaustion defects.
Recent repairs cover journal identity during rotation, publication/reclaim
coverage races, snapshot metadata after publication, HTTP admission before body
allocation, cancellation and startup ownership leaks, oversized-line progress,
exact integer counter rates, reserved diagnostic credentials, audit eviction by
denial floods, poisoned locks and one IPv6 network filling the peer table. Their
minimized failures and checks are linked from the matrix and experiment index.
Verification fixtures check finite pagination progress before complete drain, and
interrupted registry checks retain partial output without a completion receipt
([verifier protocol](experiments/formal/verifier-boundedness-protocol.md)).
## Candidate evidence to date
These results belong to the candidates they name; none qualifies the release.
- **Console reproducibility.** Separate console builds of one commit differed in four files because ImageMagick stamped time metadata into the generated icons ([#55](https://github.com/kmosoti/FabricO11y/issues/55)). Icons are now generated with `-strip`, and console and package builds refuse a PNG with time or text metadata (`CX-CONSOLE-ICON-TIMESTAMP`). The owner chose to fix this before release, so the `a5dbda7` packages are superseded: a new candidate is frozen and rebuilt from two separate console builds, and its gates rerun. Results on `a5dbda7` packages remain recorded for that candidate only.
- **Service-opening repair.** Frozen `ca84d982` failed Debian lifecycle `rel-deb-life-01`: required `RestrictSUIDSGID=yes` denied `openat2`, and the dedicated Spindle/server exited. The [failure record](https://github.com/kmosoti/FabricO11y/wiki/Experiment-formal-installation-release-run-02) retains exact packages and guest controls. A descriptor-relative `openat` walk replaces that syscall without relaxing hardening or link refusal. Its native child-filter regression failed before the repair (exit 101) and passed after; eleven Linux security tests passed, including pinned-parent replacement and exact error propagation. The [amendment](experiments/formal/service-log-opening-protocol.md) registers the checks. PR #53 merged with six commits preserved. Frozen `a5dbda7a4fd6465c94f6505b3214cbcbc2130c82` was rebuilt reproducibly with matching console assets; matching main Rust/security and identical-tree PR extended evidence passed. [Exact installation run 03](experiments/formal/installation-release-run-03.md) passed four fresh positive cells and detected all six package controls on Debian 13/Fedora 44. Old candidate failures remain failed; this does not qualify deployment or authenticated UI behavior.
- **Readiness continuation.** The [continuation](experiments/formal/readiness-continuation-results.md) completed the combined builder campaign, recovery/shutdown checks, dependency policy and finite [installation acceptance](experiments/formal/installation-acceptance-run-02.md). The [full companion-compatible R2 soak](experiments/benchmarks/soak-run-02.md) passed all ten gates. The [normal writer](experiments/formal/bounded-writer-default-run-01.md) passed 27 unflagged bounded tests, 13 kill cuts plus no-hit, and loaded-binary equivalence; all 17 final fast checks and three manual documentation checks passed on that revision.
- **Packages and console.** Debian build 04 and RPM build 03 contain console build 09, with all 637 common payload files matching. The console fixed a real 200% zoom pagination overflow; 150 staged Chrome assertions passed, including rejection of an injected overflow. A later change requests resident credentials for all four enrollment paths (two focused regressions passed on `829babe`; independent inventory checks and a nonresident-enrollment control are implemented). New exact packages and their browser acceptance are pending. Virtual devices do not establish physical passkey or mobile compatibility; Firefox's virtual-passkey diagnostic completed 127 assertions after two fixture repairs, and standalone resident-key acceptance is pending.
- **Query at scale.** `release-query-segment-seed1-successor-02` passed all eight gates on Debian build 04/RPM build 03: 200/200 independent-oracle answers, exact recovery and companion custody, query p99 at most 504.340 ms before restart and 447.509 ms after. It is not attributed to the resident-enrollment successor; the earlier 10.27-second five-page failure remains in the [cache report](https://github.com/kmosoti/FabricO11y/wiki/Experiment-benchmarks-scoped-evidence-cache-run-01).
- **Failed and open.** Full soak `production-soak-full-03` failed after 1,587.618 seconds: its storage census could not traverse directories owned by a concurrent package build, and the harness stopped rather than ignore uncounted disk use. No Fabric crash was observed. The failed and interrupted runs keep their results in the [campaign record](https://github.com/kmosoti/FabricO11y/wiki/Experiment-benchmarks-production-access-campaign-run-01). The replacement exact-package installation/lifecycle matrix is complete. The first #42 disposable smoke failed before measurement on enrollment HTTP 429: each bridge command made two protected requests but was paced at 0.15 seconds. [Command admission run 01](experiments/formal/console-command-admission-run-01.md) records the independent red schedule regression, 50 passing harness tests and a passing repaired native smoke on unchanged packages. The separate harness change retains all session checks/refusals and decision rules; the full main/query matrix, pressure/outages, cross-host cells, fresh full soak and packaged browser/PWA acceptance remain queued. Historical Fedora evidence belongs to its original `c06688c` candidate, including the [Btrfs identity repair](decisions/ADR-0028-preserve-btrfs-log-identity-across-reboots.md).
- **Hosted and local gates.** The hosted `eb5b096` candidate passed fast and focused security workflows and all ten extended commands, but final receipt completeness failed after fuzzing rewrote its stale dependency lock; the lock now matches and fuzz trials require unchanged lock bytes. The local `851d50b` delivery gate exited 0 (eight trials, unchanged oracle, strict loss control rejected) through the explicit legacy adapter, and a finite Chrome trial completed 144 assertions on debug binaries; neither qualifies packages or devices. The [integration review](https://github.com/kmosoti/FabricO11y/wiki/Release-integration-review) holds commands, candidate identities and cleanup observations. Bun CI/hooks remain disabled; documentation checks run manually.
The [research wiki](https://github.com/kmosoti/FabricO11y/wiki) owns research and
result reports; its [release progress page](https://github.com/kmosoti/FabricO11y/wiki/Release-readiness-progress)
links the acceptance queue. Product/operator documentation, architecture, registered
protocols and executable evidence inputs stay here under the
[ownership policy](documentation-policy.md#canonical-ownership); the
[migration manifest](wiki-migration.json) pins 151 migrated reports.
## Assumptions and risks
- Durability assumes successful sync calls are honored by the filesystem/device;
physical power loss has not been tested.
- Exact-byte identity uses SHA-256; collisions are assumed infeasible.
- Optional pruning structures require sound producer metadata; corrupt optional
indexes fall back to exact reads. Silent readable Parquet corruption is not
universally detected by per-query whole-file hashing.
- Layer checks enforce crate boundaries, not every module or runtime interaction.
- Earlier standalone-server benchmarks predate the dedicated Spindle; new service
measurements must account for its work and diagnostic traffic.
- Query-tail decoding, retained metadata, worker concurrency and allocator
retention can dominate process memory despite bounded sealer payload buffers.
- The historical standalone soak failed its RSS-growth gate; the passing R2
trial includes the companion and is not a causal single-variable comparison.
- A pre-containment verification process caused a global workstation OOM on
2026-10-04; its allocation mechanism remains unverified.
- Same-host producers/harnesses compete with the server and can distort latency.
Remote delivery and synthetic RCA are not deployment qualification.
Broader qualification work follows the [capability ledger](QUALIFICATION.md#capability-ledger):
default-scale retention and exporter pressure, published-storage RCA and real
applications, and reconciliation of target-profile delivery evidence. Application
adapters remain [proposed use cases](research/application-use-cases.md).
## Resource envelope
All builds, tests and experiments use the [resource launcher](CONTRIBUTING.md#resource-containment)
and its storage root `FABRIC_STORAGE_ROOT`. The laboratory ceiling is the smaller of
20 GiB and 75% of RAM with no swap; the server experiment cap is 4 GB and the total
storage ceiling 100 GB. Remote hosts use their own smaller verified budgets. Preserve
failure evidence and remove owned scratch.
scratch/installation-custody-native/docs/experiments/formal/installation-native-custody-caller-card.md
SHA-256 ac08f57cbb9535a201490ccc8bf6baa2bd6b535ed9f47dd68cdfbe2cf8116ad3, 7648 bytes.
# Native installation custody producer and caller card
Status: prospective qualification-only follow-up, 2026-10-13; NOT RUN.
This card preserves the later obligations separated from the
[native reader registration](installation-native-custody-protocol.md) under C-101.
No producer/caller implementation, native command or result is admitted here.
Source-only controls do not authenticate a guest stop, service identity or transfer.
An exact source preflight and separate native producer/caller admission remain required.
## Prospective production ceilings
| Card / owned production paths | Maximum net addition | Reason |
| --- | ---: | --- |
| Guest producer: new `tools/qualification/install/custody_native.py` | +180 | Fixed-path stopped capture, manifest, complete raw roots, source archives/query pages and candidate process/journal joins. |
| Bridge/admission/caller: `tools/qualification/install/run-qemu.py`, `lifecycle.sh`, and the same `custody_native.py` | +220 combined, separately counted from producer | Expected #40 tuple/transfer binding, bounded evidence transfer, lifecycle entry/exit and unchanged-reader invocation. |
These are ceilings, not targets or observed deltas. Together with the separately
registered +80 reader ceiling, the combined ceiling is +480. No generic caller
API or optional raw-server reread is required by the source registration.
Minimize each actual diff and preserve all earlier controls and independent oracles.
## Native producer, admission and caller obligations
The guest producer reads `/var/lib/fabrico11y/node/spool` in place after stopping
both installed units and every server-owned companion. It retains raw systemctl
query exit/state/MainPID, actual ControlGroup membership and independent process
inventory, boot ID, service UID/GID and stable descriptor/stat inventories.
No literal `stop_confirmed`, `cgroup_empty` or successful exit bit is accepted
without its independently captured, expected-bound raw observation. Any live or
unreadable writer observation refuses capture before source/archive collection.
The producer's fixed guest read-only namespace is `/var/lib`: source archives
and receipts go under a newly exclusive, root-owned 0700
`/var/lib/fabric-custody-fixture/<run_id>/`, with 0600 files. Only that disposable
QEMU guest may set `FABRIC_SCRATCH_ROOT=/var/lib` for reading the two exact
registered roots; host commands always retain their DATA-owned scratch root.
The caller must establish the guest/run/boot/configured-root tuple and exact
allowlist before invoking the reader. This namespace variable authenticates
neither the guest nor any path. The actual Spool stays service-owned; the producer
does not chown, link or relocate it. Export complete raw trees as explicitly
labelled archives after reading the actual root, never as replacement roots.
Capture an unchanged stopped pre-install checkpoint and a separate stopped
post-install checkpoint; record all original guest paths, metadata, raw hashes,
source-prefix bytes/cursor CRCs and observer command/exits. During the bounded
server outage, actually collect the predecessor pending line, then stop the node
and append a distinct unread line. The former must occur in a committed raw
Batch after the durable ACK cursor; the latter must be an adjacent unread source
suffix. Candidate install must preserve that checkpoint. After candidate restart,
capture every query page without body filtering, full ACK journal JSON and an
independently observed node `/proc/PID/exe` hash, PID/start identity, service
UID/cgroup/boot and calibrated run window. ACKs must join that exact sender and
window, every required pending payload hash, Strand and durable ACK cursor.
Observe changed reboot identity and exact projected rows through the existing L2;
unchanged historical lifecycle/removal gates stay required.
On the host, use a new exclusive DATA scratch/output pair per cell. Before boot
and again before admission, freeze and validate the owner-admitted #40 tuple:
FROZEN `8fb6f413179b094288316e25845c137440ba23fa`, package/source/console,
canonical helper manifest and binaries, exact clean merged harness/source/helper
hashes, producer manifest, raw transfer hashes and actual stopped observations.
No guessed main revision or claimed boolean replaces the frozen owner record.
Canonical package source is `73ff77867d10ef521ca985b094e1d177f7786b77`.
| Immutable owner input under `/run/media/kmosoti/data/FabricO11y` | Expected SHA-256 |
| --- | --- |
| `results/installation-package-build/fresh-8fb6f413179b-01-deb-a/receipt.json` | `b10a4e4c2d83d8dbed7c1d6e74673fd377a83123fc1bce31fffc97e034beb9af` |
| Canonical candidate DEB named by that receipt | `8edfc7709f4b5dc3da01c00a1d8c442470298e96f8c8e502cfb14e8c9e713e92` |
| `results/packages/fresh-8fb6f413179b-01-rpm-a/receipt.json` | `eadbbd217804efdeae217fe04d4800c9482cc23650942860df4d756e45ac8434` |
| Canonical candidate RPM named by that receipt | `a54f55e7bf1b2230e026b49d6b86023778a9648ae49b11c6bcb0d558db34a840` |
Read the actual canonical #40 provenance and helper receipts to complete the
expected tuple before the native preflight; this table is not permission to
omit those joins. Output owns only
`DATA/results/installation-native-custody/<run_id>/` and exclusive scratch under
`DATA/scratch/installation-native-custody-<run_id>/`. Retain a manifest mapping
each guest source to its archive, original stat/ownership, digest and raw producer
receipt. Credential files retain only original path/stat/hash witnesses; never
export credential contents. Fixture/archive/source authority remains available.
Native cells: one positive Debian and one positive enforcing Fedora lifecycle;
two additional producer refusals on Debian only. Pending-not-produced deliberately
leaves the new line uncollected and must fail the named missing-committed-pending
check after successful raw decoding. Live-writer deliberately keeps the node
active and must fail the named stop observation before checkpointing. A boot,
credential, capture, timeout, import or package setup failure is not detection.
No candidate install starts after either intended producer refusal.
Host controls mutate explicitly labelled stopped-bundle **copies**: original
payload/CRC/ACK identity, source-prefix interval, transfer digest/path/ownership,
producer/package/helper/source mismatch and incomplete stopped observations.
Credential hash/metadata drift uses literal host records plus the positive native
credential join; no extra credential VM. Controls must fail their named admission
or conservation check with otherwise coherent inputs. Never mutate immutable
actual receipts, packages, raw archives or retained historical failures.
Keep the existing one-guest 2-vCPU/6144-MiB/8-GiB disk envelope, 7/8-GiB host
high/max and zero swap; default 1800-second outer deadline and 180-second lifecycle
phase ceiling. Charge every SSH/helper/copy/query against remaining phase/outer
time; the 20-second outage and bounded readiness/drain must fit those ceilings.
Setup/copy overhead is unmeasured and must be measured before native admission;
an overrun is a refusal, never an implicit deadline extension.
Full admission reserves 9663676416 bytes (9 GiB), stays at or below 95e9 including
reserve, and enforces the unchanged 100-GB DATA ceiling. Measure raw/checkpoint
overhead before native admission; do not silently exceed the reserved 1-GiB
overhead. Stop/reap all descendants before cleanup, retain failed raw observations
and archives, and require actual stopped/cleanup receipts. No native command,
image boot, Rust build or producer/caller completion is established here.
SHA-256 fa7d55c1981edae3bbdd41e7d0babbfcdcbfc36f1e0e8d57d45521da26b7a2c4, 17322 bytes.
# Native installation custody input registration
Status: prospective qualification-only amendment, revision 3, 2026-10-13; NOT RUN.
Coordination [C-092](https://github.com/kmosoti/FabricO11y/issues/59#issuecomment-6139100360)
admits this separate registration from `77d32ece01646f39d97e98b942fa7c776463a19c`.
It extends the [stopped-custody witness](installation-release-protocol.md#prospective-stopped-custody-upgrade-witness)
for native shapes without changing its historical controls or results. No
production mechanism, guest result or package acceptance is recorded here.
## Source authority and the shape conflict
The [installation contract](../../PRODUCT-CONTRACT.md#linux-installation-contract),
[deployment view](../../architecture/deployment.md), [ADR-0013](../../decisions/ADR-0013-deliver-batches-in-order-with-bounded-dedup.md)
and [Spool invariants](../../formal/verification-matrix.md#spool) remain authoritative.
The frozen [node](../../../src/bin/fabric-node.rs) opens
`spool/diagnostics/spindle.log` and collects it as a local source before opening
the Spindle. The [installed config](../../../packaging/etc/node.conf.example)
fixes `/var/lib/fabrico11y/node/spool` and the 268435456-byte Spool allowance.
[OperationalLog](../../../crates/fabric-adapter-linux/src/operational_log.rs)
retains active `spindle.log` and optional `spindle.log.1`, each at most 262144
bytes, with single-link regular private files and a private owned directory.
Lines are at most 4096 bytes including their newline. Rotation renames the
active file over the previous old file when the next whole line would overflow
the active bound. A final stop line can remain uncollected before the next launch.
The existing [reader](../../../tools/qualification/install/custody_witness.py)
rejects the diagnostic directory. Its later all-archive equality, pending-row
restriction and all-cursor tiling also cannot admit ordinary diagnostic growth.
This is a qualification input-shape gap, not evidence of a product custody defect.
The retained three old rows plus pending/unread fixture witness remains required.
Diagnostics are best effort before Spool commit; this amendment does not infer
complete diagnostic delivery from a retained file or a frame CRC.
## Reader correction ceiling
The prospective reader card owns only
`tools/qualification/install/custody_witness.py`: maximum +80 net production
lines, at most 497 from the current 417. C-101 supersedes the earlier +120 ceiling
before this registration is executed. Reuse the strict common schemas and existing
frame/ownership checks; one native directory inventory, one shared path/interval
selection and one append/rotation comparison are sufficient responsibilities.
The estimate is not an observed implementation diff. No production correction is
admitted until the exact registration source preflight and actual red are published.
The [separate prospective producer/caller card](installation-native-custody-caller-card.md)
retains native cells, source admission, ownership, transfer and containment duties.
Those later obligations are not completed or authenticated by source fixtures.
## Full raw root and explicit finite projection
Extend `stopped_snapshot(root, stopped_receipt, archived_allowed_logs, *,
fixture_paths=None)` without changing the default legacy behavior. The native
mode is a one-element tuple containing an absolute archived source path; reject
empty, list, duplicate, relative or unarchived registrations. The native caller
must independently require exactly `('/var/log/fabric-accept/allowed.log',)`.
All three snapshots must carry the same `fixture_paths` value. An arbitrary
caller-supplied path is not authority to select a different native fixture.
In native mode the input is the actual configured, stopped, unfiltered guest
Spool root. Require the diagnostic directory with mode 0700, service ownership
and active `spindle.log`; a filtered/rebuilt root without them cannot be native input.
Inventory the whole root, including that required directory; allow only its
active/optional-old names, rejecting links, aliases, unexpected
entries, nonprivate modes, wrong guest ownership and oversized files. Retain
each diagnostic leaf in `files` and `file_observations` using its root-relative
name. Return `diagnostics[name]` with original `bytes`, SHA-256, size, device,
inode, mode, UID, GID and link count, plus the directory observation. Read with
the existing descriptor-relative no-follow and before/after drift guards.
Host synthetic controls check current UID/private modes; the guest caller must
separately establish the actual `fabricolly` UID/GID and private guest paths.
Native receipts carry strict integer `service_identity.uid` and `.gid`, and the
reader compares diagnostic ownership with those expected values. The producer
captures their actual getent/package/account authority; fabricated receipt values
in a unit fixture establish only that the reader checks the fields.
Host-copy metadata is never substituted for original guest ownership.
The caller-provided diagnostic archive must exactly match the in-root active
file's bytes and guest device/inode in each stopped snapshot. Neither an unrelated
archive nor fabricated host metadata is authority to alter that join.
Strictly frame-check, decode and hash **every** committed Batch before selecting
any row: fixture rows, diagnostic rows, metric/trace containers and other retained
signals. Preserve original payload bytes/hash, all projected raw log rows with
their original `(node_id, sequence, index)`, all latest cursors and their source
archives. Earlier raw cursor bytes and before/install diagnostic inventories
remain available; this does not reconstruct already missing historical files.
All signals' Resource, InstrumentationScope and schema URL containers use the
locked common message/string schemas, including UTF-8 Resource attribute keys
and Scope names. Only nested Metric/Span leaves stay opaque; product ingest owns
those leaf semantics. No general prost-parity claim follows.
Retain decoded `collection_gaps` strings in the returned Batch metadata and each
source record, in their original order; original payload bytes remain authority.
Reject incomplete/torn active tails, failed-writer markers, CRC errors,
malformed diagnostic log fields and drift.
Never reencode, replace or expose a filtered/copied Spool as the actual raw root.
Common Resource, InstrumentationScope and schema-URL fields in all three signals
are typed messages/UTF-8, not opaque Metric/Span leaves. The C-096/C-101
controls below register their separate existing strict-reader gap.
Only after full admission, select fixture rows by the registered path and
their exact archived source intervals, never by body text or a substring.
Keep the existing five-row comparison and exact `[0, final_cursor)` tiling on
that fixture path, with newline bytes/device/inode, no missing leading line,
gap, overlap or duplicate. Apply the same path selection to the complete raw
query pages, while keeping the unfiltered pages available. A diagnostic row
whose body equals a fixture sentinel must not enter or duplicate that fixture.
Extra fixture-path query rows still fail exact projected row conservation.
Before that finite projection, every log row of every retained Batch with sequence
above the pre-install ACK must appear exactly once, with exact projected fields,
in each complete drain and reboot query answer, regardless of source path.
Dropping the same diagnostic row from both answers does not establish custody.
The acknowledged query baseline includes the first diagnostic row and three old
fixture rows; each native-shaped Batch carries both source cursors. Preserve
that full baseline separately from the three fixture rows.
Unexpected fixture-path pending rows fail `unexpected pending log row`; an exact
five-row answer cannot erase an additional committed allowed-path row.
The actual [fixture enrollment](../../../tools/qualification/install/acceptance.sh)
adds the credential label `spindle-1`; that label is not present in Batch bytes.
[HTTP authentication](../../../crates/fabric-server/src/http.rs) supplies the
credential label to the committed Entry, and [server extraction](../../../crates/fabric-server/src/rows.rs)
preserves that label in query rows. The reader's fixed `spindle-1` is an explicit
fixture expectation, never evidence of the actual enrollment. All three old query
rows must retain that label and the node ID decoded from genuine FAI1/Batch input.
The native caller must independently join the observed enrollment/trimmed-token
hash, immutable credential-file stat/hash, actual Spool node ID and complete
unmodified query pages. Never synthesize labels, UUIDs or projected query rows
from source bytes to satisfy this join. A source-derived expected projection
and a genuine queried answer remain distinct observations.
Projection compares node label/ID, sequence/index, observed time, string body
and string attributes; it is not equality of every raw OTLP field. Query,
delivery and rate oracles remain byte-identical.
Stopped before/install inventories, raw sources, ACK, cursor and archives must
remain unchanged. Drain may append/rotate diagnostics, but every pending Batch,
including diagnostic-only Batches outside the five-row projection, still needs
the observed candidate sender's exact original-payload SHA-256 ACK. Retain the
before/install/drain diagnostic inventories and archives even after rotation
replaces an older `.1` entry. Do not impose diagnostic EOF tiling: an uncollected
stop tail is a registered legitimate shape.
Collection gaps on other sources stay visible without claiming diagnostic EOF
completeness. A gap naming the allowed fixture source in a pending or unread
Batch refuses fixture acceptance (`fixture source gap`); an exact row answer
cannot hide unknown fixture coverage. Gap strings use the actual source forms
from `log_source::issue` and the unavailable-source path in `Spindle::collect`.
For an unchanged active device/inode, after bytes must start with all before
bytes; an existing old file must remain identical. For one observed rotation,
after `.1` must have the previous active device/inode and preserve its entire
prefix; the new active identity must differ. The retained old file plus the
first new whole line must exceed 262144, the new line must fit 4096, and each
file must remain bounded/private/single-link. A short unjustified rename or a
different generation in `.1` cannot stand in for the prior prefix. Multiple
unobserved rotations are refused as insufficient preservation evidence unless
a separately registered observation history supplies every required prefix;
there is no general diagnostic-completeness claim.
## Registered source controls and expected first red
The [hand-encoded controls](../../../tools/qualification/install/test_custody_native.py)
reuse only old fixture wire primitives and adapter loading. Temporary trees and
process/stop records are synthetic source controls, never native receipts.
All 30 original installation methods, production and the three independent
oracles stay byte-identical. `test_query_oracle_stays_byte_identical` fixes the
query oracle SHA-256 to `fc696ee1173093c258ff4451c4a796dd8e5832761f9e0ff49f89a922b411d23a`.
No wider query-oracle caller scope is introduced by this card.
C-101's Wren review supplies the new prospective controls; its independent probe
is review evidence, not Olo's contained registration run. Commits `bf4320ef` and
`a273fd21` and their earlier predictions remain history; this revision supersedes
them before the first owner source cell. No published failure revision is rewritten.
| Requirement / literal control | Expected behavior |
| --- | --- |
| Full-root availability | Acknowledged starting diagnostic, pending diagnostic and all original payloads remain available; the explicit fixture tuple is recorded. Legacy reader's actual first error is `unexpected or failed Spool input`, converted to the named `native reader shape unavailable (qualification harness, not product): unexpected or failed Spool input` assertion failure. |
| Common containers | Field-zero ResourceMetrics/ResourceSpans/ScopeMetrics messages refuse. Valid literal Resource/Scope metadata accepts. |
| Common UTF-8 | Across metrics/traces, invalid Resource attribute key, Scope name, Resource schema URL and Scope schema URL refuse: eight cases independent of the native keyword. Existing two logs URL cases also refuse. Metric/Span nested leaves remain opaque. |
| Query oracle freeze / legacy root | Immutable query-oracle digest and legacy refusal of diagnostic directories remain required. |
| Original decoder authority | Mixed literal indices roundtrip; malformed diagnostic wire/UTF-8 cannot hide behind path projection. Every original Batch is decoded and hashed without mutation. |
| Native root completeness | Require private owned diagnostics directory and active file; missing directory/file, mode 0755 and symlink refuse. |
| Failed/interrupted frame custody | All seven existing legacy frame/marker/ACK-sidecar mutations also refuse with the native keyword, including both failure markers. |
| Diagnostic archive join | Archive bytes/device/inode must agree with the actual in-root active file. |
| Pending raw query rows | Drop, duplicate or rewrite pending diagnostic seq4/index1 in either answer or both refuses exact raw-row conservation before projection. |
| Path selection | PENDING and UNREAD bodies each collide in seq4 and in a later diagnostic row; only the registered path contributes the five fixture rows. |
| Extra pending fixture row | Legacy and native controls require `unexpected pending log row`, even when complete query answers retain the five intended fixture rows. |
| Rotation/preservation | Genuine full-file rotation accepts with prior active identity in `.1`; short/exact-cap rotation, changed generation/inode, first line over4096 and old-file rewrite without rotation refuse. Unchanged old file without rotation accepts. |
| Collection gaps | Literal decoded gap stays visible. Other-source gap remains visible and does not imply complete diagnostics; allowed-path pending/unread gap refuses fixture acceptance. |
| Existing finite checks | Diagnostic-only exact-byte ACK, source-prefix tiling, extra query row, substituted/invalid path registration, leaf alias/mode/size/ownership/type and old-row label/UUID controls remain required. |
The first diagnostic row occurs in seq1 and `queries.before`, before the three
fixture rows; every Batch carries both cursors as the native runtime does.
Native-shape changes are literal registrations, not claims that synthetic bytes
or fabricated current UID authenticate a guest. The exact enrollment label
`spindle-1` is established by `acceptance.sh:223`; raw FAI1 supplies the UUID,
while HTTP authentication supplies the query label. Never relabel genuine pages
or manufacture a query answer from raw expected rows. Consistent predecessor
query label/UUID substitution still refuses an old-row identity/baseline check.
Primary authority is the locked `opentelemetry-proto=0.33.0` generated
ResourceMetrics/ScopeMetrics and ResourceSpans/ScopeSpans message and string
attributes, `prost=0.14.4` string decoding, `Batch::validate`,
`OperationalLog::append` (strict overflow `>262144`, line `<=4096`),
`fabric-node.rs:157` (starting event) and `spindle/runtime.rs:1197-1222`
(all configured cursors). Common Resource attribute keys and Scope names are
UTF-8; their bytes are not opaque Metric/Span leaves. The original raw payload
and source archives remain authority throughout projection.
From this worktree, the exact inside-launcher source command is:
```sh
env PYTHONPATH=tools/qualification/install:tools/qualification:tools \
python3 -B -m unittest discover -v -s tools/qualification/install -p 'test_*.py'
Root uses python3 -B tools/resource_group.py --, a fresh serial 1-GiB
reservation and default 1800 seconds/16-GiB high/20-GiB max/no swap. Predicted initial result:
60 methods:35 positive, eight named failing methods with15 failed assertions/
subtests,17 explicit dependent skips, exit1. This is NOT RUN.
Failure entries: native-root availability1; common container3; schema URL4;
old query identity2; Resource-key/Scope-name UTF-8 four; visible gap1.
The original30 plus new30 definitions total60; native17 methods individually
skip until the keyword exists. Missing gap metadata is an assertion failure,
not a KeyError. The unchanged final green must execute all60 with zero skips.
Preserve the actual named red before separately committed production correction.
Exact source/tree/ancestry and every changed-path pin go into plan02 for
independent review before execution; review is not completion evidence.
QUAL-INSTALL/QUAL-H mapping stays queued until the first genuine native result.
See the prospective producer/caller card for both-family positives, Debian producer refusals, owner-admitted package/helper/ source binding, credentials, raw observations, transfers, deadlines and cleanup. Those requirements remain mandatory and NOT RUN; no source-only green completes installation acceptance or authenticates a guest stop/service UID.
## scratch/installation-custody-native/docs/experiments/formal/installation-release-protocol.md
SHA-256 `59a028727c20d518f0075e1dcea334f1ff5542d8c46e2a73e6d1c4649b4f147d`, 25271 bytes.
```text
# Registered release package and lifecycle checks
Status: revision 5 amended 2026-10-13 for the existing unreadable-audit gate.
Revision 4 registered 2026-10-12 the frozen candidate's exact candidate and L1
predecessor versions and the dated Debian image. Revision 3,
registered 2026-10-10 before stopped-overlay continuation, named the earlier
candidate; a result belongs to the revision it ran under. This extends the
[baseline installation protocol](installation-acceptance-protocol.md) for the
owner-selected Debian/Fedora release matrix. No result is recorded here.
These checks are finite local guest measurements, not deployment qualification.
Revision 5 preregisters the literal baseline Fedora F3 control
`InstallationShellControls.test_acceptance_fedora_rejects_unreadable_audit`:
execute the actual audit decision with readable-empty audit evidence (PASS) and
an audit-file `grep` exit 2 (FAIL, with no F3 PASS). An unavailable or unreadable
audit already cannot support a pass; this adds its baseline control without
changing guest limits, gate outcomes or the frozen package inputs.
## Stopped-overlay continuation
A separately recorded continuation may finish L2/removal from an immutable
stopped trial whose historical baseline exited 0 and whose exact candidate
L1 upgrade passed, but whose reboot observer failed. Preserve the original
overall failure. Bind its receipt/acceptance/harness hashes, exact candidate and
predecessor, source identity, stopped overlay SHA-256 and official backing image.
Use a new owned overlay with that stopped disk as read-only backing; never
mutate the retained input. Freeze the saved pre-reboot boot ID, config hashes,
account identity and exact row witnesses from that disk before L2. A genuinely
different live boot identity remains required; mere SSH reconnection is insufficient.
Use stdin for the privileged boot comparison so the unprivileged SSH shell
cannot expand protected paths. Record each actual phase exit, caps and cleanup
in a new continuation receipt. This does not qualify unrun baseline criteria on
the candidate or relabel the original failed run. The final new candidate still
requires fresh full installation/lifecycle trials.
## Fixed inputs and containment
Freeze source (including Rust crates, UI inputs, vendored dependencies, package
scripts, helpers, license/notice), Rust 1.99.0, Cargo.lock, a successful exact
console build, package SHA-256, and the candidate/predecessor versions before
execution. Candidate package version is `0.1.0~alpha.1` on Debian and
`0.1.0-0.alpha.1` on RPM. L1 predecessors are the registered historical Debian
package re-versioned to `0.1.0~alpha.0` (payload and maintainer scripts unchanged,
`packaging/reversion-predecessor-deb.sh`) and the constructed RPM fixture
`0.1.0-0.alpha.0`. This metadata does not authorize a release tag.
`build-isolated.py --console-build BUILD --family debian|fedora` builds in the
bounded rootless Debian 12 sysroot; RPM construction uses `rpmbuild` there too,
keeping the same registered glibc floor. Copy verified UI assets into owned
storage; recheck their receipt/hashes in the frozen-source guest before packaging.
A source or asset mismatch blocks construction.
The candidate RPM may instead use `build-rpm.sh OUT CANDIDATE_DEB
--candidate-deb BUILD_RECEIPT [--mutate=NAME]`. Require a completed successful
Debian build/cleanup receipt, its exact package SHA-256, exact candidate Debian metadata,
and its matching frozen-source manifest/bundle hashes. Preserve every regular
Debian data member unchanged, including native binaries, UI assets and notices;
record their hashes plus source and input package identity in RPM documentation.
Use the frozen candidate account-check script for RPM installation. RPM metadata,
scriptlets, dependencies and SELinux context restoration remain Fedora-specific.
Reject historical/wrong-version/substituted payloads before staging. This is
repackaging the same Debian12-sysroot binaries, not an independent Fedora compiler
result. Actual Fedora enforcing-mode lifecycle cells still remain necessary.
Run every host command through `python3 -B tools/resource_group.py -- COMMAND`.
Use only the mounted data drive, a maximum 20 GiB/no-swap descendant cgroup and
the default 30-minute deadline. The package-build container inherits that cgroup;
QEMU is its descendant. One guest at a time, 2 virtual CPUs, 6144 MiB guest RAM,
8 GiB virtual disk, disk-backed owned overlay/scratch, no KVM requirement. Preserve
failures/receipts and stop/reap guest processes before cleanup. Enforce the
existing 100 GB data-tree budget, stop admission at 95 GB including the reserved
8 GiB build or full guest disk plus image/log overhead, and record free disk
before admission. Do not
interpret these guest limits or a tiny fixture as the 100 GB retention-capacity
release result.
Debian: Debian 13 genericcloud amd64 build 20261001-2618/systemd 257, from that
build's [dated directory](https://cloud.debian.org/images/cloud/trixie/20261001-2618/),
SHA-512 `f46f0671…b468c3d` as pinned in the harness. Fedora: Cloud Base Generic 44-1.7 x86_64, from the
[official Fedora image directory](https://dl.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/),
583729152 bytes, SHA-256
`28680fe5b371a5a82ebf43a31926e086a168e59949d03969c5093e7071f90b7f`.
Match the official HTTPS checksum manifest and the downloaded image before boot.
The Fedora checksum manifest is signed; this harness pins its reviewed HTTPS
checksum and does not claim independent OpenPGP signature validation. Record
actual guest OS, systemd, kernel, PID1, cgroup controllers and SELinux mode.
Fedora must remain enforcing; no permissive switch or broad allow policy is a
workaround. An unavailable tool, image mismatch or unavailable gate is not a pass.
## CI host containment
GitHub-hosted runner bootstrap binds disk-backed `/mnt` to the required data
mount and starts the user systemd manager. Before project children run, configure
`user-UID.slice` as a stricter ancestor: memory maximum is the lesser of 12 GiB
and 75% of actual MemTotal, high is five-sixths of that maximum, swap maximum
zero, CPU quota no greater than the available CPU count. Verify the actual
kernel memory/high/swap/CPU files and write `ci-parent.json`; a missing or
mismatching bound fails closed. `RUST_TEST_THREADS=2` and `CARGO_BUILD_JOBS=2`
pass through the launcher. This hosted-runner ancestor does not alter the local
20 GiB contract or establish release workload acceptance.
## Gates
The existing A1–A12 criteria remain unchanged. Backend package operations are
`dpkg` on Debian and `rpm` on Fedora; guest extraction uses `rpm2cpio/cpio`.
The fixture's explicit 20 GiB retention policy is a narrow delivery/lifecycle
fixture. The release's 100 GB default is independently checked in configuration
and examples, and needs the release storage workload before capacity acceptance.
| Gate | Pass rule |
| --- | --- |
| A2c/A2d | Fresh candidate install refuses duplicate UID0 and GID0 named fabricolly before unpack. Preserve root account/group; delete only disposable duplicate fixture entries. Historical predecessor migration cells do not claim this corrected admission behavior. |
| P1 | Both exact packages contain a complete hash-verified console (HTML, compiled WASM, manifest, normal/maskable icons, worker), CSP metadata, binaries, units, examples and Apache-2.0 license/notice. Reject changed/missing/unlisted/symlink assets, a failed build receipt and a mismatching source identity. |
| F1/F2 | Fedora reports SELinux Enforcing before delivery and after restart/removal; every A1–A12 delivery/sandbox/custody criterion still succeeds. |
| F3 | Boot audit contains no AVC/USER_AVC with a Fabric binary `comm`; retain audit and service journal. An unavailable audit facility is a failed prerequisite. |
| L1 | Install an exact strictly older predecessor, complete A1–A12, upgrade to the candidate with the native package manager, restart services, and preserve exact three fixture rows including source/sequence/index/timestamps/body/attributes, config checksums and service-account identity. Validate installed UI hashes after upgrade. An equal-version reinstall is refused. |
| L2 | Record the guest boot ID, reboot the VM, reconnect only after a distinct boot ID, and observe both enabled services active. Exact retained rows/config/account still match. Fedora remains enforcing. |
| A13a | Native removal stops both services and removes binaries while retaining telemetry/configuration/account. |
| A13b | Debian native purge removes state/configuration. RPM has no purge operation: verify removal preserved both, then explicitly delete the disposable fixture state/configuration. This cleanup is not presented as a package purge guarantee. |
| F4 | Fedora remains enforcing through reboot/removal; post-reboot audit contains no Fabric-process AVC. |
L1's expected bodies are `accept-line-before-start`, `accept-line-during-outage`
and `accept-line-after-pressure`, each exactly once from `spindle-1`. Identity
comparison includes every projected field and uses Python JSON integers. The
query fixture intentionally has a denied-log gap; these lifecycle checks do not
claim full-source completeness or change the independent query oracle.
These are **explicit legacy-admin migration fixtures**. Fresh candidate tests use
`--legacy-server`; upgrade installs a guest-only systemd drop-in selecting
`serve-legacy`. The shipped normal service uses passkey configuration, and the
separate authenticated-console protocol must exercise its production entrypoint.
Legacy-admin success cannot satisfy access-control or authenticated UI gates.
Use `run-qemu.py --family FAMILY --package CANDIDATE --source-commit COMMIT
--run-id ID --memory-stressor parallel --upgrade-from PREDECESSOR` for L1/L2.
Without `--upgrade-from`, the harness runs the ordinary baseline/removal path and
has no upgrade/reboot result. The previous package's hash/version belongs in the
receipt. A synthetic predecessor must be labeled as such and cannot establish
migration from a historical implementation it did not contain.
## Negative controls and evidence
### Prospective removal-checker counterexample controls
Coordination C-035 / C-aud-7 minimized an unenforced-check defect at harness
`de77de7399710c5afd50ae0de981fda6e0b75f30`: negated commands and the first
member of `&&` lists bypass Bash errexit; baseline A13a accepted one live unit.
`InstallationShellControls` in `tools/qualification/install/test_run_qemu.py`
invokes the actual lifecycle remove phase with isolated paths and package/service
stubs. Either active, activating, deactivating or reloading unit, literal Fabric
AVC, missing retained state and incomplete purge must exit1 without the affected
PASS; clean fixtures must exit0. Baseline A13a rejects the same live states.
Service-query controls additionally require both observed states: empty or one-line
output and exit1 despite two inactive lines fail A13a. A literal grep exit2 at the
post-reboot AVC check fails F4; clean inactive queries and readable empty audits
remain positive controls. These are verifier prerequisites, not guest outcomes.
The actual QEMU cleanup AST additionally requires an independently detected RPM mutation
and confirmed process stop; unrelated/provenance failures retain scratch.
These prospective controls precede repair and claim no guest outcome. Preserve
the contained red first; after repair, register its actual fix commit in
`docs/formal/counterexamples.json` in a separate trust-boundary commit. RPM
construction plans explicitly create each fresh mutant output directory before
`build-rpm.sh`; an existing directory is refused. Limits and gate rules remain.
Revision-bound source-only checker execution evidence is recorded in the
[installation checker results wiki](https://github.com/kmosoti/FabricO11y/wiki/Experiment-formal-installation-checker-prerequisite-results-01).
These controls establish no guest or package acceptance.
### Prospective stopped-custody upgrade witness
The separately registered [native input-shape amendment](installation-native-custody-protocol.md)
adds prospective unfiltered diagnostic-root handling and explicit finite fixture
projection. It leaves this legacy reader contract and existing controls unchanged;
native producer/admission/caller acceptance remains pending.
Issue #41's Spool/backlog upgrade obligation needs a predecessor-produced
unacknowledged Batch and a distinct unread newline-terminated source suffix
across stopped native install, then exact candidate ACK/query and real reboot.
Historical L1/L2 checks remain required. A retained file hash alone cannot
establish recoverable delivery; shared-Rust `spool_dump` is a diagnostic.
`test_custody_witness.py` independently hand-encodes small FAB1/FAC1/protobuf
fixtures for prospective `custody_witness.stopped_snapshot` and `conserve`.
The initial named `MechanismAvailability` failure records absence of the
adapter; it is not a product defect or corrupt-frame counterexample. The
[`custody_witness.py`](../../../tools/qualification/install/custody_witness.py)
adapter now exists; source-control results belong to the linked checker-results
wiki, while native stopped-custody integration and guest acceptance remain
pending. Its unchanged
`CustodyWitnessControls` require complete CRC-checked frames, envelope/wire/u64
bounds, nonterminal sequence/ACK, full Cursor/Btrfs identity and consumed-prefix
CRC, last cursor in sequence order rather than maximum offset, complete stopped
service observations, no-follow owned paths, and exact two-line custody joins.
Reset source identity, missing/changed pending bytes, premature ACK, duplicated
or changed projected query rows and unchanged reboot identity must be refused. ACK events
must match the independently observed candidate node PID/unit/boot/run window
and exact raw Batch hash; their message alone omits Strand identity.
Query equality compares the projected node label, node ID, sequence, index,
observed timestamp, string body and string attributes, not every raw LogRecord
field or nonstring AnyValue arm. Retained source hashes and candidate exact-byte
ACKs bind the original node-side Batch bytes. This witness does not independently
reread raw server storage; its raw-byte commitment evidence is the observed
node's ACK. Host-copy device/inode/mode/uid/gid are drift witnesses, not guest
ownership-policy admission. Native callers must bind and verify their guest
observations separately. The raw reader refuses an incomplete active tail,
including a legitimate torn tail after a crash; it does not repair or tolerate
that tail as complete custody.
#### Complete archived-prefix conservation amendment, 2026-10-13
Coordination [C-aud-11](https://github.com/kmosoti/FabricO11y/issues/59#issuecomment-6138325736)
identified a consumed leading line absent from every Batch and query row while
the two sentinels and candidate ACKs remained valid. Preregister
`CustodyWitnessControls.test_archived_leading_lost_line_cannot_escape_source_conservation`
before repair. Its archive is exactly `accept-line-lost-by-reader\n` followed by
the literal pending and unread lines. Hand-encoded source offsets, Cursor prefix
length/CRC, FAB1/FAC1 CRCs and exact-byte candidate ACK hashes are rebuilt for that
archive; the lost line is absent from all raw and projected rows. Snapshot
admission must succeed, then `conserve` must raise `ValueError` naming
`source prefix conservation`, rather than rejecting a stale CRC or missing ACK.
For each archived path with a final cursor, the expected old, pending and unread
projected rows for that path must tile `[0, final_cursor_offset)` exactly in
source order, with no gap, overlap or duplicate interval. Every interval must
match the archived newline-terminated body bytes and source device/inode. The
existing three-row lifecycle baseline and sentinel checks remain required;
matching them alone cannot establish full archived-prefix conservation.
This tiling witness is limited to the registered finite fixture of complete
UTF-8 newline-terminated source lines. The actual
[`log_source.rs`](../../../crates/fabric-adapter-linux/src/log_source.rs)
collector skips oversized lines (more than 4,096 body bytes) and invalid UTF-8,
reports collection gaps, and can commit cursor-only progress through a skipped
oversized line. Such consumed bytes have no `body + '\n'` row interval, so this
witness refuses that archive. It does not establish general source conservation
for intentional skips, incomplete tails or arbitrary log-input histories.
Keep every prior control, oracle and gate unchanged. Run the named fixture only
through the registered serial default1800/full1GiB control cell and preserve its
actual red before production correction. This amendment changes no product wire
semantics and records no execution or native acceptance.
Run only through the admitted serial resource launcher: default1800 outer and
1GiB reservation for synthetic controls. Native cells keep1800 outer,180-second
lifecycle phases and9GiB whole guest reservation. Preserve failures and raw
source custody, exporting only hashes of credentials. Frozen packages, wire
semantics, query/delivery/rate oracles and all existing gate limits stay fixed.
This prospective registration claims no checker or guest acceptance.
#### Nested OTLP structural amendment, 2026-10-13
Preregister `CustodyWitnessControls.test_nested_otlp_structural_refusals` with
hand-encoded inner OTLP logs inside CRC-valid FAB1/FAC1 frames. A sparse LogRecord
containing valid unknown field99 is a positive, even though its projected body is
empty. ExportLogsServiceRequest field0 (`00 00`), a nested LogRecord field number
greater than `(1<<29)-1`, and an eleven-byte nested varint must raise `ValueError`
for the malformed protobuf field/tag or varint; accepting an empty projection or
rejecting only missing sentinels is insufficient. The locked prost0.14.4
`encoding::decode_key` bounds the key to u32 and tags to1 through `(1<<29)-1`, and
its varint decoder rejects values requiring more than ten bytes. The existing
Rust `Batch::validate` decodes each OTLP signal before admitting its bytes.
These source constraints add no new wire semantics or oracle changes. Run the
named control only through the registered default1800s/full1GiB serial driver;
retain the actual red before a separately reviewed structural-validator repair.
#### Bounded reader and original-byte projection amendment, 2026-10-13
Coordination C-061/X-106 admits a 420-line production ceiling, at most +120 after
the 297-line adapter, with no oracle or limit change. Register seven additional
`CustodyWitnessControls` methods before repair; the three malformed cases and
sparse positive in `test_nested_otlp_structural_refusals` stay unchanged.
`test_generic_reader_positive_forms` admits opaque unknown bytes, nonminimal
valid varints, the largest legal tag/u64 and matched unknown groups, including
100 empty nested groups. `test_generic_reader_structural_refusals` rejects
invalid keys/wires, truncation, overlong/overflowing value and length varints,
lone/mismatched/unterminated groups, 101 nested groups and an unknown field at
exhausted depth. Controls call the existing `fields(raw, types)` reader directly.
`test_packed_numeric_reader_forms_and_refusals` registers its explicit
`('packed', element_wire_kind)` spec for kinds 0/1/5: both unpacked and packed
forms are admitted; malformed packed varints and nondivisible fixed-8/fixed-4
payloads are refused. Arbitrary opaque bytes are never guessed to be messages.
`test_logs_common_schema_and_utf8_refusals` requires declared wires and UTF-8
through ExportLogsServiceRequest/ResourceLogs/ScopeLogs/LogRecord and all
consumed common schemas: Resource, InstrumentationScope, EntityRef, KeyValue,
AnyValue, ArrayValue and KeyValueList. `test_logs_common_and_message_depth_boundaries`
admits literal common fields, opaque bytes and nonzero bools. From the export
root, 48 AnyValue.array/ArrayValue.values pairs place a known string at depth 100
and remain valid; 49 pairs or an unknown scalar at exhausted depth fail.
Each signal export has its own recursion budget, matching pinned prost 0.14.4.
`test_metrics_traces_container_scope` requires strict Export→Resource→Scope→
Metric/Span wire containers; malformed keys/varints/lengths and container wires
fail. Hand-encoded `system.memory.total` Gauge and `system.cpu.time` cumulative
Sum use the actual host shapes in `src/spindle/runtime.rs:266-299` and
`crates/fabric-adapter-linux/src/host.rs:190-217`; a literal Span is also positive.
Metric/Span nested leaves remain opaque, explicitly witnessed by a malformed
nested Gauge/Event fixture that this reader admits without asserting semantic
validity. Product ingest decoding is the leaf-semantics authority; frame CRC
establishes byte integrity, not semantic validity. No general prost parity or
native product acceptance is claimed by these finite controls.
`test_original_payload_log_projection_binding` registers exact literal log rows
for valid unknown groups, last oneof arm and merged optional body/value messages.
The stored original Batch payload and SHA-256 must remain exact. A separately
filtered log projection may feed the byte-identical query oracle; it cannot
replace raw custody or exact-byte candidate ACK evidence. The caller projects
log rows only and must not reinterpret opaque metric/trace leaves through the
oracle merely to obtain those rows. Query/delivery/rate oracles remain unchanged;
native caller authentication against immutable candidate receipts stays separate.
Run these source-only fixtures through the same serial default 1800s/full 1GiB
driver; preserve actual red before production repair and rerun unchanged controls.
Refusals name the violated wire/string/group/depth property. Legitimate-form
rejections or wrong projections on the old reader are harness defects, not
product qualification failures. No execution result is recorded by this amendment.
#### Nested envelope recursion amendment, 2026-10-13
Static review of the bounded repair found that Cursor and BtrfsIdentity readers
restart their recursion budget. Register two literal controls before correction:
`test_cursor_inherits_batch_recursion_budget` and
`test_btrfs_inherits_cursor_recursion_budget`. Locked prost 0.14.4
`Message::merge` starts at budget 100; `encoding::message::merge` decrements it
for each nested message. Batch field 7 is Cursor at depth 1, and Cursor field 8 is
BtrfsIdentity at depth 2 (`fabric-frame::envelope`). Unknown group and scalar
skipping use that inherited context, including its exhausted-budget refusal.
Inside CRC-valid Batch frames, Cursor admits 99 matched empty field 99 groups and
refuses 100; BtrfsIdentity admits 98 and refuses 99. An unknown field 98 scalar inside
the accepted number of groups must also be refused at exhausted depth. Positives
retain the literal cursor identity/offset and exact original payload. Existing
root-reader 100-group positives and all earlier expectations remain unchanged.
This is an envelope/strict-reader correction within the existing 420/+120 scope;
it adds no metric/trace leaf semantics or general prost-parity claim. Root must
retain the contained named red before a separate production repair. These new
controls and native acceptance are NOT RUN in this registration.
Keep the existing named Debian package defects: root-user must fail A6,
no-collision-check must fail A2a/A2b, and no-memory-max must fail A8. The existing
mutation grader must reject unrelated failures and bad provenance/environment.
Fedora package mutations are separate exact RPM artifacts built with
`build-rpm.sh OUT BUILD --mutate=NAME`; grade them with
`run-qemu.py --family fedora --package MUTANT --expected-mutation NAME` and the
same named A6/A2a+A2b/A8 failure requirements. The harness refuses the Debian
repacker on Fedora. An unrelated failure, missing fixture/provenance or NOT-RUN
cannot count as detection. Required RPM negative controls remain unrun until
these artifacts are constructed and independently graded; a baseline success
cannot erase that missing gate.
A Fedora migration predecessor can be constructed by
`build-rpm.sh OUT HISTORICAL_DEB --historical-deb`. Its source is the exact
`readiness-debian12-03` alpha1 Debian artifact, SHA-256
`ac71995b3c8b88cbb6f7dd05d9854e347b7a783caadc1b602f6590d6460f4a2f`.
Preserve its historical binaries, static units, examples, sysusers and account
collision preinstall script. The fixture's `MIGRATION-FIXTURE.json` records
those payload hashes; native RPM lifecycle scripts adapt package operations.
It contains no historical console and is a **constructed predecessor fixture**,
not evidence that an alpha1 Fedora package shipped or was accepted. L1 upgrades
its historical implementation/state into the exact current candidate RPM.
Every receipt records the exact package/helper/image/source identities, command,
exit, chosen A12 fixture, measured resources, real lifecycle stages and cleanup.
A failed, interrupted or missing required phase blocks the corresponding cell.
First-run results and limitations belong in a separate result record.
SHA-256 db7c96fc572b3825e9e327e6b8cf86d74e058b4d3cb6a4d1d1ae292f76db037f, 31873 bytes.
"""C-092 native-shaped inputs, hand-encoded independently of the product.
Temporary trees are synthetic controls, never native guest receipts or Spools.
"""
import copy
import hashlib
import inspect
import os
from pathlib import Path
import stat
import struct
import tempfile
import unittest
import zlib
import test_custody_witness as legacy
from test_custody_witness import BINDING, NODE, PROCESS, SOURCE, adapter, frame, ld, vi
DIAG = '/var/lib/fabrico11y/node/spool/diagnostics/spindle.log'
OLD = (b'accept-line-before-start\n', b'accept-line-during-outage\n',
b'accept-line-after-pressure\n')
PENDING = b'accept-upgrade-pending-literal\n'
UNREAD = b'accept-upgrade-unread-literal\n'
DIAG_OLD, DIAG_PENDING, DIAG_NEW = b'event starting\n', b'event pending\n', b'event sample\n'
STOP = b'event stopping\n'
CAP = 262144
def literal_row(sequence, index, path, device, inode, start, line):
return dict(node='spindle-1', node_id=NODE.hex(), sequence=sequence, index=index,
observed_ns=123, body=line[:-1].decode(), attributes={
'log.file.path': path, 'log.file.device': str(device),
'log.file.inode': str(inode), 'log.file.offset.start': str(start),
'log.file.offset.end': str(start + len(line))})
def record(row):
raw = b'\x59' + struct.pack('<Q', 123) + ld(5, ld(1, row['body'].encode()))
return raw + b''.join(ld(6, ld(1, key.encode()) + ld(2, ld(1, value.encode())))
for key, value in row['attributes'].items())
def malformed_diagnostics():
raw = record(literal_row(4, 1, DIAG, 7, 19, 0, DIAG_OLD))
return (('wire', raw + vi(3, 0), 'wire'),
('UTF8', raw + ld(5, ld(1, b'\xff')), 'UTF|utf'))
def source_cursor(path, device, inode, raw, offset):
prefix = raw[:min(offset, 64)]
return (ld(1, path.encode()) + vi(2, device) + vi(3, inode) + vi(4, offset)
+ vi(6, len(prefix)) + vi(7, zlib.crc32(prefix)))
def payload(sequence, rows, cursors, gaps=()):
logs = ld(1, ld(2, b''.join(ld(2, record(row)) for row in rows)))
return (vi(1, 1) + ld(2, NODE) + vi(3, 1) + vi(4, sequence) + ld(6, logs)
+ b''.join(ld(7, current) for current in cursors)
+ b''.join(ld(8, gap.encode()) for gap in gaps))
def signal_batch(signal, export):
row = literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)
return payload(4, [row], []) + ld(signal, export)
class NativeFixture:
def setUp(self):
self.temp = tempfile.TemporaryDirectory(dir=os.environ['FABRIC_SCRATCH_ROOT'])
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.spool = self.root / 'spool'; self.spool.mkdir(mode=0o700)
self.diagnostics = self.spool / 'diagnostics'; self.diagnostics.mkdir(mode=0o700)
self.active = self.diagnostics / 'spindle.log'
self.active.write_bytes(DIAG_OLD); self.active.chmod(0o600)
(self.spool / 'identity').write_bytes(b'FAI1' + NODE + struct.pack('<Q', 1))
(self.spool / 'writer.lock').touch()
def snapshot(self, phase, payloads, ack, allowed, *, fixture_paths=(SOURCE,),
receipt_change=None, change=None, archive_change=None):
raw = b''
for item in payloads: raw += frame(item, len(raw))
(self.spool / 'batches.faj').write_bytes(raw)
(self.spool / 'acked').write_bytes(b'FAK1' + struct.pack('<QQ', 1, ack))
logs = {}
for index, (path, content, device, inode) in enumerate((
(SOURCE, allowed, 7, 9),
(DIAG, DIAG_OLD, 0, 0) if not self.active.exists() else
(DIAG, self.active.read_bytes(), self.active.stat().st_dev, self.active.stat().st_ino))):
archive = self.root / f'{phase}-{index}.log'; archive.write_bytes(content)
logs[path] = dict(archive_path=str(archive), device=device, inode=inode,
btrfs_identity=None)
if archive_change: archive_change(logs)
receipt = dict(binding=dict(BINDING), spool_root=str(self.spool),
spool_bytes=268435456, stop_confirmed=True, boot_id='literal-boot-a',
extra_project_units=[], units={unit: dict(active_state='inactive', query_exit=3,
main_pid=0, cgroup_empty=True, companions=[])
for unit in ('fabrico11y-server.service', 'fabrico11y-node.service')},
service_identity=dict(uid=os.geteuid(), gid=os.getegid()),
candidate_process=copy.deepcopy(PROCESS) if phase == 'drain' else None)
if receipt_change: receipt_change(receipt)
if change: change(self.spool)
def inventory():
return {str(path.relative_to(self.root)): (path.lstat().st_mode,
str(path.readlink()) if path.is_symlink() else
path.read_bytes() if stat.S_ISREG(path.lstat().st_mode) else None)
for path in self.root.rglob('*')}
original = inventory()
try:
if 'fixture_paths' not in inspect.signature(self.module.stopped_snapshot).parameters:
return self.module.stopped_snapshot(self.spool, receipt, logs)
observed = self.module.stopped_snapshot(self.spool, receipt, logs, fixture_paths=fixture_paths)
self.assertEqual([source['payload'] for source in observed['sources']], payloads,
'snapshot replaced or dropped an original payload')
self.assertEqual([source['sha256'] for source in observed['sources']],
[hashlib.sha256(item).hexdigest() for item in payloads])
return observed
finally:
self.assertEqual(inventory(), original, 'reader mutated the full raw input')
def chain(self, *, rotate=False, rotation_full=True, rotation_size=None, rotated_old=None,
rotated_replace=False, prior_old=False, old_rewrite=False, rewritten=False,
lost_prefix=False, diag_pending=DIAG_PENDING, diag_line=DIAG_NEW,
diag_only_pending=False, extra_pending=False, pending_gap=(), unread_gap=()):
lost = b'accept-line-lost-by-reader\n' if lost_prefix else b''
allowed = lost + b''.join(OLD) + PENDING + UNREAD
prefix = b''.join(OLD)
initial = DIAG_OLD + diag_pending + (b'event prior-sample\n' if diag_only_pending else b'')
if rotate:
if rotation_full:
padding = (CAP if rotation_size is None else rotation_size) - len(initial)
initial = b'x\n' * (padding // 2) + (b'\n' if padding % 2 else b'') + initial
if rotate or prior_old:
previous = self.diagnostics / 'spindle.log.1'
previous.write_bytes(b'event previous-generation\n'); previous.chmod(0o600)
self.active.write_bytes(initial)
device, inode = self.active.stat().st_dev, self.active.stat().st_ino
cursor_a = lambda offset: source_cursor(SOURCE, 7, 9, allowed, offset)
cursor_d = lambda offset: source_cursor(DIAG, device, inode, initial, offset)
diag_start = len(initial) - len(DIAG_OLD + diag_pending) - (
len(b'event prior-sample\n') if diag_only_pending else 0)
starting = literal_row(1, 0, DIAG, device, inode, diag_start, DIAG_OLD)
payloads, old_rows, offset = [], [starting], len(lost)
for sequence, line in enumerate(OLD, 1):
row = literal_row(sequence, 1 if sequence == 1 else 0, SOURCE, 7, 9, offset, line)
old_rows.append(row); offset += len(line)
payloads.append(payload(sequence, [starting, row] if sequence == 1 else [row],
[cursor_a(offset), cursor_d(diag_start + len(DIAG_OLD))]))
pending_end = len(lost + prefix + PENDING)
pending_rows = [literal_row(4, 0, SOURCE, 7, 9, len(lost + prefix), PENDING),
literal_row(4, 1, DIAG, device, inode, diag_start + len(DIAG_OLD), diag_pending)]
if extra_pending:
pending_rows.append(literal_row(4, 2, SOURCE, 7, 9, len(lost + prefix), b'extra-fixture-row\n'))
payloads.append(payload(4, pending_rows, [cursor_a(pending_end),
cursor_d(diag_start + len(DIAG_OLD + diag_pending))], pending_gap))
if diag_only_pending:
row = literal_row(5, 0, DIAG, device, inode, diag_start + len(DIAG_OLD + diag_pending), b'event prior-sample\n')
pending_rows.append(row)
payloads.append(payload(5, [row], [cursor_a(pending_end), cursor_d(len(initial))]))
before = self.snapshot('before', payloads, 3, allowed)
installed = self.snapshot('installed', payloads, 3, allowed)
if rotate:
self.active.replace(self.diagnostics / 'spindle.log.1')
self.active.write_bytes(diag_line + STOP); self.active.chmod(0o600)
if rotated_old is not None:
(self.diagnostics / 'spindle.log.1').write_bytes(rotated_old)
if rotated_replace:
replacement = self.root / 'replacement.log'; replacement.write_bytes(initial)
replacement.chmod(0o600); replacement.replace(self.diagnostics / 'spindle.log.1')
else:
self.active.write_bytes((b'event rewritten\n' if rewritten else initial) + diag_line + STOP)
if old_rewrite:
(self.diagnostics / 'spindle.log.1').write_bytes(b'event modified-old\n')
after_diag = self.active.read_bytes()
sequence = len(payloads) + 1
unread_rows = [literal_row(sequence, 0, SOURCE, 7, 9, pending_end, UNREAD),
literal_row(sequence, 1, DIAG, self.active.stat().st_dev, self.active.stat().st_ino,
len(after_diag) - len(diag_line + STOP), diag_line)]
payloads.append(payload(sequence, unread_rows, [cursor_a(len(allowed)), source_cursor(
DIAG, self.active.stat().st_dev, self.active.stat().st_ino, after_diag,
len(after_diag) - len(STOP))], unread_gap))
drained = self.snapshot('drain', payloads, sequence, allowed)
rows = old_rows + pending_rows + unread_rows
self.assertEqual([row for source in drained['sources'] for row in source['rows']], rows)
queries = dict(before=copy.deepcopy(old_rows), after_drain=copy.deepcopy(rows),
after_reboot=copy.deepcopy(rows), boot_before='literal-boot-a', boot_after='literal-boot-b',
pending_body=PENDING[:-1].decode(), unread_body=UNREAD[:-1].decode())
events = [dict(unit=PROCESS['unit'], pid=55, boot_id=PROCESS['boot_id'], at_ns=150,
message=f'delivery sequence={seq} sha256={hashlib.sha256(item).hexdigest()} '
f'status=ack committed_through={seq} elapsed_us=1')
for seq, item in enumerate(payloads, 1) if seq > 3]
return before, installed, drained, queries, events
class NativeShapeAvailability(NativeFixture, unittest.TestCase):
def test_unfiltered_native_root_and_explicit_fixture_projection_are_supported(self):
self.module = adapter()
try:
observed = self.chain()[0]
except ValueError as error:
self.fail(f'native reader shape unavailable (qualification harness, not product): {error}')
self.assertEqual(observed.get('fixture_paths'), (SOURCE,), 'explicit fixture projection is absent')
self.assertIn('diagnostics/spindle.log', observed['files'], 'full raw-root inventory is absent')
class LiteralNativeRecords(unittest.TestCase):
def test_mixed_record_indices_and_original_payload_remain_literal(self):
fixture = literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)
diagnostic = literal_row(4, 1, DIAG, 7, 19, 0, DIAG_OLD)
original = payload(4, [fixture, diagnostic], [])
self.assertEqual(adapter().envelope(original)[2], [fixture, diagnostic])
def test_query_oracle_stays_byte_identical(self):
oracle = Path(__file__).resolve().parents[1] / 'query_oracle.py'
self.assertEqual(hashlib.sha256(oracle.read_bytes()).hexdigest(),
'fc696ee1173093c258ff4451c4a796dd8e5832761f9e0ff49f89a922b411d23a')
def test_collection_gap_remains_visible_in_decoded_batch(self):
gap = 'log rotated; old tail unknown: ' + DIAG
original = payload(4, [literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)], [], [gap])
self.assertEqual(adapter().envelope(original)[0].get('collection_gaps'), [gap])
def test_malformed_diagnostic_row_cannot_hide_behind_fixture_selection(self):
fixture = literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)
prefix = vi(1, 1) + ld(2, NODE) + vi(3, 1) + vi(4, 4)
for name, malformed, reason in malformed_diagnostics():
original = prefix + ld(6, ld(1, ld(2, ld(2, record(fixture)) + ld(2, malformed))))
with self.subTest(diagnostic=name), self.assertRaisesRegex(ValueError, reason):
adapter().envelope(original)
def test_resource_metrics_resource_must_be_strictly_decoded(self):
resource = ld(1, ld(1, b'host.name') + ld(2, ld(1, b'literal-host')))
self.assertEqual(adapter().envelope(signal_batch(5, ld(1, ld(1, resource))))[2],
[literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)])
with self.assertRaisesRegex(ValueError, 'protobuf (field|tag)'):
adapter().envelope(signal_batch(5, ld(1, ld(1, b'\x00\x00'))))
def test_resource_spans_resource_must_be_strictly_decoded(self):
resource = ld(1, ld(1, b'host.name') + ld(2, ld(1, b'literal-host')))
self.assertEqual(adapter().envelope(signal_batch(9, ld(1, ld(1, resource))))[2],
[literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)])
with self.assertRaisesRegex(ValueError, 'protobuf (field|tag)'):
adapter().envelope(signal_batch(9, ld(1, ld(1, b'\x00\x00'))))
def test_scope_metrics_scope_must_be_strictly_decoded(self):
scope = ld(1, b'literal-scope') + ld(2, b'1.0')
self.assertEqual(adapter().envelope(signal_batch(5, ld(1, ld(2, ld(1, scope)))))[2],
[literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)])
with self.assertRaisesRegex(ValueError, 'protobuf (field|tag)'):
adapter().envelope(signal_batch(5, ld(1, ld(2, ld(1, b'\x00\x00')))))
def test_schema_url_utf8_is_required_at_every_common_placement(self):
for signal, name in ((5, 'metrics'), (9, 'traces'), (6, 'logs')):
for placement in ('resource', 'scope'):
def export(url):
field = ld(3, url)
return ld(1, field if placement == 'resource' else ld(2, field))
with self.subTest(signal=name, placement=placement):
self.assertEqual(adapter().envelope(signal_batch(
signal, export(b'https://schemas.example.test/1.0')))[2],
[] if signal == 6 else [literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)])
with self.assertRaisesRegex(ValueError, 'UTF|utf'):
adapter().envelope(signal_batch(signal, export(b'\xff')))
def test_common_resource_keys_and_scope_names_are_utf8(self):
for signal in (5, 9):
for placement in ('resource-key', 'scope-name'):
def export(text):
return ld(1, ld(1, ld(1, ld(1, text) + ld(2, ld(1, b'host'))))
if placement == 'resource-key' else ld(2, ld(1, ld(1, text))))
with self.subTest(signal=signal, placement=placement):
self.assertEqual(adapter().envelope(signal_batch(signal, export(b'literal')))[2],
[literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)])
with self.assertRaisesRegex(ValueError, 'UTF|utf'):
adapter().envelope(signal_batch(signal, export(b'\xff')))
def test_legacy_mode_still_refuses_a_diagnostics_directory(self):
fixture = legacy.CustodyWitnessControls('test_clean_raw_snapshot_and_two_line_conservation')
fixture.module = adapter(); self.addCleanup(fixture.doCleanups); fixture.setUp()
with self.assertRaisesRegex(ValueError, 'unexpected or failed Spool input'):
fixture.snapshot('legacy-diagnostics', change=lambda root: (root / 'diagnostics').mkdir())
def test_extra_allowed_path_row_in_pending_batch_is_refused(self):
fixture = legacy.CustodyWitnessControls('test_clean_raw_snapshot_and_two_line_conservation')
fixture.module = adapter(); self.addCleanup(fixture.doCleanups); fixture.setUp()
baseline = fixture.chain(); fixture.module.conserve(*baseline)
rows = [literal_row(4, 0, SOURCE, 7, 9, 0, PENDING),
literal_row(4, 1, SOURCE, 7, 9, 0, b'extra-fixture-row\n')]
pending = payload(4, rows, [legacy.cursor(len(PENDING))])
unread = legacy.batch(5, body=UNREAD, offset=len(PENDING + UNREAD))
before = fixture.snapshot('extra-before', [pending])
installed = fixture.snapshot('extra-installed', [pending])
drained = fixture.snapshot('extra-drain', [pending, unread], ack=5,
receipt_change=lambda receipt: receipt.update(candidate_process=copy.deepcopy(PROCESS)))
events = copy.deepcopy(baseline[-1])
events[0]['message'] = ('delivery sequence=4 sha256=' + hashlib.sha256(pending).hexdigest()
+ ' status=ack committed_through=4 elapsed_us=1')
with self.assertRaisesRegex(ValueError, 'unexpected pending log row'):
fixture.module.conserve(before, installed, drained, baseline[3], events)
def test_predecessor_query_rows_cannot_substitute_label_or_strand(self):
# Source deduction: queries.before currently becomes its own expected baseline.
fixture = legacy.CustodyWitnessControls('test_clean_raw_snapshot_and_two_line_conservation')
fixture.module = adapter(); self.addCleanup(fixture.doCleanups); fixture.setUp()
chain = fixture.chain(); fixture.module.conserve(*chain)
for field, value in (('node', 'unrelated-enrollment'), ('node_id', '00' * 16)):
args = copy.deepcopy(chain)
for phase in ('before', 'after_drain', 'after_reboot'):
for row in args[3][phase][:3]: row[field] = value
self.assertEqual(args[0]['node_id'], NODE.hex())
self.assertEqual(args[0]['sources'], chain[0]['sources'])
self.assertEqual(args[4], chain[4])
with self.subTest(substitution=field), self.assertRaisesRegex(ValueError, 'old.row|baseline'):
fixture.module.conserve(*args)
class NativeShapeControls(NativeFixture, unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.module = adapter()
def setUp(self):
if 'fixture_paths' not in inspect.signature(self.module.stopped_snapshot).parameters:
raise unittest.SkipTest('NOT RUN: explicit native reader shape is not implemented')
NativeFixture.setUp(self)
def test_crc_valid_malformed_diagnostic_frame_is_rejected_before_projection(self):
fixture = literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)
prefix = vi(1, 1) + ld(2, NODE) + vi(3, 1) + vi(4, 4)
for name, bad, reason in malformed_diagnostics():
original = prefix + ld(6, ld(1, ld(2, ld(2, record(fixture)) + ld(2, bad))))
with self.subTest(diagnostic=name), self.assertRaisesRegex(ValueError, reason):
self.snapshot('malformed', [original], 3, PENDING + UNREAD)
def test_append_and_uncollected_stop_tail_preserve_full_authority(self):
chain = self.chain()
before, installed, drained, queries, _ = chain
self.assertEqual(before['fixture_paths'], (SOURCE,))
self.assertEqual(before['files'], installed['files'])
self.assertEqual(before['diagnostics']['spindle.log']['bytes'], DIAG_OLD + DIAG_PENDING)
self.assertEqual(drained['diagnostics']['spindle.log']['bytes'], DIAG_OLD + DIAG_PENDING + DIAG_NEW + STOP)
self.assertEqual(drained['logs'][DIAG]['bytes'], DIAG_OLD + DIAG_PENDING + DIAG_NEW + STOP)
self.assertEqual(drained['cursors'][DIAG]['offset'], len(DIAG_OLD + DIAG_PENDING + DIAG_NEW))
self.assertEqual(len(queries['before']), 4)
self.assertEqual(before['sources'][0]['rows'][0]['attributes']['log.file.path'], DIAG)
for source in before['sources'][:3]:
self.assertEqual({current['path'] for current in self.module.envelope(source['payload'])[1]},
{SOURCE, DIAG})
self.assertIn(ld(7, source_cursor(DIAG, before['logs'][DIAG]['device'],
before['logs'][DIAG]['inode'], DIAG_OLD + DIAG_PENDING, len(DIAG_OLD))),
before['sources'][0]['payload'])
self.assertEqual(len(queries['after_drain']), 8)
self.assertEqual(sum(row['attributes']['log.file.path'] == SOURCE
for row in queries['after_drain']), 5)
self.assertEqual(len(drained['sources'][-1]['rows']), 2)
self.assertEqual(drained['sources'][3]['payload'], before['sources'][3]['payload'])
self.assertEqual(drained['sources'][3]['sha256'], before['sources'][3]['sha256'])
self.module.conserve(*chain)
def test_source_path_not_body_selects_the_five_fixture_rows(self):
for body in (PENDING, UNREAD):
with self.subTest(body=body):
chain = self.chain(diag_pending=body, diag_line=body)
self.assertEqual(sum(row['body'] == body[:-1].decode()
for row in chain[3]['after_drain']), 3)
self.assertEqual(chain[0]['sources'][3]['rows'][1]['body'], body[:-1].decode())
self.module.conserve(*chain)
self.temp.cleanup(); self.setUp()
def test_one_observed_rotation_preserves_prior_archived_inventory(self):
chain = self.chain(rotate=True)
before, _, drained, _, _ = chain
self.assertEqual(len(before['diagnostics']['spindle.log']['bytes']), CAP)
self.assertEqual(before['diagnostics']['spindle.log.1']['bytes'], b'event previous-generation\n')
self.assertEqual(drained['diagnostics']['spindle.log.1']['bytes'],
before['diagnostics']['spindle.log']['bytes'])
self.assertEqual(drained['diagnostics']['spindle.log.1']['inode'],
before['diagnostics']['spindle.log']['inode'])
self.assertNotEqual(drained['diagnostics']['spindle.log']['inode'],
before['diagnostics']['spindle.log']['inode'])
self.module.conserve(*chain)
def test_rewritten_diagnostic_prefix_is_refused(self):
with self.assertRaisesRegex(ValueError, 'diagnostic.*(prefix|rotation|preserv)'):
self.module.conserve(*self.chain(rewritten=True))
def test_unjustified_or_unobserved_rotation_cannot_replace_prior_prefix(self):
for options in (dict(rotate=True, rotation_full=False),
dict(rotate=True, rotated_old=b'event different-generation\n'),
dict(rotate=True, rotated_replace=True),
dict(rotate=True, rotation_size=CAP - len(DIAG_NEW)),
dict(rotate=True, diag_line=b'x' * 4096 + b'\n'),
dict(prior_old=True, old_rewrite=True)):
with self.subTest(options=options), self.assertRaisesRegex(
ValueError, 'diagnostic.*(prefix|rotation|preserv)'):
self.module.conserve(*self.chain(**options))
self.temp.cleanup(); self.setUp()
def test_unchanged_old_diagnostic_file_is_preserved_without_rotation(self):
chain = self.chain(prior_old=True)
self.assertEqual(chain[0]['diagnostics']['spindle.log.1'],
chain[2]['diagnostics']['spindle.log.1'])
self.module.conserve(*chain)
def test_every_pending_raw_log_row_is_in_both_complete_query_answers(self):
baseline = self.chain(); self.module.conserve(*baseline)
for phase in ('after_drain', 'after_reboot', 'both'):
for mutation in ('drop', 'duplicate', 'rewrite'):
args = copy.deepcopy(baseline)
for selected in (('after_drain', 'after_reboot') if phase == 'both' else (phase,)):
rows = args[3][selected]
index = next(index for index, row in enumerate(rows)
if (row['sequence'], row['index']) == (4, 1))
if mutation == 'drop': rows.pop(index)
elif mutation == 'duplicate': rows.append(copy.deepcopy(rows[index]))
else: rows[index]['body'] = 'rewritten-diagnostic'
with self.subTest(phase=phase, mutation=mutation), self.assertRaisesRegex(
ValueError, 'exact row conservation'):
self.module.conserve(*args)
def test_extra_allowed_path_row_in_native_pending_batch_is_refused(self):
with self.assertRaisesRegex(ValueError, 'unexpected pending log row'):
self.module.conserve(*self.chain(extra_pending=True))
def test_fixture_source_gap_in_pending_or_unread_batch_is_refused(self):
allowed = self.chain(pending_gap=('log source unavailable ' + DIAG + ': unavailable',))
self.assertEqual(allowed[0]['sources'][3].get('collection_gaps'),
['log source unavailable ' + DIAG + ': unavailable'])
self.module.conserve(*allowed)
for phase in ('pending_gap', 'unread_gap'):
self.temp.cleanup(); self.setUp()
with self.subTest(phase=phase), self.assertRaisesRegex(ValueError, 'fixture.*gap'):
self.module.conserve(*self.chain(**{phase: ('log truncated; previous tail unknown: ' + SOURCE,)}))
def test_legacy_frame_and_failure_markers_are_also_refused_in_native_mode(self):
item = legacy.batch()
changes = (
lambda root: (root / 'batches.faj').write_bytes(frame(item)[:-1]),
lambda root: (root / 'batches.faj').write_bytes(frame(item) + b'F'),
lambda root: (root / 'batches.faj').write_bytes(frame(item)[:16] + b'X' + frame(item)[17:]),
lambda root: (root / 'recovery-required').write_bytes(b'failure'),
lambda root: (root / 'append-in-progress').write_bytes(b'interrupted'),
lambda root: (root / 'acked').write_bytes(b'FAK1' + struct.pack('<QQ', 1, 2**64 - 1)),
lambda root: (root / 'acked').write_bytes(b'FAK1' + struct.pack('<QQ', 2, 0)))
for index, change in enumerate(changes):
with self.subTest(change=index), self.assertRaises(ValueError):
self.snapshot('native-marker', [item], 3, PENDING + UNREAD, change=change)
self.temp.cleanup(); self.setUp()
def test_native_root_requires_private_diagnostic_directory_and_active_file(self):
item = legacy.batch()
for variant in ('missing-directory', 'missing-active', 'mode', 'symlink'):
if variant.startswith('missing'):
self.active.unlink()
if variant == 'missing-directory': self.diagnostics.rmdir()
elif variant == 'mode': self.diagnostics.chmod(0o755)
else:
outside = self.root / 'outside-diagnostics'; self.diagnostics.replace(outside)
self.diagnostics.symlink_to(outside, target_is_directory=True)
with self.subTest(directory=variant), self.assertRaisesRegex(ValueError, 'diagnostic'):
self.snapshot('required-directory', [item], 3, PENDING + UNREAD)
self.temp.cleanup(); self.setUp()
def test_diagnostic_archive_must_match_actual_in_root_bytes_and_identity(self):
item = legacy.batch()
for variant in ('bytes', 'device', 'inode'):
def change(logs):
if variant == 'bytes': Path(logs[DIAG]['archive_path']).write_bytes(b'forged-diagnostic\n')
else: logs[DIAG][variant] += 1
with self.subTest(archive=variant), self.assertRaisesRegex(ValueError, 'diagnostic.*(archive|source)'):
self.snapshot('mismatched-archive', [item], 3, PENDING + UNREAD, archive_change=change)
def test_diagnostic_only_pending_batch_still_needs_original_byte_ack(self):
chain = self.chain(diag_only_pending=True)
self.assertEqual(chain[0]['sources'][-1]['sequence'], 5)
self.assertTrue(all(row['attributes']['log.file.path'] == DIAG
for row in chain[0]['sources'][-1]['rows']))
self.module.conserve(*chain)
events = [event for event in chain[-1] if not event['message'].startswith('delivery sequence=5 ')]
with self.assertRaisesRegex(ValueError, 'missing candidate exact-byte ACK'):
self.module.conserve(*chain[:-1], events)
def test_fixture_prefix_gap_and_extra_fixture_query_row_are_refused(self):
with self.assertRaisesRegex(ValueError, 'source prefix conservation'):
self.module.conserve(*self.chain(lost_prefix=True))
self.temp.cleanup(); self.setUp()
chain = self.chain()
extra = copy.deepcopy(chain[3]['after_drain'][0]); extra['body'] = 'unexpected-fixture-row'
chain[3]['after_drain'].append(extra)
with self.assertRaisesRegex(ValueError, 'exact row conservation'):
self.module.conserve(*chain)
def test_invalid_or_substituted_fixture_path_registration_is_refused(self):
row = literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)
item = payload(4, [row], [source_cursor(SOURCE, 7, 9, PENDING + UNREAD, len(PENDING))])
for paths in ((), [SOURCE], (SOURCE, SOURCE), ('relative.log',), ('/missing.log',)):
with self.subTest(paths=paths), self.assertRaisesRegex(ValueError, 'fixture.*path'):
self.snapshot('invalid', [item], 3, PENDING + UNREAD, fixture_paths=paths)
chain = self.chain()
chain[2]['fixture_paths'] = (DIAG,)
with self.assertRaisesRegex(ValueError, 'fixture.*path'):
self.module.conserve(*chain)
def test_unexpected_aliased_or_nonprivate_diagnostic_input_is_refused(self):
row = literal_row(4, 0, SOURCE, 7, 9, 0, PENDING)
item = payload(4, [row], [source_cursor(SOURCE, 7, 9, PENDING + UNREAD, len(PENDING))])
for variant in ('unexpected', 'symlink', 'hardlink', 'mode', 'size',
'uid', 'gid', 'uid-float', 'gid-float'):
receipt_change = None
with self.subTest(diagnostic=variant), self.assertRaisesRegex(ValueError, 'diagnostic'):
if variant == 'unexpected':
(self.diagnostics / 'other.log').write_bytes(b'unregistered\n')
elif variant == 'symlink':
outside = self.root / 'outside'; outside.write_bytes(DIAG_OLD)
self.active.unlink(); self.active.symlink_to(outside)
elif variant == 'hardlink':
os.link(self.active, self.root / 'alias')
elif variant == 'mode':
self.active.chmod(0o644)
elif variant == 'size':
self.active.write_bytes(b'x' * (CAP + 1))
else:
field = variant.removesuffix('-float')
value = os.geteuid() if field == 'uid' else os.getegid()
value = float(value) if variant.endswith('-float') else value + 1
receipt_change = lambda receipt: receipt['service_identity'].update(
{field: value})
self.snapshot('unsafe', [item], 3, PENDING + UNREAD, receipt_change=receipt_change)
self.temp.cleanup(); self.setUp()
if __name__ == '__main__':
unittest.main()