Historical report imported 2026-10-10 from docs/experiments/formal/server-self-observation-findings.md. Original SHA-256: 51bfcd5a31e89f5271d24264bcaf354cb159411b49ff2d07086897020a3e06ca. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Dedicated Spindle delivery and process ownership

The production server CLI now owns a dedicated Spindle with a separate credential and persistent Spool. Edge Spindles keep their configured destination and collect their own diagnostic file. This record follows the separately committed protocol (local checkout reference docs/experiments/formal/server-self-observation-protocol.md; not present at origin/main) and ADR-0026 (local checkout reference docs/decisions/ADR-0026-launch-a-dedicated-spindle-with-each-server.md; not present at origin/main). It is a finite correctness investigation, not installation qualification or a measurement of overhead under load.

Invariants and failure discovered

The child uses ordinary verified HTTPS, FAB1 Batches, Spool commits and durable server ACKs. Remote configuration cannot remove local diagnostic pins. Restart reuses credentials and Strand identity. The supervisor owns child cleanup and fails if that child exits unexpectedly. Logs have fixed bounds and do not emit per-Batch/ACK events. Admin pause retains its existing collection semantics.

Static review found a credential collision error in the first implementation: checking only the active authentication map allowed a token assigned to a revoked different identity to enroll as fabric-server-self. The deterministic revoked_other_identity_cannot_reuse_its_credential_under_the_self_name regression failed before the repair (exit 101; seven other tests succeeded). Enrollment now checks all persisted credential hashes, including revoked identities. The same eight tests then exited 0. The counterexample is retained in self_identity.rs (local checkout reference crates/fabric-server/tests/self_identity.rs; not present at origin/main).

Review also distinguished malformed server TLS from a valid certificate with an incorrect hostname. The managed Spindle now requires an authenticated startup configuration poll. A wrong SAN makes the child exit and its supervisor stop; ordinary edge nodes still retry an unavailable server. No wire or durability ordering was changed.

Executed checks

All commands use python3 -B tools/resource_group.py -- (or --delegate -- for the native script), with CARGO_BUILD_JOBS=2 for Cargo. The launcher routes temporary/build paths to $FABRIC_DATA; three legacy integration-fixture paths discovered in the full suite were also moved from the checkout's target to FABRIC_SCRATCH_ROOT. Exact commands and exits are retained in the launcher receipts (local evidence omitted from stage). The table names their command suffixes. Toolchain: Rust/Cargo 1.99.0, Python 3.14.7; debug binaries and existing offline dependency cache.

Check	Outcome
cargo check --offline --workspace --all-targets	Exit 0; two existing research-example unused-variable warnings
cargo test --offline --locked -p fabric-server --test self_identity, before repair	Exit 101; revoked-token regression rejected the implementation
Same identity tests, after repair	Exit 0; eight tests
cargo test --offline --locked -p fabric-adapter-linux -p fabric_o11y -p fabric-server --lib --bins	Exit 0; 111 tests passed, three registered research probes ignored
cargo build --offline --locked --workspace --bins	Exit 0
python3 -B tools/self_observation_smoke.py --out $FABRIC_DATA/results/self-observation-01	Exit 0
Same script with --out $FABRIC_DATA/results/self-observation-02, after extending the harness	Exit 0; timer boundary and default TLS trust covered
cargo xtask checks --profile fast, first run	Exit 1; sixteen checks passed, workspace test gate failed on the old CLI log-count expectation
Focused old sigterm_stops_run_between_cycles_and_leaves_a_reopenable_spool test	Exit 101; four committed records versus the former one-record expectation
cargo test --offline --locked -p fabric_o11y --test spindle --test spool --test log_source, first fixture-edit attempt	Exit 101 at compilation; unnecessary u64 cast disagreed with the inspection count's usize
Same three integration targets after removing that cast	Exit 0; 33 tests, including exact application/diagnostic bodies and shutdown/reopen
Full fast rerun self-observation-fast-02	Incomplete: service-bus disconnect; only fmt and check receipts exist despite outer launcher exit 0
cargo xtask checks --only test --receipts $FABRIC_DATA/results/self-observation-test-03	Exit 0; full cargo test --workspace --locked --all-features gate completed in 240,044 ms
cargo xtask checks --only clippy --receipts $FABRIC_DATA/results/self-observation-clippy-03	Exit 0; all targets/features with warnings denied
python3 -B tools/bench/labs/cross_system/sweep_documentation.py --out $FABRIC_DATA/results/self-observation-docs-01	Exit 0; all three manual documentation checks passed; temporary authenticated Bun copy removed, hooks/CI remain disabled

The receipt index (local evidence omitted from stage) selects completed evidence for every one of the seventeen fast checks: unchanged gates from run 01, fresh formatting/compilation from run 02, and the complete test and Clippy recovery invocations. The failed and incomplete original runs remain preserved. Manual documentation receipts (local evidence omitted from stage) are separate from application correctness evidence.

The incomplete rerun is preserved as an environment interruption. Its launcher reported a bus disconnect and returned zero; the command's required receipt set was absent. The user-service journal shows the unit stopped before the workspace test produced a receipt. No success is inferred from that outer exit. The launcher's reliance on the service wrapper exit is a remaining tooling limitation; these results require inner check receipts or a complete native result as well.

The first native trial compared five-line and fourteen-line retained producer prefixes against real queried bodies. Each comparison rejected dropped, changed and repeated rows; an absent predicate returned zero rows. Both components' process samples were present. Empty remote desired logs did not remove the pins after restart, and token/Spool identity remained unchanged. Duplicate launch was refused while the first service stayed available. Normal stop, child kill and abrupt parent kill left no live companion. Missing executable, invalid server TLS, wrong SAN, wrong/missing token and revoked token cases failed as intended.

The native child cgroup covered server and Spindle together with memory.max=3999997952, memory.high=3200000000, no swap, two CPU equivalents and 256 tasks. The first trial recorded 7,516,160 bytes peak charged cgroup memory and no memory high/max/OOM events. This is not process RSS: existing shared file pages may already be charged elsewhere. The fixture was tiny and short, so this is containment evidence, not a capacity or overhead claim.

The extended native receipt (local evidence omitted from stage) records four exact retained prefixes of 6, 8, 16 and 42 lines (overlapping prefixes, not 72 unique observations), twelve rejected body corruptions and four empty absent-predicate results. Repeated queries over sixteen quiet seconds produced exactly one additional periodic sample per component, with no recursive log growth. A separate self-signed certificate worked with the default trust input and no self_spindle_ca override. All lifecycle/credential controls were repeated. Observed peak process RSS was 19.867 MiB server and 8.496 MiB Spindle; charged group peak was 8,044,544 bytes. Group CPU across this roughly twenty-second multi-start trial was 0.334726 seconds; fixture files peaked at 82,019 bytes. There were no memory high/max/OOM events and no swap. These are tiny debug-fixture observations, with shared pages and caches, not incremental production overhead.

Both native runs removed owned fixture trees and child cgroups. Their raw stderr, queries, producer prefixes, source/binary/harness hashes and resources are retained under native run 01 (local evidence omitted from stage) and native run 02 (local evidence omitted from stage); originals remain on the data drive. The cleanup receipt (local evidence omitted from stage) records removal of empty failed-run directories and preservation of 162 bytes of earlier full-suite failure fixtures. Shared build caches are retained. The inspected data tree occupied about 31.4 decimal GB before the final checks and 44.35 GB after compilation/verification caches grew, below the 100 GB ceiling. This is a sampled footprint, not a filesystem quota.

Limits and remaining work

The lower-level library serving primitive remains explicitly composed by its caller. Diagnostics are best effort before Spool commit: rotation can discard uncollected lines, with existing gap reporting, and the final shutdown marker can wait for restart. A colocated collector cannot independently report a host outage while that host is down. Installed systemd behavior, long-duration resource use and memory/latency overhead under representative load remain unqualified. Process RSS/CPU samples are currently log bodies; existing host and output-meter data retain their ordinary OTLP metrics.

Future production-CLI performance fixtures must install both binaries, configure the companion's local TLS trust, and account for its telemetry and resources. Historical isolated-server comparisons remain attached to their original source and binary hashes. A deliberately embedded library test can still compose only the server, but cannot stand in for the new complete production process topology.