Experiment formal installation incremental cache plan - kmosoti/FabricO11y GitHub Wiki
Preparation only. Miso ran no import, syntax check, inventory, compiler/control, validator, qualification workload or deletion. Commands below are prospective; Olo owns review, contained execution, board replies and publication. This addresses C-037, C-042 and C-aud-9. It carries C-aud-8 as input, not admission of a suspended proposal.
Archived before editing:
- reclaim-incremental-cache-before-installation-first-e8f78e08.py: SHA256 e8f78e08fa46cd504d4bdbb17ff928790f03c0555ebc4d6dc9aec27bfef41022.
- incremental-cache-capacity-plan-before-installation-first-6d186798.md: SHA256 6d1867984b006e03fe37091be9b2cd53a254cc36e17fa28575a40f65fef90bca.
- incremental-cache-rebuild-controls-before-installation-first-595e93fc.py: SHA256 595e93fc9466038b2f9d6303a278a37b78d6683a3d071d2b9152d9fe73db4b5b.
- Actual prior global metadata cargo-rustc-info-before-installation-first-01.json: SHA256 00e1cf91094d33a858bbd88766480576212398c4092ecb4db9a5034585fa6fcf. Observed Rust1.101 nightly; retain unchanged as history.
The exact unexecuted X069 helper/control/plan are preserved as reclaim-incremental-cache-before-per-file-compiler-4b278c61.py, incremental-cache-rebuild-controls-before-per-file-compiler-a7e9f1c5.py and incremental-cache-capacity-plan-before-per-file-compiler-644a7333.md. Their previous hashes remain history; the following proposal supersedes their compiler prerequisite only.
Before this C-aud9 re-pin, exact unexecuted X072 drafts were archived as reclaim-incremental-cache-before-admission-review-9e0b6375.py (SHA256 9e0b63758e2fe9df2c57a9f6b0a6fadf0a2d2af40c69d3b3735c1bf7aef34b58), incremental-cache-rebuild-controls-before-admission-review-e875e0a0.py (e875e0a07a35b35416adbeb4c278095802c040968d939312b6a62b3b22e03c51) and incremental-cache-capacity-plan-before-admission-review-3c6fb5cb.md (3c6fb5cb85a21dac7bf07a0082a61c253a1cff36d5e2f88f882e95e129f6fa1c). These old bytes remain HOLD/NOT RUN. This proposal addresses B1/S1–S3 and all four nits together.
Purpose: #41 installation first, unchanged 9,663,676,416-byte reservation, strict DATA census maximum 85,336,323,584 bytes, unchanged 1,073,741,824-byte output margin, unchanged 5,368,709,120-byte removal cap, and sufficient real free disk. Census plus reservation stays at most95,000,000,000 bytes. Actual goal: max(0, actual_data_bytes - 85,336,323,584 + 1,073,741,824). Skip threshold: 84,262,581,760 bytes. No manually supplied minimum or projected cleanup substitutes for actual in-script census.
#43 is separate and unadmitted. Its unchanged13,958,643,712-byte reservation and81,041,356,288-byte census maximum require a fresh decision/census after #41 output growth. This plan makes no cross-family admission claim.
Actual separate recovery is now complete: stopped recovery results observed post-execute census89,180,770,008 bytes and statvfs available401,753,972,736 bytes; actual #41 goal at that snapshot4,918,188,248 bytes. This fits the unchanged5GiB cap but is not an eligible-capacity claim or a substituted inventory census. Inventory must measure its own fresh census/goal and report eligibility/exclusions.
C-042 corrects the compiler prerequisite: no full shared-target compile is required or admitted for cache cleanup. Exact per-member RSIC Rust1.99 writer framing determines compiler eligibility. The global .rustc_info.json is archived/read and hash-bound only as last-invocation history; its version is not a per-session compiler or revision claim. The before-run nightly snapshot and the actual interrupted1.99 run remain separate historical observations. No restamp, fallback compile or historical-head pass reuse. #64 exact-head hosted CI is independent of this cleanup. The interrupted run's new executable recovery is a separate pending scope; post-recovery actual census is posted before this inventory.
Require actual successful stopped #40 cell18 fresh-8fb6f413179b-01-package-inspect, clean frozen A at8fb6f413179b094288316e25845c137440ba23fa, exact candidate-admission record, no active compiler/server/VM/other project unit and unchanged storage/cgroup/source guards. Pending cleanup does not count until a new census observes it.
Bound audit incremental-cache-read-only-audit-02.json SHA25649ec16e5f2b5996876abaec74f49fa273046aada5ca14e3cd4524b75cc057c08 remains inspection only. Its earlier5,585,382,257-byte single-link-bin count and later read-only Rust1.99=5,576,784,063/nightly=207,376,669-byte header classes do not establish current eligible capacity. Nightly sessions, packages including supersededca84/a5, executables/libraries, sources, bundles, fingerprints/dep-info, receipts, failures, images, other session members and every hardlink remain excluded/preserved. Current ambiguity exclusions, including fabric_server and delivery, stay unchanged. Historical compiled revision is not attested by the declared-target/source witnesses.
Select whole finalized sessions only. Every present admitted name (dep-graph.bin, query-cache.bin, work-products.bin) passes the existing path/owner/regular/single-link/non-executable/same-device/compiler/unambiguous-target/fingerprint/source/dep-info gates. One ineligible member excludes the whole session. Preserve all other names/directories. Existing largest-first selection is unchanged; C-aud-8's non-newest-first efficiency suggestion is deferred, not claimed implemented.
Headers now require exact RSIC + little-endian format0 + one-byte exact version length + exact Rust1.99 version, following pinned writer and reader. A correct version substring inside malformed/different framing is rejected. This strengthens cache-kind checks without admitting nightly files.
The exact 29-byte header version is 1.99.0 (b940084d7 2026-09-28), preceded by RSIC, format0 and length byte29. Residual: nightly/bootstrap's deliberate RUSTC_FORCE_RUSTC_VERSION test hook can spoof the writer string; per-file framing is not adversarial compiler attestation or historical revision provenance. Controls refuse a supplied hook; historical hook use remains unverified. Global .rustc_info.json remains last-invocation history only.
Expected unit fingerprint source (S1): both actual same-helper compiler modes archive each synthetic Rust1.99 unit JSON as <case>-compiler-fingerprint.json, recording raw SHA256 and its actual integer rustc field. All three actual fixtures and both stopped mode receipts must agree. Inventory binds these exact reports/archives/helper/control/stop hashes and takes expected_unit_fingerprint_rustc only from those unit fields, never global .rustc_info.json or a copied numeric literal. A real target's hash-bound Cargo unit JSON rustc must equal that actual expected field; merely being an integer is insufficient. Existing target/source/dep-info/ambiguity gates stay. Missing or inconsistent actual controls block inventory.
Hold existing read-only O_NOFOLLOW nonblocking exclusive flock on native session locks through revalidation/removal. Before first unlink recheck all members/witnesses, mounts, idle/stop conditions and whole-session membership. Remove graph first; before either remaining member require graph absent. Keep unchanged held-file directory-FD-relative unlink, immediate full identity/hash checks and own nlink-to-zero/ctime transition check. No directory removal, hardlink deletion, metadata rewriting, auto-retry or widened selection. Preserve exclusive reports, fsynced progress journal and all partial failures; attempt fresh census after partial removal.
Mandatory contained controls retain C-aud-6's V0 hash5f714433ee20cbe09fa949a31a7441a21208774db144bf531fd973ff91797909, E1, edition2024/default dev/CARGO_INCREMENTAL=1. Baseline145/3/9/V0; whole-session and interrupted-after-graph positives rebuild0 to155/3/9/V1 and changed binary hash. Query-only negative requires101, failed-to-load-disk-cached-value-for-green-node diagnostic and stale binary, with raw/failing fixture preserved. T/E2/repeated cases and real shared-cache reconstruction are not claimed. Control roots are fresh owned scratch targets; real CACHE stays DATA/cargo/debug/incremental. A synthetic target cannot be/contain shared CACHE.
Both compiler modes require five literal wrong-format, wrong-length, embedded-version, wrong-version-with-correct-substring and correctly framed nightly-writer refusals. For every actual fixture in both modes, literal omitted-query-member and forced query-before-graph order refuse through the actual locked_inventory/selection contract before the first unlink. Two additional copied native-session fixtures inject (i) a nightly-framed graph alongside 1.99 query/work members and (ii) nlink2 graph alongside single-link 1.99 members. The reusable actual inventory session_candidate decision must exclude the whole session, record KIND/HARDLINK as its first failed gate, attempt zero mutations and leave all three injected fixture files identity/hash-identical. Before/after raw snapshots, exact refusal reasons and stub observations enter each mode's report; restore no ineligible member into a selected session. The positive original compiler fixtures and the independent query-only ICE101 fixture remain unchanged.
Full raising boundary set (S2): os.unlink, os.remove, os.rmdir, os.removedirs, os.rename, os.replace, os.truncate, and write-mode os.open, builtins.open, io.open, Path.open. Every fixture probes every entry point and requires its named raising stub, with all three files unchanged; real admission/order/header refusals then require zero boundary invocations under that full set. The scoped executable-path control adds its deletion callback to the same full set. Read-only opens remain available. These are active exceptions before mutation, not a boolean that suppresses deletion.
Observed excluded capacity (S3): inventory records each excluded session's first failed gate and lstat-measured regular-name logical path bytes, grouped by that first gate. Hardlinked paths contribute their measured size as observations, never freeable bytes. Working sessions are explicitly reported/preserved; orphan working hardlinks may exclude a finished session and cost eligible capacity. This plan never removes them; any separate recovery touching working sessions must follow rustc's native lock protocol. Nonregular/unreadable names have bytes: null and an explicit unknown count; groups report measured partial sums plus null totals where incomplete. Uninspectable target roots report null session counts/bytes. Early uncomputed measurements remain null. Eligible whole-session bytes and excluded observed bytes are distinct. Largest-first selection is unchanged; temporary or permanent exclusion is not guessed from bytes alone.
Helper SHA256 242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0; control SHA256 1d8b8803223fbaa23fce8d10480a41ceb50a7f8378d50f7b5965015da693e99a. Review exact bytes first. Every cell uses serial A driver,1GiB archive reservation/default16/20GiB memory/no swap/1800s outer. Read #59 before each cell and after every run; verify raw exit/stop receipts. No command below was run by Miso.
bash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
RECLAIM_SCRIPT="$RECLAIM_META/reclaim-incremental-cache.py"
RECLAIM_CONTROL="$RECLAIM_META/incremental-cache-rebuild-controls.py"
test "$(sha256sum "$RECLAIM_SCRIPT" | cut -d ' ' -f1)" = 242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0
test "$(sha256sum "$RECLAIM_CONTROL" | cut -d ' ' -f1)" = 1d8b8803223fbaa23fce8d10480a41ceb50a7f8378d50f7b5965015da693e99a
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-syntax-01 1073741824 python3 -B -c 'from pathlib import Path; import hashlib; m=Path("/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01"); pairs=[("reclaim-incremental-cache.py","242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0"),("incremental-cache-rebuild-controls.py","1d8b8803223fbaa23fce8d10480a41ceb50a7f8378d50f7b5965015da693e99a")]; [(None if hashlib.sha256((m/n).read_bytes()).hexdigest()==s else (_ for _ in ()).throw(ValueError("script hash differs")),compile((m/n).read_bytes(),str(m/n),"exec")) for n,s in pairs]; print("syntax/hash checks completed")'
EOFRead board and actual syntax/stop receipts before each separate next block:
bash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-rebuild-positive-01 1073741824 python3 -B "$RECLAIM_META/incremental-cache-rebuild-controls.py" --expected-script-sha256 242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0
EOFbash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-rebuild-negative-01 1073741824 python3 -B "$RECLAIM_META/incremental-cache-rebuild-controls.py" --expected-script-sha256 242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0 --negative-only
EOFAfter mandatory controls, separate recovery's actual posted census and the cell18/clean-source/idle prerequisites:
bash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
test "$(sha256sum "$RECLAIM_META/reclaim-incremental-cache.py" | cut -d ' ' -f1)" = 242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-inventory-01 1073741824 python3 -B "$RECLAIM_META/reclaim-incremental-cache.py" --inventory
EOFInventory report installation-incremental-cache-inventory-result-01.json outcomes: not_needed_bound_census/exit0 (skip cleanup); selected/exit0 (exclusive installation-incremental-cache-capacity-inventory-01.json); unresolved/exit1 before removal (preserve failure and post BLOCKED with actual census/goal, eligible sizes and measured exclusion groups; uncomputed values remain null). Require the actual result report and stopped receipt: exit1 without a valid report is a crash, not the controlled unresolved state. Goal over5GiB or inadequate eligible whole-session capacity is unresolved. Control/dry/execute refuse anything except exact selected #41 v4 schema/purpose/reviewed digest, including actual unit-fingerprint bindings. Review actual eligible/excluded sessions, canonical paths, source/audit/member/lock/order/bytes and stopped receipts. No measured excluded path bytes are presented as eligible/freeable capacity.
Replace each placeholder below with the reviewed literal inventory SHA256. Placeholder fails before launching. Read board after control and launch dry as a separate block:
bash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
RECLAIM_INVENTORY_SHA=REPLACE_WITH_REVIEWED_LITERAL_INVENTORY_SHA256
[[ "$RECLAIM_INVENTORY_SHA" =~ ^[0-9a-f]{64}$ ]]
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-control-01 1073741824 python3 -B "$RECLAIM_META/reclaim-incremental-cache.py" --negative-control --expected-inventory-sha256 "$RECLAIM_INVENTORY_SHA"
EOFbash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
RECLAIM_INVENTORY_SHA=REPLACE_WITH_REVIEWED_LITERAL_INVENTORY_SHA256
[[ "$RECLAIM_INVENTORY_SHA" =~ ^[0-9a-f]{64}$ ]]
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-dry-01 1073741824 python3 -B "$RECLAIM_META/reclaim-incremental-cache.py" --expected-inventory-sha256 "$RECLAIM_INVENTORY_SHA"
EOFRead board and review zero removals, literal INCREMENTAL-CACHE-SCOPE refusal, unchanged inventory/helper hashes, whole-session/graph-first assertions and actual stopped receipts before execute. Scope control changes only an in-memory path to DATA/cargo/debug/fabric-server; callback/raw mutation/write boundaries raise. Real inventory remains unchanged.
bash -euo pipefail <<'EOF'
RECLAIM_META=/run/media/kmosoti/data/FabricO11y/results/candidate-build-fresh-8fb6f413179b-01
RECLAIM_INVENTORY_SHA=REPLACE_WITH_REVIEWED_LITERAL_INVENTORY_SHA256
[[ "$RECLAIM_INVENTORY_SHA" =~ ^[0-9a-f]{64}$ ]]
test "$(sha256sum "$RECLAIM_META/reclaim-incremental-cache.py" | cut -d ' ' -f1)" = 242742e2f8e1ff5c238f0e7d118a6a55537545661d6f1bc348b4b259ab453ea0
test "$(sha256sum "$RECLAIM_META/installation-incremental-cache-capacity-inventory-01.json" | cut -d ' ' -f1)" = "$RECLAIM_INVENTORY_SHA"
python3 -B "$RECLAIM_META/run-step-a.py" installation-incremental-cache-execute-01 1073741824 python3 -B "$RECLAIM_META/reclaim-incremental-cache.py" --execute --expected-inventory-sha256 "$RECLAIM_INVENTORY_SHA"
EOFExecute binds current helper/control/raw compiler/archive/stop receipts, scope control, dry-run, inventory, cell18/admission/source/resource witnesses. Before opening exclusive progress journal or removing anything, repeat census and refuse if current goal exceeds immutable selection. If census has dropped, execute still removes the whole immutable reviewed selection, within its original5GiB cap; it never widens or partially reselects sessions. New reports/journal use installation-incremental-cache- prefix. Record actual post-removal census and future_installation_admissible_now; this observation is not VM admission. Repeat full9GiB reservation/free-disk/strict census/idle checks immediately before each #41 cell. Failed/interrupted/partial runs remain preserved; no automatic retry. #43 needs a separate future proposal/admission. Publish compact actual B1 raw refusal/snapshot/stub outcomes with stopped receipt hashes after they run; do not claim success from this prospective text.