Experiment formal encoded page memory run 01 - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/formal/encoded-page-memory-run-01.md. Original SHA-256: b987f767aa98eeaa49067cccfb4c04a1986bf88f45406407e332b827c2e0c4a6. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Encoded-page memory counterexample and disk-store diagnostic

Observed 2026-10-09 under the prospective protocol (local checkout reference docs/experiments/formal/encoded-page-memory-protocol.md; not present at origin/main). These are two diagnostic pairs, a finite fault screen and the subsequently registered repeated campaign, not a promotion or universal heap claim. The original aligned candidate exceeded the 80 MiB probe ceiling; the separately frozen disk-store candidate stayed below it while preserving the required exact output.

Frozen fixture and measured outcomes

Both pairs used seed 0xA11FA001, exactly 8192 distinct 16384-byte base64 log bodies, 256 ordinary batches and one full 8192-row physical log group. The sealed journal was 134415362 bytes with SHA256 76ea6009cfd27e93e67736a2051250e90050c56b69916f6ebc02265140236510. Each generation streamed 32 rows at a time; generation and differential readback were excluded from the counted build boundary.

Observation Aligned in-memory store Aligned disk store
Candidate incremental counted peak heap 143075502 bytes 42011899 bytes
Reference incremental counted peak heap 647059826 bytes 647059826 bytes
Candidate build elapsed time, single observation 3.369718811 s 4.452622721 s
Candidate /proc/self/io rchar delta 269199111 471765567
Candidate /proc/self/io wchar delta 337562272 540129583
Candidate /proc/self/io write_bytes delta 337612800 540848128
Candidate /proc/self/io cancelled_write_bytes delta 134815744 338051072

The ceiling is 83886080 bytes. Both pairs satisfied unchanged comparator checks for input, ordered logical rows, manifest semantics, applicable Parquet bytes, physical filter validity, pruning and no leftovers. Additional checks verified fixture counts, one full log group, exact filter bytes, counted allocator and unchanged frozen sources/input. Only the first pair's heap check failed. logs.parquet was exactly 101041273 bytes in all four builds, SHA256 5735ebb05b6ded206758671c1813c1e6797b5153b898786444279aa95b247fb0.

The disk backend counted 101019985 completed serialized log blob bytes, with exactly that peak outstanding payload, largest blob 788900 bytes, 284 puts and 284 takes, zero outstanding bytes and successful cleanup. The counted heap reduction was 101063603 bytes, only 43618 bytes different from the completed log payload removed from resident storage. This isolated intervention and byte count support the completed-page explanation over a full-group dictionary explanation. The raw table also spilled 101546471 blob bytes (884 puts/takes), with a much smaller 2778426-byte peak because its original chunk flush policy remains.

Logical page scratch writes totaled 202566456 bytes across tables. The rchar increase was exactly that amount; the wchar increase was that amount plus 855 bytes of instrumented stderr observations. These are process I/O counters and logical backend bytes, not a measurement of physical device traffic. The existing .run- observer measures row-run spill only and does not include page scratch. Single ordered pairs support the finite heap counterexample and correction; they establish no timing improvement or general performance comparison.

Executed verification and receipts

Every workload used python3 tools/resource_group.py -- ..., enforced 20 GiB memory maximum and zero swap, and exited 0. Exact argv, build settings, source copies/hashes, frozen binaries, per-child stdout/stderr/exit and raw measurements are retained on the data drive:

  • $FABRIC_DATA/results/readiness-continuation-encoded-page-01 (unit fabric-work-a239e0059636442480ad6831068b2368, 24.719 s, 987.6M cgroup peak).
  • $FABRIC_DATA/results/readiness-continuation-encoded-page-02 (unit fabric-work-b346b7c15ad84eddab1f3975f86461d9, 27.716 s, 985.2M cgroup peak).
  • $FABRIC_DATA/results/readiness-continuation-page-store-faults-01 (unit fabric-work-7ebc39e8a33141b69e425e64b39ef8b7, 22.307 s, 215M cgroup peak).

The corrected frozen builder SHA256 is 1b97e7b1e8372cf36f6d5b0697dffcfd12732b7b66c0f85f57913063e42138de. Its release build command used both FABRIC_ROW_GROUP_CHUNKS_EXPERIMENT=1 and FABRIC_PAGE_STORE_EXPERIMENT=1, other experiment flags unset, cargo build --offline --release -p fabric-server --example completion_builder --example completion_fault --features responsibility-alloc-probe (unit fabric-work-696d7cd23a2140ef83ebcb3a88182bf0, exit 0). The final cargo test --offline -p fabric-server --features responsibility-alloc-probe segment::bounded --lib with the same experiment flags passed all 27 tests (unit fabric-work-8eb1d179f9a345df810f3682efa42019, exit 0). Tests included exact independent file-byte comparisons, stale keys, real file write/read/cleanup failures and default selector behavior.

All seven native page fault cases met their registered outcomes: ENOSPC write, EIO read, partial write/read followed by EIO, SIGKILL at write/read and failed unlink after a paused read's owned pathname was replaced with a directory. The clean and nonexistent-path controls behaved as specified. Evidence was saved before recovery; ordinary failures cleaned scratch before retry, killed cases recovered, every retry matched the clean manifest and sealed input was unchanged. All owned fixtures/state were removed after evidence publication.

The initial census was 61279928320 bytes with over 424 GB filesystem space free. The first harness used a 100 GiB admission constant; the actual census was safely below the owner's 100 GB ceiling. The corrected harness enforces 100000000000 bytes and verifies copied source hashes against originals before measurement. The first frozen harness and its receipts remain unchanged.

Registered screen, campaign and supplemental spill observations

The owner registered the follow-up under RC-GROUP commit 39af3e2 before measurement, using the unchanged chunk_groups_screen.py, continuation_campaign.py and builder.py. The three-cell screen passed all existing checks: steady64, bigrows64 and steady256 used 36943714, 41406229 and 39115218 bytes of counted incremental heap respectively. Steady64-to-steady256 growth was 5.88%, within 10%. Its cleanup receipt confirms removal of owned fixtures. Resource unit fabric-work-07cb099b12b44a18b8113c688ff36380 exited 0 after 244.953 s, with 1.6G cgroup peak.

The full eight-cell campaign ran three alternating fresh-process pairs per cell against the same frozen binary. All 24 pairs passed unchanged equivalence, applicable byte equality, filters, pruning, heap and cleanup checks; all cells passed repeated determinism. The full scaling check also passed. Median elapsed and CPU values below are seconds; heap is the worst candidate repetition in bytes.

Cell Reference wall Candidate wall Reference CPU Candidate CPU Candidate heap
steady16 0.609116 0.636886 0.57 0.60 36943414
steady32 1.148046 1.236412 1.11 1.15 36943414
steady64 2.302079 2.385170 2.15 2.27 36943414
steady128 4.654998 5.206753 4.49 5.06 39114918
steady256 9.549153 9.714243 9.25 9.39 39114918
outage64 2.436877 2.549793 2.22 2.37 36943414
adversarial64 2.512495 2.638882 2.39 2.52 36943514
bigrows64 1.497550 1.457251 1.40 1.38 41405704

The supplemental unchanged observer then ran exactly one replay per cell against the same binary and admitted campaign hashes. All eight replay inputs and final manifests matched their original pair, the exact 9-byte write/writev control passed, and observer library/binary hashes remained unchanged. Page-byte totals below were identical across all three paired repetitions; their parser accepted the actual positive receipt and rejected injected malformed JSON, unbalanced puts/takes and cleanup-false records. All actual table observations had balanced puts/takes, zero outstanding blobs and successful cleanup.

Cell Row-run payload writes Encoded page blob writes Combined logical scratch writes
steady16 19295590 12093188 31388778
steady32 38380800 24209889 62590689
steady64 76539600 48438607 124978207
steady128 153061770 96869561 249931331
steady256 336302946 193741173 530044119
outage64 76539600 48544776 125084376
adversarial64 76539600 51970216 128509816
bigrows64 67640110 50654286 118294396

These two logical scratch components exclude filesystem metadata and physical device traffic. Row-run observations are supplemental and noncomparative for timing/heap. Candidate median /proc/self/io character counters provide the following distinct read/write observation; complete reference/candidate I/O deltas, syscall counts and all three samples are retained in observations.json.

Cell Candidate rchar Candidate wchar
steady16 48167677 44037528
steady32 96150332 87768873
steady64 192089250 175342523
steady128 384154309 350508463
steady256 798485854 731189566
outage64 192195419 175554876
adversarial64 195620860 182406233
bigrows64 185478509 169127524

Commands and exits are retained under the resource-containment receipts:

  • python3 tools/resource_group.py -- python3 tools/bench/labs/completion/chunk_groups_screen.py --out $FABRIC_DATA/results/readiness-continuation-page-store-screen-01: unit above, exit 0.
  • python3 tools/resource_group.py -- python3 tools/bench/labs/completion/continuation_campaign.py --out $FABRIC_DATA/results/readiness-continuation-page-store-group-01 --screen $FABRIC_DATA/results/readiness-continuation-page-store-screen-01: fabric-work-e615ef879f284c07a4355328f474e8d9, exit 0, 723.058 s, 1.8G cgroup peak.
  • python3 tools/resource_group.py -- python3 tools/bench/labs/completion/spill_measure.py --campaign-root $FABRIC_DATA/results/readiness-continuation-page-store-group-01 --binary $FABRIC_DATA/results/readiness-continuation-page-store-group-01/frozen/completion_builder --out $FABRIC_DATA/results/readiness-continuation-page-store-row-spill-01: fabric-work-082bdce8a93241589d899031f73f2761, exit 0, 393.914 s, 802M cgroup peak.

The auxiliary page summary and sensitivity controls exited 0 under fabric-work-ec2079f60f03493e8b9537441ed4db0a; their exact argv and checker hash are in page-blob-summary.json and the retained control records. Report aggregation exited 0 under fabric-work-accef187dc68475b9a7e9ca5c1d66dd1, 252 ms, 24.1M peak; its argv/hash and all timing/CPU/heap/I/O/scratch observations are in readiness-continuation-page-store-group-01/observations.json.

Admission censuses were 62047199232 bytes before the screen, 62048727040 before the campaign and 62071234560 before replay, each below 100000000000 bytes with ample fixture reserve. Owned campaign and replay fixtures were removed; result receipts and frozen artifacts remain. The owner-controlled Fabric workload slot remained exclusive during timed pairs. The desktop host was shared: during supplemental replay, the owner's read-only census observed unrelated interactive work at approximately 3.4 CPU cores and 4.6 GB RSS, plus about 0.3 core of other work. This was not a dedicated isolated benchmark machine; absolute timing changes cannot be attributed solely to the candidate. Compiler version was not separately captured in the new entropy provenance; exact build argv, feature/selector settings, source copies and binary hashes were recorded.

Remaining limits

The opt-in backend changes scratch I/O, not persisted encoding, grouping or the publication boundary. Scratch cleanup failure prevents publication, and page locator state is scoped to a single row-group column chunk. Product defaults remain unchanged. The registered multi-cell builder campaign is complete for its implemented gates; native soak and the remaining acceptance checks are separate work. The unchanged campaign's full_bs_acceptance remains false; supplemental observations do not rewrite its historical pending-metric fields. Arbitrary individual row size, segment metadata/filter cardinality and whole-server resource behavior remain outside this finite probe's heap result. File-backed page cache remains charged to the actual cgroup; reduced counted heap does not establish a whole-server RSS bound.