Experiment formal encoded page memory run 01 - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/formal/encoded-page-memory-run-01.md. Original SHA-256:b987f767aa98eeaa49067cccfb4c04a1986bf88f45406407e332b827c2e0c4a6. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Encoded-page memory counterexample and disk-store diagnostic
Observed 2026-10-09 under the prospective protocol (local checkout reference docs/experiments/formal/encoded-page-memory-protocol.md; not present at origin/main).
These are two diagnostic pairs, a finite fault screen and the subsequently
registered repeated campaign, not a promotion or
universal heap claim. The original aligned candidate exceeded the 80 MiB probe
ceiling; the separately frozen disk-store candidate stayed below it while
preserving the required exact output.
Frozen fixture and measured outcomes
Both pairs used seed 0xA11FA001, exactly 8192 distinct 16384-byte base64 log
bodies, 256 ordinary batches and one full 8192-row physical log group. The sealed
journal was 134415362 bytes with SHA256
76ea6009cfd27e93e67736a2051250e90050c56b69916f6ebc02265140236510.
Each generation streamed 32 rows at a time; generation and differential readback
were excluded from the counted build boundary.
| Observation | Aligned in-memory store | Aligned disk store |
|---|---|---|
| Candidate incremental counted peak heap | 143075502 bytes | 42011899 bytes |
| Reference incremental counted peak heap | 647059826 bytes | 647059826 bytes |
| Candidate build elapsed time, single observation | 3.369718811 s | 4.452622721 s |
Candidate /proc/self/io rchar delta |
269199111 | 471765567 |
Candidate /proc/self/io wchar delta |
337562272 | 540129583 |
Candidate /proc/self/io write_bytes delta |
337612800 | 540848128 |
Candidate /proc/self/io cancelled_write_bytes delta |
134815744 | 338051072 |
The ceiling is 83886080 bytes. Both pairs satisfied unchanged comparator checks
for input, ordered logical rows, manifest semantics, applicable Parquet bytes,
physical filter validity, pruning and no leftovers. Additional checks verified
fixture counts, one full log group, exact filter bytes, counted allocator and
unchanged frozen sources/input. Only the first pair's heap check failed.
logs.parquet was exactly 101041273 bytes in all four builds, SHA256
5735ebb05b6ded206758671c1813c1e6797b5153b898786444279aa95b247fb0.
The disk backend counted 101019985 completed serialized log blob bytes, with exactly that peak outstanding payload, largest blob 788900 bytes, 284 puts and 284 takes, zero outstanding bytes and successful cleanup. The counted heap reduction was 101063603 bytes, only 43618 bytes different from the completed log payload removed from resident storage. This isolated intervention and byte count support the completed-page explanation over a full-group dictionary explanation. The raw table also spilled 101546471 blob bytes (884 puts/takes), with a much smaller 2778426-byte peak because its original chunk flush policy remains.
Logical page scratch writes totaled 202566456 bytes across tables. The rchar
increase was exactly that amount; the wchar increase was that amount plus 855
bytes of instrumented stderr observations. These are process I/O counters and
logical backend bytes, not a measurement of physical device traffic. The existing
.run- observer measures row-run spill only and does not include page scratch.
Single ordered pairs support the finite heap counterexample and correction;
they establish no timing improvement or general performance comparison.
Executed verification and receipts
Every workload used python3 tools/resource_group.py -- ..., enforced 20 GiB
memory maximum and zero swap, and exited 0. Exact argv, build settings, source
copies/hashes, frozen binaries, per-child stdout/stderr/exit and raw measurements
are retained on the data drive:
$FABRIC_DATA/results/readiness-continuation-encoded-page-01(unitfabric-work-a239e0059636442480ad6831068b2368, 24.719 s, 987.6M cgroup peak).$FABRIC_DATA/results/readiness-continuation-encoded-page-02(unitfabric-work-b346b7c15ad84eddab1f3975f86461d9, 27.716 s, 985.2M cgroup peak).$FABRIC_DATA/results/readiness-continuation-page-store-faults-01(unitfabric-work-7ebc39e8a33141b69e425e64b39ef8b7, 22.307 s, 215M cgroup peak).
The corrected frozen builder SHA256 is
1b97e7b1e8372cf36f6d5b0697dffcfd12732b7b66c0f85f57913063e42138de.
Its release build command used both FABRIC_ROW_GROUP_CHUNKS_EXPERIMENT=1 and
FABRIC_PAGE_STORE_EXPERIMENT=1, other experiment flags unset,
cargo build --offline --release -p fabric-server --example completion_builder --example completion_fault --features responsibility-alloc-probe (unit
fabric-work-696d7cd23a2140ef83ebcb3a88182bf0, exit 0).
The final cargo test --offline -p fabric-server --features responsibility-alloc-probe segment::bounded --lib with the same experiment flags
passed all 27 tests (unit fabric-work-8eb1d179f9a345df810f3682efa42019, exit 0).
Tests included exact independent file-byte comparisons, stale keys, real file
write/read/cleanup failures and default selector behavior.
All seven native page fault cases met their registered outcomes: ENOSPC write, EIO read, partial write/read followed by EIO, SIGKILL at write/read and failed unlink after a paused read's owned pathname was replaced with a directory. The clean and nonexistent-path controls behaved as specified. Evidence was saved before recovery; ordinary failures cleaned scratch before retry, killed cases recovered, every retry matched the clean manifest and sealed input was unchanged. All owned fixtures/state were removed after evidence publication.
The initial census was 61279928320 bytes with over 424 GB filesystem space free. The first harness used a 100 GiB admission constant; the actual census was safely below the owner's 100 GB ceiling. The corrected harness enforces 100000000000 bytes and verifies copied source hashes against originals before measurement. The first frozen harness and its receipts remain unchanged.
Registered screen, campaign and supplemental spill observations
The owner registered the follow-up under RC-GROUP commit 39af3e2 before
measurement, using the unchanged chunk_groups_screen.py,
continuation_campaign.py and builder.py. The three-cell screen passed all
existing checks: steady64, bigrows64 and steady256 used 36943714, 41406229 and
39115218 bytes of counted incremental heap respectively. Steady64-to-steady256
growth was 5.88%, within 10%. Its cleanup receipt confirms removal of owned
fixtures. Resource unit fabric-work-07cb099b12b44a18b8113c688ff36380 exited 0
after 244.953 s, with 1.6G cgroup peak.
The full eight-cell campaign ran three alternating fresh-process pairs per cell against the same frozen binary. All 24 pairs passed unchanged equivalence, applicable byte equality, filters, pruning, heap and cleanup checks; all cells passed repeated determinism. The full scaling check also passed. Median elapsed and CPU values below are seconds; heap is the worst candidate repetition in bytes.
| Cell | Reference wall | Candidate wall | Reference CPU | Candidate CPU | Candidate heap |
|---|---|---|---|---|---|
| steady16 | 0.609116 | 0.636886 | 0.57 | 0.60 | 36943414 |
| steady32 | 1.148046 | 1.236412 | 1.11 | 1.15 | 36943414 |
| steady64 | 2.302079 | 2.385170 | 2.15 | 2.27 | 36943414 |
| steady128 | 4.654998 | 5.206753 | 4.49 | 5.06 | 39114918 |
| steady256 | 9.549153 | 9.714243 | 9.25 | 9.39 | 39114918 |
| outage64 | 2.436877 | 2.549793 | 2.22 | 2.37 | 36943414 |
| adversarial64 | 2.512495 | 2.638882 | 2.39 | 2.52 | 36943514 |
| bigrows64 | 1.497550 | 1.457251 | 1.40 | 1.38 | 41405704 |
The supplemental unchanged observer then ran exactly one replay per cell against the same binary and admitted campaign hashes. All eight replay inputs and final manifests matched their original pair, the exact 9-byte write/writev control passed, and observer library/binary hashes remained unchanged. Page-byte totals below were identical across all three paired repetitions; their parser accepted the actual positive receipt and rejected injected malformed JSON, unbalanced puts/takes and cleanup-false records. All actual table observations had balanced puts/takes, zero outstanding blobs and successful cleanup.
| Cell | Row-run payload writes | Encoded page blob writes | Combined logical scratch writes |
|---|---|---|---|
| steady16 | 19295590 | 12093188 | 31388778 |
| steady32 | 38380800 | 24209889 | 62590689 |
| steady64 | 76539600 | 48438607 | 124978207 |
| steady128 | 153061770 | 96869561 | 249931331 |
| steady256 | 336302946 | 193741173 | 530044119 |
| outage64 | 76539600 | 48544776 | 125084376 |
| adversarial64 | 76539600 | 51970216 | 128509816 |
| bigrows64 | 67640110 | 50654286 | 118294396 |
These two logical scratch components exclude filesystem metadata and physical
device traffic. Row-run observations are supplemental and noncomparative for
timing/heap. Candidate median /proc/self/io character counters provide the
following distinct read/write observation; complete reference/candidate I/O
deltas, syscall counts and all three samples are retained in observations.json.
| Cell | Candidate rchar | Candidate wchar |
|---|---|---|
| steady16 | 48167677 | 44037528 |
| steady32 | 96150332 | 87768873 |
| steady64 | 192089250 | 175342523 |
| steady128 | 384154309 | 350508463 |
| steady256 | 798485854 | 731189566 |
| outage64 | 192195419 | 175554876 |
| adversarial64 | 195620860 | 182406233 |
| bigrows64 | 185478509 | 169127524 |
Commands and exits are retained under the resource-containment receipts:
python3 tools/resource_group.py -- python3 tools/bench/labs/completion/chunk_groups_screen.py --out $FABRIC_DATA/results/readiness-continuation-page-store-screen-01: unit above, exit 0.python3 tools/resource_group.py -- python3 tools/bench/labs/completion/continuation_campaign.py --out $FABRIC_DATA/results/readiness-continuation-page-store-group-01 --screen $FABRIC_DATA/results/readiness-continuation-page-store-screen-01:fabric-work-e615ef879f284c07a4355328f474e8d9, exit 0, 723.058 s, 1.8G cgroup peak.python3 tools/resource_group.py -- python3 tools/bench/labs/completion/spill_measure.py --campaign-root $FABRIC_DATA/results/readiness-continuation-page-store-group-01 --binary $FABRIC_DATA/results/readiness-continuation-page-store-group-01/frozen/completion_builder --out $FABRIC_DATA/results/readiness-continuation-page-store-row-spill-01:fabric-work-082bdce8a93241589d899031f73f2761, exit 0, 393.914 s, 802M cgroup peak.
The auxiliary page summary and sensitivity controls exited 0 under
fabric-work-ec2079f60f03493e8b9537441ed4db0a; their exact argv and checker hash
are in page-blob-summary.json and the retained control records. Report aggregation
exited 0 under fabric-work-accef187dc68475b9a7e9ca5c1d66dd1, 252 ms, 24.1M peak;
its argv/hash and all timing/CPU/heap/I/O/scratch observations are in
readiness-continuation-page-store-group-01/observations.json.
Admission censuses were 62047199232 bytes before the screen, 62048727040 before the campaign and 62071234560 before replay, each below 100000000000 bytes with ample fixture reserve. Owned campaign and replay fixtures were removed; result receipts and frozen artifacts remain. The owner-controlled Fabric workload slot remained exclusive during timed pairs. The desktop host was shared: during supplemental replay, the owner's read-only census observed unrelated interactive work at approximately 3.4 CPU cores and 4.6 GB RSS, plus about 0.3 core of other work. This was not a dedicated isolated benchmark machine; absolute timing changes cannot be attributed solely to the candidate. Compiler version was not separately captured in the new entropy provenance; exact build argv, feature/selector settings, source copies and binary hashes were recorded.
Remaining limits
The opt-in backend changes scratch I/O, not persisted encoding, grouping or the
publication boundary. Scratch cleanup failure prevents publication, and page
locator state is scoped to a single row-group column chunk. Product defaults
remain unchanged. The registered multi-cell builder campaign is complete for its
implemented gates; native soak and the remaining acceptance checks are separate
work. The unchanged campaign's full_bs_acceptance remains false; supplemental
observations do not rewrite its historical pending-metric fields. Arbitrary
individual row size, segment metadata/filter cardinality and
whole-server resource behavior remain outside this finite probe's heap result.
File-backed page cache remains charged to the actual cgroup; reduced counted heap
does not establish a whole-server RSS bound.