Resident Memory Comparison - Mainkill1/SpiritBoot GitHub Wiki
SpiritBoot and Complex: resident memory investigation
Separate complete layout references: Complex-BIOS-Size-and-Layout and SpiritBoot-BIOS-Size-and-Layout.
What is established
The original audited default 64 MB SpiritBoot image retains 408 KiB of kernel backing, compared with 180 KiB for the supplied Complex reference. The image difference is 228 KiB. The original matched public allocation checkpoint reports 656 KiB less available memory with SpiritBoot. Those are different quantities: the second includes other allocations and the probe itself.
The reference's advantage is supported by three observations:
- It has substantially less resident kernel content.
- Its public small-allocation capacities are much finer than SpiritBoot's power-of-two classes.
- It returns physical backing after the tested allocation and thread lifetimes; SpiritBoot retains additional backing.
The covered traces do not show Complex executing runtime functions directly from ROM. ROM execution was observed during startup, then none after title entry in two bounded workloads. Uncovered gameplay, errors and reset paths remain unknown. A mapped ROM alias alone does not demonstrate a function call.
Detailed inventories
- Owned function and table ledger: all 1,438 merged-main compiler extents, source attribution, freshly recomputed exact exports/conservative graph, and 34 matching table-size/declaration proofs. The COFF index has 4,298 records and 531 compiled object/source edges.
- The original local inventory has 4,300 COFF records and 1,439 compiler extents. Its original and fresh candidate CSV/JSON artifacts remain in the retained lab evidence; these wiki links lead to human-readable ledgers, not downloads.
- Reference census: all 330 public size comparisons, 1,361 inferred entry measurements, ROM observations and explicit unknown boundaries. It publishes original metadata, not reference code.
- Combined-tree checkpoint below: the frozen allocation/lifetime guest and independently verified corrections. The original 656 KiB initial availability deficit was 636 KiB after #126/#127 and is now 572 KiB with merged #129.
Private reference instructions, pseudocode and captured contents remain segregated. Implementations use SpiritBoot's own source and independently authored public-API tests.
1. Kernel layout
| Resident backing | Complex | Original audited SpiritBoot default 64 MB |
|---|---|---|
| Unique kernel pages, header included | 45 | 102 |
| Kernel backing | 180 KiB | 408 KiB |
| Executable-region content | 153,136 bytes, includes unidentified readonly content | 331,432 bytes, .text plus PAGE |
| Separate readonly content | Not independently classified | 25,864 bytes |
| Writable/BSS content | 28,772 bytes | 46,008 bytes |
Complex's section organization differs from ours. Its executable section can contain tables, so the executable rows are not pure instruction counts. Its writable regions have metadata extents of 21,964, 6,544 and 264 bytes. They share physical pages. Its discardable region contains 453,082 bytes and intersects 112 pages; 111 are unmapped at the checkpoint and one shares live content.
SpiritBoot's exact original audited layout is:
| Section | Content bytes | Resident pages | Backing KiB |
|---|---|---|---|
| Header | One header page | 1 | 4 |
.text |
328,932 | 81 | 324 |
.data |
2,064 | 1 | 4 |
.rdata |
25,864 | 7 | 28 |
.bss |
43,944 | 11 | 44 |
PAGE |
2,500 | 1 | 4 |
| Resident total | 403,304 plus header/padding | 102 | 408 |
INIT, already discarded |
89,760 | 0 resident | — |
.rsrc, already discarded |
2,760 | 0 resident | — |
All 495,824 shipped section-content bytes are partitioned without overlap in the owned inventory. Named compiler extents and proved tables are accounted once; remaining assembly, data and alignment remain explicitly unattributed. That is complete byte accounting, not complete semantic identification of every byte.
Ideal packing of the current resident contents gives a 99-page lower bound versus 102 actual pages. Thus packing alone offers at most about 12 KiB under that assumption, far below the 228 KiB image gap. Writable/readonly permissions also constrain packing. Removing one function does not free every page it touches: 84 resident section pages contain multiple proved extents.
2. Function and runtime breadth
Representative large owned compiler extents from the original audit:
| Function | Emitted bytes | Current role |
|---|---|---|
VfatDispatchRequest |
10,516 | Runtime filesystem dispatch |
AtaDispatchPnp |
5,522 | Published storage callback; boot seal needs lifetime proof |
PciIdeXDispatchPnp |
5,081 | Published storage callback; same boundary question |
AtaPortWorkerThread |
3,764 | Runtime storage worker |
IopParseDevice |
3,281 | Object/device path resolution |
DoQuery |
3,271 | Runtime directory query |
NtSetInformationFile |
2,824 | Public file operation |
PspExitThread |
2,818 | Thread teardown |
These are compiler extents, not exclusive call trees. LTO can inline shared helpers, fold bodies or create clones. The inventory records that attribution limit.
The existing heuristic storage/filesystem buckets account for about 109 KiB of named resident code/data, before unclassified storage content and shared kernel services. This is a useful scale estimate, not a ready 109 KiB deletion. A runtime read/write implementation remains needed after boot.
An existing boot seal removes seven drivers' PnP dispatch pointers and clears AddDevice before INIT reclamation. A conservative denylist still keeps the callback bodies resident. Thirteen emitted dispatch/AddDevice roots occupy 18,878 bytes; four large dispatch roots account for 14,780 bytes. This is the clearest code-lifetime lead. Before moving even one body to INIT, we must prove that no queued request, completion, copied callback, active return address, media/error path or reset consumer still needs it. Their gross callee union contains shared runtime code and is not a saving estimate.
3. Tables and other live state
| Owned object | Actual bytes | Lifetime or reason retained |
|---|---|---|
P0BootStackData |
12,288 | Becomes the idle CPU stack; boot name does not mean disposable |
| Double-fault stack | 4,096 | Fault/NMI handling |
| Unicode case table | 4,870 | Public Unicode and path matching |
| Interrupt shadows | 4,192 | Runtime title interrupt compatibility |
| Pool bucket occupancy | 4,096 | Active allocation/free/reclaim metadata |
| Two persistence bitmaps | 4,096 combined | Warm-reset ownership |
| NTSTATUS translation arrays/index | 5,184 combined | Public status translation contract |
| Export scaffold | 1,552 | Public ordinal addresses |
| Cache descriptors | 1,360 | Ordinary cache plus write-back reserves |
| Font subset | 1,248 | Boot and bugcheck output |
| Initial contiguous ownership table | 1,024 | Runtime allocations; can grow |
| Boot persistence descriptors | 20,200 | Already discarded in INIT |
The full inventory lists every proved table, source and reader/lifetime rationale. Symbol gaps are not treated as sizeof: for example, the Unicode table is 4,870 bytes despite its 4,872-byte symbol span.
Outside the image, the owned allocation formulas include:
| Allocation | Ordinary 64 MB | Ordinary 128 MB | Interpretation |
|---|---|---|---|
| PFN links and pin state | 96 KiB | 192 KiB | Live state; the old pin-batch scratch was separately removed |
| Pool run lengths/bitmap | 12 KiB rounded | 12 KiB rounded | Metadata for the virtual pool window |
| Eager HAL bounce backing | 64 KiB | 64 KiB | Concrete omission candidate, not yet recovered |
| Initial cache buffers plus reserves | 96 KiB | 96 KiB | 64 KiB public capacity plus 32 KiB recursive write-back reserve |
| Standard scheduler stack | 12 KiB each | 12 KiB each | Guard virtual page has no physical backing |
Static tables already counted in the image must not be added again. Public cache/pool counters also overlap actual owners. The two write-back reserves prevent recursive metadata deadlock; simply deleting them is not a valid footprint fix.
4. Allocation behavior
The identical public guest successfully allocated 32 blocks at each of 18 request sizes, queried capacity, checked lifecycle state and freed all 576 blocks per BIOS.
| Requested bytes | Complex usable capacity | SpiritBoot usable capacity |
|---|---|---|
| 64 | 88 | 120 |
| 128 | 152 | 248 |
| 256 | 280 | 504 |
| 512 | 536 | 1,016 |
| 720 | 728 | 1,016 |
| 1,024 | 1,048 | 2,040 |
| 2,384 | 2,392 | 4,096 |
The reference's small capacities rise in much finer steps. This public behavior does not establish its private allocator algorithm. Our own source confirms power-of-two classes with an eight-byte header.
After all blocks were freed, Complex returned to pool 5 pages and 16,226 available pages. SpiritBoot went from pool 41 to 71 pages and available 16,062 to 16,032: 120 KiB retained after this stress history. SpiritBoot has pressure reclamation, so this is not proof of a leak. It shows a different retention policy and a concrete test for finer classes and bounded empty-page trimming.
The comparison does not justify freeing every empty page immediately: repeated I/O allocations could then thrash. A future allocator PR needs physical-page recovery, hot-churn cost, pressure latency, fragmentation, contents and locking tests.
5. Stack retirement and the independently tested fix
Eight concurrent default threads use 24 live stack pages on both implementations. Complex's default minimum is also 12 KiB in the measured public ladder. Reducing our live stack minimum is therefore not the supported fix.
After the reference lifetime workload, all 24 pages return. SpiritBoot retains 12 additional pages above its starting history: 48 KiB. Its public thread and handle counts return to baseline, distinguishing retained stacks from still-running workers.
Owned-source diagnosis found the reason: Xbox's zero dead-stack policy is overwritten with the generic five-stack limit during memory initialization. A separate candidate preserves zero while leaving generic limits, live stacks and worker counts unchanged.
Matched ordinary 64/128 MB candidate tests completed 192 synchronized thread lifetimes:
| Matched checkpoint | Candidate recovery |
|---|---|
| Cold public guest checkpoint | 12 KiB |
| After default/12 KiB thread waves | 60 KiB |
| Nondefault 64 KiB stack cleanup | Both paths return all backing |
The 12 and 60 KiB gains describe different histories and must not be added. The research's 48 KiB retention measures growth from a baseline that already had one cached stack. Fix evidence records the independent main/candidate pair and exact identities. Release 64/128 builds and focused host gates pass; review/publication status is tracked separately in issue #125.
6. ROM calls and functions not observed
The first trace records 11,084,723 executed block entries and the second 12,572,266. Startup ROM execution is present. No post-title ROM execution was observed in the covered workloads, and the decoded reference entries yielded no direct call target resolving into the live ROM window.
This makes widespread runtime ROM execution an unsupported explanation for the measured advantage. It does not exclude an indirect, error, debugger or reset path outside the captured workloads.
The reference census has 1,361 inferred entry walks, including interior entries and shared tails; these are not certified compiler functions. It includes 330 public function comparisons. 199 public entry walks were not observed across the bounded traces. None is proved never used.
There are also genuine identification limits: 58,410 bytes of the retained executable region remain unclassified, and ten retained-executable-region indirect-jump table candidates have unknown extents. Without reliable boundaries, inventing per-table or per-function sizes would be misleading. These ranges are retained explicitly for continued investigation.
Our conservative source model leaves only 218 executable-symbol records / 1,932 inferred bytes outside its root closure; some are assembly labels or data. Automatic INIT analysis finds eight remaining straightforward candidates totaling 538 inferred bytes. Neither supports a large blanket dead-code deletion. Public exports, asynchronous callbacks and rare failure/reset paths require lifetime proof.
What direct ROM execution would require in our implementation
Read-only machine code can potentially execute from a mapped flash window rather than a RAM copy. That is a separate placement mechanism, not evidence that Complex uses it during runtime. It also differs from compressed firmware: compressed bytes cannot execute directly and first need an executable representation.
For SpiritBoot, a candidate must have stable mapped addresses throughout boot, gameplay, reset and restoration; correct executable/read permissions and cache behavior; resolved calls and relocations; and no writable state mixed into the ROM range. Any writable globals still need RAM. Public export addresses, registered callbacks and copied function pointers must keep pointing to live code. Moving readonly tables also needs their consumers to retain valid addresses.
Our existing ROM-backed candidate #118 already explored placement and reported a historical 352 KiB reduction. It failed PGR2 qualification and remains held. This investigation does not replace that failure with a success claim, nor prove ROM access costs negligible. The bounded reference observations favor first fixing unnecessary RAM lifetimes and retained runtime breadth; a repaired ROM experiment can be evaluated separately against the same public and game compatibility gates.
7. Combined-tree checkpoint after the first corrections
The original inventory above is pinned to main 15594c5 / reconstructed tree dfbb6cb16da7a78694e94171842f4175f4b23188. It is retained rather than silently relabeled as a newer build. A fresh candidate census is available in current candidate inventory and symbols.
The new inventory covers reconstructed tree 1277f2b7bf45a0992ef56d45d55815396d46ac8d: 1,438 compiler input extents, 531 compiled object/source edges, and complete nonoverlapping section accounting. .text is 328,884 bytes, 48 fewer than the original; INIT is 89,664 bytes, 96 fewer. Resident image backing is still 102 pages / 408 KiB. The original 32 table-size proofs remain associated with their original source/build; the fresh ledger does not falsely claim those measurements were repeated.
The same frozen allocation/lifetime guest was rerun on merged main and the DMA candidate, using identical disposable HDD/EEPROM seeds, xemu, RAM setting and procedure. The reference column reuses the retained, completed reference census; it is not a new Complex run this revision.
| Public checkpoint | Retained Complex reference | Original SpiritBoot | Merged #126 + #127 | Merged plus #129 |
|---|---|---|---|---|
| Initial available pages | 16,226 | 16,062 | 16,067 | 16,083 |
| Initial pool pages | 5 | 41 | 39 | 39 |
| Initial stack pages | 19 | 40 | 37 | 37 |
| Final available pages after pool/thread stress | 16,226 | 16,020 | 16,036 | 16,052 |
| Final pool pages | 5 | 71 | 70 | 70 |
| Final stack pages | 19 | 52 | 37 | 37 |
The combined candidate recovers 21 pages / 84 KiB at the initial checkpoint relative to the original measured SpiritBoot tree. Of that, 20 KiB belongs to #126/#127, and 64 KiB belongs to merged #129. The resulting initial availability difference from the retained reference is 143 pages / 572 KiB, compared with 164 pages / 656 KiB originally. This is an availability comparison, not a firmware-only physical ownership subtotal.
After the full stress history the combined candidate recovers 32 pages / 128 KiB relative to the original, a different history-dependent comparison. Do not add 84 and 128. Both new sides retain 31 additional pool pages / 124 KiB after the allocation ladder, leaving pool policy as an active target; their stacks return to their initial count. The old tree retained 30 extra pool pages / 120 KiB. Retention changes with allocation history; the matched DMA difference remains exactly 16 pages at every checkpoint.
Separately, the unchanged thread-lifecycle guest completes 96 successful lifetimes across the DMA parent/candidate pair, with identical pool and stack counters and exactly 64 KiB recovered at all checkpoints. The frozen storage/I/O guest passes 189/189 checks on each side. These results do not establish title FPS gains, physical Xbox DMA, or exhaustive asynchronous growth/failure behavior.
8. Incremental implementation order
- Merged #126: omit the unused Xbox private storage error ring, independently recovering 8 KiB at boot. Exact-head CI and review passed.
- Merged #127: preserve Xbox's zero dead-stack-retention policy. Exact integration-head CI passed; repeated 64/128 lifecycle tests and the refreshed combined 64 MiB pair passed. It recovers 12 KiB at boot and 60 KiB after its focused thread churn, without shrinking active stacks.
- Merged #129 / closed issue #128: queued and immediate hardware scatter/gather requests now agree. The eager 64 KiB bounce allocation is omitted on Xbox. Production-function checks, independent review, full release64 build, 48 maintained host-gate logs, 105 product tests / 3 existing skips, and matched native memory/storage checks pass. All twelve exact-head release/debug64/128, host and plugin checks passed before merge.
- Use the frozen capacity ladder to evaluate finer small-object classes and bounded empty-page return under #14. Stress retention is a test opportunity, not a promised cold-boot saving.
- Establish the boot PnP seal's final-consumer boundary, then move one proven callback/cone at a time to INIT and measure page-rounded output.
- Revisit runtime specialization, remaining unknown reference ranges and metadata layouts after these bounded reductions. Preserve public compatibility and reset/persistence behavior.
Savings are not additive until measured in one combined tree. No whole-firmware parity, game FPS gain or slow-loading diagnosis follows from this investigation. The Windows user session and both test devices were left untouched during these local probes.
Current delivery status
PR #126 merged at 12fda7bfa396474d5d31d05d244a471c83579b43; issue #121 closed. PR #127 merged at 17aec7afaaf0b14170cf7c375466e4d5ce456b33; issue #125 closed. Ordered patches 0055/0056 and both source-tree assertions were reconciled, and the integration head passed all twelve GitHub check runs.
PR #129, fixing issue #128, merged at d4350b603452e029d0862fb48e7d4907e489e8ef after all twelve exact-head checks passed on dbae9e1ced2754f00079e39798255841fe32487a. Its reconstructed tree and local tested tree are identical. Evidence stays outside product main. No new release has been posted in this checkpoint.
The comparison is tracked on issue #106. Issue #14 carries the pool follow-up. The Windows user session and test devices were left untouched by these local probes.
Fresh ledger and next-lifetime audit
The function/table ledger was refreshed against merged main, including fresh table proofs and current-address export/callback graph data. Its disjoint section total is 495,680 bytes; 86 resident section pages contain multiple proved extents. The original audit figures above remain pinned to their original image. See Memory-Lifetime-and-Reduction-Plan for the storage private-callback audit and the unmerged #131 recovery/churn result. The current initial availability gap remains 572 KiB; #131 is not counted as landed.
Current deeper breakdown
See Memory-Resident-Page-Atlas for every main image page and additional exact object sizes, Memory-Runtime-Physical-Owners for the fresh current-main physical census, Memory-ROM-and-Cold-Code for owned placement/consumer contracts, and Memory-Reference-Content-and-ROM for the second bounded reference pass. These pages retain exact build/checkpoint identities and explicit unknown ranges. They do not merge older availability totals or prototype results into current main.
Current-main ROM refresh
The fresh current-main comparison recovers352 KiB with optional ROM backing: total unavailable RAM1328→976 KiB at the identical final allocation/lifetime checkpoint. The previously matched Complex result is632 KiB, leaving344 KiB. Resident kernel RAM alone is56 KiB versus Complex180 KiB. Full owner accounting and qualification limits are recorded in SpiritBoot-BIOS-Size-and-Layout#current-main-rom-residency-measurement. This is not a claim that total firmware compatibility is complete.
Directly measured ROM and pool integration
PR #135 reuses wholly empty slab backing when another size class needs a refill. Headers, public capacities, class sizes and live objects are preserved. It does not use #131's idle trim.
| Same final allocation/lifetime checkpoint | Unavailable physical RAM | Gap from retained Complex |
|---|---|---|
| Retained Complex reference | 632 KiB | — |
| Current main, RAM kernel | 1328 KiB | 696 KiB |
| Refreshed ROM kernel (#118) | 976 KiB | 344 KiB |
| Independent pool reuse, RAM kernel (#135) | 1264 KiB | 632 KiB |
| Directly built ROM + pool integration | 912 KiB | 280 KiB |
All sides use the same final checkpoint; do not combine these with the older initial572 KiB gap. Integration treeee941fca1df6216d3e4080623b067f90d7c1d409 completes576 allocations and40 thread lifetimes, with zero unattributed pages. The combined 416 KiB recovery was directly measured. ROM size512 KiB is within the approved1 MiB allowance. Parity is not met; neither PR is declared title-qualified.
The2,093-event production allocation/free replay retains86 pool pages with existing classes,70 with empty-slab reuse. Uniform finer classes instead model117 or125 pages without reuse. These modeled alternatives are not implemented firmware: better nominal sizes can still produce worse physical packing.
Live stack owners
The paused owned thread list, using offsets compiled from exact production source, identifies four executive workers (48 KiB), one filesystem overflow worker (12 KiB), one ATA worker (12 KiB), the title thread (64 KiB), and initial idle stack (12 KiB). Their37 committed pages match the public stack count. The idle stack is already charged in resident image/BSS ownership; do not add it again. A terminated Phase1 thread has no mapped stack pages.
Xbox already omits the balance-manager thread and already has one filesystem overflow queue. The hypercritical worker serves thread teardown through a direct KiInsertQueue, bypassing ExQueueWorkItem. Counting only that public queueing function would incorrectly label the reaper unused. Issue #143 tracks the measured ownership and one-worker investigation. No worker is removed solely for being idle.
Current plain-main comparison and third reference
The new Retail-5838-BIOS-Size-and-Layout keeps the supplied retail kernel separate from Complex. SpiritBoot-Main-Emitted-Extent-Ledger records every exact emitted extent in the source/artifact bound to the plain-main capture at f60f8de605cb, excluding ROM/pool experiments. SpiritBoot-Main-Physical-Ownership accounts for all 332 unavailable pages at its completed guest checkpoint. Memory-Reduction-Targets compares those results, preserves historical revision boundaries, and links the updated reduction issues. The historical figures on this page are not silently replaced or upgraded.