Memory Reduction Targets - Mainkill1/SpiritBoot GitHub Wiki
Memory reduction targets: retail 5838, Complex and SpiritBoot
This is the implementation decision page for #106. Plain merged main is the primary removal target. ROM and pool experiments remain separate. Full inventories: Retail-5838-BIOS-Size-and-Layout, Complex-BIOS-Size-and-Layout, SpiritBoot-Main-Emitted-Extent-Ledger, SpiritBoot-Main-Physical-Ownership.
The objective is equal or greater usable title allocation capacity than the reference, with no lost compatibility and no noticeable performance tradeoff. A ROM container up to 1 MiB is permitted. Moving firmware into an extra 64 MiB and hiding that capacity from titles is a compatibility experiment, not footprint reduction.
1. Measured differences and remaining gaps
| Quantity / checkpoint | Retained Complex 4627 | Retail 5838 | Plain SpiritBoot main | Optional ROM + pool integration |
|---|---|---|---|---|
| ROM file | 1 MiB, four repeated 256 KiB quarters | 256 KiB | 256 KiB | 512 KiB |
| Retained executable-section contents | 153,136 B | 156,912 B | 328,932 B | 330,352 B |
| Settled image RAM | 180 KiB measured | Not qualified; approximately 180 KiB expected from layout | 408 KiB measured | 56 KiB in the bound capture; placement remains experimental |
| Initial available pages | 16,226 retained reference | Not qualified | 16,083 | Not substituted from final count |
| Final allocation/lifetime available pages | 16,226 | Not qualified | 16,052 | 16,156 |
| Final unavailable pages | 158 | Not qualified | 332 | 228 |
| Final unavailable KiB | 632 | Not qualified | 1,328 | 912 |
| Final availability gap to Complex | 0 | Unknown | 696 KiB | 280 KiB |
Product main verified on 2026-10-10: f60f8de605cbbdce2f22e4bc3489e1c1487dcb59. Its retained artifact/census binds reconstructed source tree 6e7666c73c916a235d29ec793dc9ee0920df299b to flash SHA256 75bee35302ea9b004f1e6fc1ea9d1e5857985c96489635c56b7461a46849ca73. The optional integration tree is ee941fca1df6216d3e4080623b067f90d7c1d409.
The separate ROM-only capture recovers 88 pages / 352 KiB, leaving 244 unavailable pages / 976 KiB. Pool-only recovers 16 pages / 64 KiB, leaving 316 pages / 1,264 KiB. The directly measured combined capture leaves 228 pages / 912 KiB. These are separate measured builds/checkpoints, not assumed arithmetic across unrelated histories. The earlier inventory check found both candidates draft and open. A later 2026-10-10 check confirms #135 is now Ready for validation, open and unmerged at 2ca9a3aa73e3b3284f9db4ad9f8cdb2ca0aeab8d. This status change does not waive its qualification limits or change the retained historical ROM/pool measurements. See Memory-Pool-Reuse-Cost-and-Attribution for the new repair and actual costs.
The initial main gap is 572 KiB; the post-activity gap is 696 KiB. The extra 124 KiB appears at a different workload-history checkpoint. Reclaiming backing after activity is useful, but does not establish the same cold-boot saving. Unavailable RAM includes title and hardware owners. It is not pure firmware overhead and cannot be directly compared to a 512 KiB firmware-only budget.
2. Why a function hit list alone will not close the gap
Debug COFF symbols are not runtime allocations. Deleting symbol metadata does not release image pages. INIT, loader scratch, pin-batch scratch, obsolete boot tables, the terminated initialization stack, duplicate pool-size tracking, the storage error ring and eager DMA bounce backing already have independent reductions. They cannot be promised again.
The earlier census identified only approximately 538 inferred bytes in its automatic INIT candidate set and approximately 1,932 inferred bytes of rootless executable symbols. These are historical estimates, not current deletion approvals. They cannot explain the 228 KiB image gap, and neither “rootless” nor “unobserved” means unused.
Public entries can be small thunks over large helper frameworks. Retail AvSetDisplayMode has a 1,687-byte inferred walk while main's entry extent is 128 bytes. That does not prove SpiritBoot's complete video service is smaller: its helpers and state are outside the entry extent. Conversely, an inlined function can appear large without retaining a separately allocated helper.
A deleted body saves RAM only when a new layout or valid discard transition returns unique physical pages. All other code, tables and data on those pages must be accounted for. Public exports, saved callback pointers, return addresses, queued work and shared callees can retain apparently cold memory.
3. Exact main code-size attribution
The following groups are a disjoint attribution of the 316,892-byte compiler-extent union. Family labels come from matched historical owned definitions; current emitted sizes are exact. Unknown definitions remain separate. Another 12,040 .text bytes lack this function-extent attribution. These are size priorities, not safe deletion sets.
| Owned family label | Exact emitted .text bytes | Extent records |
|---|---|---|
| ntoskrnl/xb | 47014 | 262 |
| drivers/filesystems/vfatfs | 44444 | 111 |
| ntoskrnl/ke | 34777 | 160 |
| unresolved / ambiguous | 34139 | 290 |
| ntoskrnl/io | 32547 | 108 |
| drivers/storage/ide | 31568 | 90 |
| ntoskrnl/ob | 21563 | 51 |
| drivers/storage/class | 15359 | 39 |
| ntoskrnl/ex | 12784 | 67 |
| hal/ | 11200 | 99 |
| sdk/ | 9061 | 53 |
| drivers/filesystems/xdvdfs | 6895 | 16 |
| ntoskrnl/ps | 5965 | 14 |
| drivers/storage/partmgr | 4590 | 6 |
| ntoskrnl/mm | 3216 | 15 |
| ntoskrnl/fsrtl | 1770 | 10 |
Storage and filesystem services are substantial because SpiritBoot retains a general runtime implementation. I/O naming, object management, completion, cancellation and worker dependencies also matter. A narrower fixed-platform service could remove an exclusive code closure and its objects/workers together. No measured minimal replacement yet proves how much is safely recoverable.
4. Largest retained bodies
These are the top 40 plain-main extents, including compiler inlining and padding. The complete 1,438-row list is in SpiritBoot-Main-Emitted-Extent-Ledger. None is approved for deletion by size alone.
| Body / aliases | Bytes | Owned definition candidate |
|---|---|---|
| _VfatDispatchRequest.lto_priv.0 | 10516 | drivers/filesystems/vfatfs/misc.c |
| _AtaDispatchPnp@8 | 5522 | drivers/storage/ide/atapi/pdo.c |
| _PciIdeXDispatchPnp@8 | 5081 | drivers/storage/ide/pciidex/pdo.c |
| _AtaPortWorkerThread@4 | 3764 | drivers/storage/ide/atapi/portstate.c |
| _IopParseDevice@40 | 3281 | ntoskrnl/io/iomgr/file.c |
| _DoQuery.lto_priv.0 | 3271 | drivers/filesystems/vfatfs/dir.c |
| _NtSetInformationFile@20 | 2824 | ntoskrnl/io/iomgr/iofunc.c |
| _PspExitThread@4 | 2818 | ntoskrnl/ps/kill.c |
| _VfatWrite | 2816 | drivers/filesystems/vfatfs/rw.c |
| _ObpLookupObjectName@44 | 2671 | ntoskrnl/ob/obname.c |
| _ObInsertObject@24 | 2538 | ntoskrnl/ob/obhandle.c |
| _TransferPktComplete@12 | 2285 | drivers/storage/class/classpnp/xferpkt.c |
| _PartMgrPnp@8 | 2171 | drivers/storage/partmgr/partmgr.c |
| _RawDispatch@8 | 2048 | ntoskrnl/io/iomgr/rawfs.c |
| _wstreamout | 2040 | unresolved |
| _ClassDispatchPnp@8 | 2006 | drivers/storage/class/classpnp/class.c |
| [email protected] | 2000 | ntoskrnl/ob/oblife.c |
| _PartMgrDeviceControl@8 | 1848 | drivers/storage/partmgr/partmgr.c |
| _VfatMount.lto_priv.0 | 1761 | drivers/filesystems/vfatfs/fsctl.c |
| _vfatGetFCBForFile | 1752 | drivers/filesystems/vfatfs/fcb.c |
| _NtWriteFile@36 | 1664 | ntoskrnl/io/iomgr/iofunc.c |
| _streamout | 1606 | sdk/lib/crt/printf/streamout.c |
| _KeRemoveQueue@12 | 1592 | ntoskrnl/ke/queue.c |
| _VfatCommonRead | 1565 | drivers/filesystems/vfatfs/rw.c |
| _DiskDeviceControl@8 | 1482 | unresolved |
| _KeWaitForMultipleObjects@32 | 1463 | ntoskrnl/ke/wait.c |
| [email protected] | 1452 | drivers/storage/class/classpnp/class.c |
| _XdvdDispatchDirectoryControl@8 | 1431 | drivers/filesystems/xdvdfs/xdvdfs.c |
| _IopDeviceFsIoControl@44 | 1398 | ntoskrnl/io/iomgr/iofunc.c |
| [email protected] | 1381 | ntoskrnl/ps/thread.c |
| _AtaReqStartRequest.lto_priv.0 | 1356 | drivers/storage/ide/atapi/scsi.c |
| _XeLoadXbe.isra.0 | 1354 | ntoskrnl/xb/xbe.c |
| _NtQueryInformationFile@20 | 1352 | ntoskrnl/io/iomgr/iofunc.c |
| _NtReadFile@36 | 1290 | ntoskrnl/io/iomgr/iofunc.c |
| _XeNtDuplicateObject@12 | 1283 | ntoskrnl/xb/xbe.c |
| _XbScatterGather | 1281 | unresolved |
| [email protected] | 1247 | ntoskrnl/ke/clock.c |
| _vfatFindDirSpace | 1235 | drivers/filesystems/vfatfs/dirwr.c |
| _KeWaitForSingleObject@20 | 1203 | ntoskrnl/ke/wait.c |
| _NtQueryDirectoryFile@44 | 1195 | ntoskrnl/io/iomgr/iofunc.c |
One PCIIDE PnP lifetime first — #142
Issue #142 selects PciIdeXDispatchPnp only, with 5,081 emitted bytes. The boot seal replaces public dispatch and AddDevice pointers, but private copies, queued IRPs, completions, timer/DPC callbacks, returning frames and reset/media-change users must drain too.
Prove the final consumer and add a checked late-call assertion before moving its exclusive section/callees into INIT. Shared runtime callees remain resident. The result must show an actual smaller physical image, not an automatic 5,081-byte donation. If a valid live consumer remains, stop and document it. Cover enumeration, failed boot, unexpected PnP requests, media changes, retries, normal I/O, save/load, shutdown and warm reset. Other ATA/class/partition callbacks belong in separate follow-ups.
Filesystem specialization — #141
Issue #141 starts with the 10,516-byte VfatDispatchRequest extent. Outline one measured cold arm and compare linked pages and hot-path costs. Outlining may merely move bytes; it does not automatically eliminate resident backing.
The larger opportunity is a narrower filesystem boundary preserving the FATX contracts actually used: fragmented chains, file creation, allocation, rename, truncate, queries, ordinary save/reopen, short completions and disk errors. Do not delete those semantics after a menu-only trace. Xbox-specific FAT pruning already exists. Keep any compatible specialization separate from allocator and ROM changes.
5. Runtime RAM outside code
| Owner | Main footprint | Reduction hypothesis | Required proof / current limit |
|---|---|---|---|
| PFN links and pins | 96 KiB | Compact representation or different operation lifetime | Preserve owner, multiplicity, overflow and deferred return; current packing trial fails the cost gate |
| Pool metadata | 12 KiB | Representation or populated-region reduction | No allocation dependency under the PFN lock; preserve query/free behavior |
| Resident BSS | 44 KiB | One packed or lifetime-proved page | Active stacks and tables share pages; a sum of sizeof reductions is not physical recovery |
| Pool window | 344 KiB after activity | Reuse empty backing; reduce real live object population | Includes cache and reserve; distinguish cold from warm capacity |
| Service-worker stacks | 72 KiB in the separately scoped owner census | At most one unnecessary 12 KiB execution context | Reaper, blocking I/O, queue self-waits, IRQL/APC/DPC and backlog contracts |
| Installed page tables | 68 KiB | Retire an actually empty, unreferenced table | PDE, alias and TLB lifetime proof; obsolete loader tables are already gone |
| Ordinary file cache | 64 KiB, included in pool | Workload-dependent capacity/lifetime policy | Smaller capacity may increase reads and completion latency |
| Private write-back reserve | 32 KiB, included in pool | Remove recursive cache dependency or provision before dirty activity | Prevents dirty-cache/FAT metadata deadlock; cannot allocate only after exhaustion |
Pool geometry must save pages
A 1,024-byte request plus its 8-byte header currently rounds to a 2,048-byte class. Finer classes can help some populations, but numerous partially occupied class pages can make the whole layout worse. The existing 2,093-event mixed replay reproduced 86 baseline pool pages; broad finer ladders modeled 117–125 pages, while production backing reuse in #135 reached 70 pages. These are workload results, not guaranteed cold-boot savings.
A 2,384-byte object still fits only once per 4 KiB page if its nominal block shrinks to 2,400 bytes. Without remainder reuse or a changed lifetime, its physical charge can remain a whole page. Rank private structure reductions by whether another object fits per page, whether a live page becomes empty, and whether peak demand also improves.
#135 has measured 64 KiB recovery, but refill-scan cost and conflicting Deck Conker p99 remain unresolved. #130 idle trimming and #134 backing reuse are different mechanisms. Preserve live addresses, contents, domains and capacities.
One worker, not wholesale thread removal — #143
The owner census on the separately identified ROM/pool tree, whose worker creation is unchanged, found:
| Owner | Live threads | Stack pages | KiB |
|---|---|---|---|
ExpWorkerThreadEntryPoint |
4 | 12 | 48 |
FsRtlWorkerThread |
1 | 3 | 12 |
AtaPortWorkerThread |
1 | 3 | 12 |
Title XeThreadTrampoline |
1 | 16 | 64 |
KiInitialThread idle/boot stack |
1 | 3 | 12, already in image/BSS |
The 34 ordinary thread pages plus three idle pages explain the public 37-page counter. Adding the idle pages to image ownership again would double-count them. Six service workers own 72 KiB, excluding the title's 64 KiB. The terminated Phase1 stack is already unmapped.
#143 permits an ownership/dependency census and at most one later retirement PR. Dynamic worker escalation and the balance manager are already disabled on Xbox. Hypercritical reaper jobs arrive directly from KiRundownThread, outside ExQueueWorkItem; watching only that API misses real consumers. FsRtl queues are already consolidated. Measure backlog, blocking and teardown before claiming a 12 KiB saving. Keep the 12 KiB stack minimum unchanged.
Packed pin state is currently a measured no-go — #144
The five-byte PFN trial has a theoretical 16 KiB / 32 KiB saving at 64/128 MiB, but no measured native page recovery. Its exhaustive state and alias/rollback fixtures passed, while balanced host pin/unpin measurements were approximately 25–43% slower initially and 23–48% slower with a refined common-case encoding. Those are fixture cycles, not game FPS, but the before-games cost gate failed. Latest #144 finding.
Do not advance this representation to games just because it packs bytes. It needs a different representation or operation design. Preserve all 16 pin-state bits, owner meaning, free-list encoding, sticky overflow, deferred return, relocation and whole-request rollback. Dense pin-batch scratch is already removed.
6. ROM placement: largest demonstrated mechanism, still unqualified for release
#118 demonstrates 352 KiB of readonly RAM recovery in its current bound capture. It keeps virtual addresses stable; mutable state and self-modified text stay in RAM. Default release remains RAM-only. This changes service placement, not their semantic existence.
The historical PGR2 bugcheck in #120 remains unresolved. Host checks, green CI and one later diagnostic race entry do not establish ordinary title/reset/save/load compatibility. Flash latency and effective cache type may also affect performance. Bounded follow-ups already exist:
- #139: verify MTRR/PTE cache type and measure physical flash execution latency.
- #140: trial one immutable page through the cacheable upper BIOS mirror.
- #137: retain one measured hot page in RAM.
- #138: group one hot/cold cluster into page-aligned sections.
- #136: correlate exact extents, physical pages and gameplay hotness.
Keep these changes separate. A ROM mapping is not proof that a function executed from ROM. The sampled Complex entries are RAM-backed; post-title ROM execution was unobserved in its retained bounded traces. That does not prove every late/reset path is RAM-only, and it does not establish ROM execution as the reason for Complex's small resident image.
Live ROM cacheability diagnosis — #139 / PR #151
The new standalone diagnostic at PR #151 reads the running CPU/MSR/page state without changing the BIOS. Eight local TCG fresh starts in ABBA + BAAB order complete with matching results. The tested #118 c227292 immutable byte-swap leaf maps VA 0x8403DAF0 to physical 0xFF062AF0, PTE 0xFF062021. Its live MTRR type is UC, selected PAT type WB, and combined effective type UC. Its exact-byte RAM copy resolves to WB. The default MTRR is 0x800, the WP flash range remains 0xFFF80000–0xFFFFFFFF, and live PAT is 0x0007010600070106. This confirms the first-mirror cacheability concern in the tested emulated CPU state; no physical Xbox has been measured.
| Matched diagnostic loop | Mean guest TSC ticks/call | Interpretation |
|---|---|---|
| Main kernel RAM leaf | 20.144 | A different full BIOS tree; same verified leaf bytes |
| Main identical RAM copy | 20.388 | Within-process RAM control |
| ROM-profile kernel leaf | 20.035 | No clear warm TCG penalty in this tiny path |
| ROM-profile identical RAM copy | 20.306 | Within-process RAM control |
These are equal-weight means of four per-run summaries after excluding round zero; each repeated round has 32,768 calls. Round zero is retained and is not a proved cold-cache sample. Empty indirect-call controls cost approximately the same, so subtraction would give unstable near-zero or negative isolated estimates. Guest TSC intervals under TCG are not physical flash latency, exclusive host CPU cycles or title frame times. The RAM instruction control crosses 4 KiB alignment but remains in a single 4 MiB guest PDE; it is not a guest translation-page crossing. Current xemu directly backs readonly ROM reads with host memory, so this architectural UC finding cannot by itself explain the prior PGR2 bugcheck or justify a game-performance claim.
Independent review reproduced two incomplete/contradictory capture acceptance defects, both repaired with failing-then-passing tests at cddc308. All eight retained captures pass the stricter analyzer. The eight new checks and full 113-test host suite pass, with three existing skips. The diagnostic changes no release firmware behavior and has no ordinary release overhead. PR #151 passed all 14 exact-head CI checks and was merged at c357f2a7d0a6d8ddd85d44fe619728540b4363f8. This is a diagnostic-only merge; the firmware memory census above remains tied to its original measured build. Physical cold/warm flash, working-set eviction and actual ROM-crossing measurements remain unqualified. #139 and #120 remain open. #140 must separately prove upper-mirror/MCPX overlay admission, reset and save-load; this diagnostic neither remaps nor executes that mirror. Detailed finding and limits.
7. Implementation order with decisive deliverables
| Order | Work | First decisive result | Stop condition |
|---|---|---|---|
| 1 | Resolve existing ROM/pool costs without expanding their scope | Cause of ROM stall/cache latency and refill scan/tail cost on controlled fixtures | Compatibility failure or material cost remains |
| 2 | One PCIIDE lifetime under #142 | Final-consumer/inflight proof and fewer linked resident pages | Valid saved callback, error or reset consumer remains |
| 3 | One worker dependency under #143 | Actual owner, backlog and blocking closure; at most one retirement candidate | Reaper, queue or I/O self-wait; worse completion latency |
| 4 | One filesystem cold arm under #141 | Before/after extents/pages with identical file contracts | Bytes merely move or hot path/compatibility worsens |
| 5 | One BSS/pool packing change | Actual pages returned at peak and settled checkpoints | Only nominal size improves or pressure cost increases |
| 6 | A different PFN design if necessary | Pass the host cost gate before native census | The current packing overhead repeats |
The inventory now exists. The next implementation must prove an exclusive lifetime or reduced physical page population. A larger fixed-platform storage design could remove more, but its minimum compatible closure remains unmeasured and needs its own focused plan.
8. Acceptance for each memory change
- Bind plain main and candidate to exact source/artifacts. Preserve the public procedure, launch route and checkpoint.
- Count each physical page once. Separate title/hardware owners and report unresolved ownership.
- Measure cold boot, title-ready state, post-activity/reset state and temporary peak. Preserve contiguous allocation capacity as well as aggregate free pages.
- Prove callback, return, interrupt and inflight references are gone before donating code/data. Exercise 64/128 MiB, persistence, failed boot, warm reset, save/load and pressure where relevant.
- Measure the changed hot path before broad gameplay. Include lock hold, completion tails and work shifted elsewhere. Final performance claims use ordinary builds.
- Keep ROM bytes, function bytes, image RAM, heap/stack RAM and total checkpoint availability separate. Do not add overlapping closures or hypothetical savings.
- Loss of function or rendering blocks acceptance. Assess cadence, meaningful p95/p99/max changes and CPU per useful progress; one microtest percentage is not automatically a game-level regression.
- Put only maintainable product/test changes in code PRs. Private captures and long research stay outside main; curated numerical findings belong in the wiki.
9. Success is not yet reached
Main has a 696 KiB final availability gap. The optional combined candidate has a 280 KiB gap, but is not game-accepted. Retail 5838 is another close image-size reference; its public allocation probe has not run, so its expected 180 KiB image cannot replace Complex's measured total census.
Success requires a normal 64 MiB release with firmware-owned overhead and usable title allocation capacity equal to or better than the reference, without losing storage, input, reset, rendering or performance. A larger ROM is acceptable. A RAM tax hidden in extra guest memory is not footprint reduction.
Three BIOS comparison guide
Memory-Comparison-and-Removal-Guide reconciles the initial and post-activity gaps, distinguishes image from runtime backing, and lists the owner/lifetime proof needed for each saving. Memory-Three-BIOS-Public-API-Comparison places all 366 reference ordinals in one table, keeping inferred reference walks separate from exact owned compiler extents.
Physical removal decisions and architectural work
Use Memory-Reduction-Workbook for each owner's retirement/capacity requirements and the distinction between a 1% installed-RAM gap and actual title headroom. Use Kernel-Overhaul-Size-and-Performance for the fixed-platform storage, filesystem, worker and metadata overhaul beyond already-completed bounded reductions.