Memory Comparison and Removal Guide - Mainkill1/SpiritBoot GitHub Wiki

Memory comparison and removal guide

SpiritBoot needs to recover 174 physical pages / 696 KiB to match the retained Complex allocation/lifetime checkpoint. Of that difference, 57 pages / 228 KiB are image backing and 117 pages / 468 KiB are outside the image. Reducing the image alone cannot reach parity. This guide connects the retail 5838, Complex and SpiritBoot inventories to specific changes that can recover usable RAM while preserving game behavior and performance.

The primary objective is an ordinary 64 MiB machine with at least the reference's usable allocation capacity. A ROM container up to 1 MiB is acceptable. The 128 MiB firmware with a 64 MiB title limit remains a separate compatibility experiment: hiding capacity does not reduce the firmware's physical footprint.

Measurement identity and confidence

The main measurements below belong to the completed allocation/lifetime guest and reconstructed source tree 6e7666c73c916a235d29ec793dc9ee0920df299b, recorded for product revision f60f8de605cbbdce2f22e4bc3489e1c1487dcb59. The retained flash is 75bee35302ea9b004f1e6fc1ea9d1e5857985c96489635c56b7461a46849ca73. The later verified main head c357f2a7d0a6d8ddd85d44fe619728540b4363f8 adds the standalone ROM diagnostic; the census is still attributed to its measured build, rather than claimed as a fresh measurement of that later head.

Complex means the specific operator-supplied Complex4627 image, not every variant bearing that name. Its allocation/lifetime guest completed. Retail means the separately supplied 5838 image. Its kernel and storage initialization were observed, but the public guest did not reach BEGIN/END; retail's final allocation availability is therefore unknown. The reference captures are retained experiments, not a newly synchronized three-BIOS campaign.

All sizes use 4,096-byte pages and 1,024-byte KiB. Physical-page totals charge aliases once. A virtual section, a public statistic, a live object and the physical page containing it are different accounting levels.

The size comparison

Quantity Complex4627 Retail 5838 SpiritBoot measured main
Supplied ROM container 1,048,576 B 262,144 B 262,144 B
Distinct repeated ROM quarter 262,144 B; four identical quarters Not classified as repeated quarters Not used to calculate RAM
Expanded kernel image envelope 636,160 B 625,824 B See the owned section ledger; file size is not the resident charge
Retained executable-section contents 153,136 B 156,912 B 328,932 B
Settled image RAM 45 pages / 180 KiB Not measured after INIT retirement 102 pages / 408 KiB
Image RAM at retail's incomplete capture Different checkpoint 153 pages / 612 KiB Different checkpoint
Expected retail image after normal INIT retirement Not applicable Approximately 45 pages / 180 KiB, layout estimate Not applicable
Final available pages in the allocation/lifetime guest 16,226 Unknown 16,052
Final unavailable pages 158 / 632 KiB Unknown 332 / 1,328 KiB
Final unavailable backing outside the image 113 / 452 KiB Unknown 230 / 920 KiB

The last row is an exact subtraction of the measured image from the corresponding total; it is not a classification of every hidden reference owner. It includes title and hardware allocations as well as firmware. Complex's private PFN, heap, cache-reserve and driver owners have not all been established. Retail's 16,213 unresolved physical pages cannot be counted as either free memory or firmware memory.

Reconciliation of the remaining gap

Comparison Difference in pages Difference in KiB Meaning
Final image backing: 102 minus 45 57 228 Linked resident code/data and page placement
Final backing outside the image: 230 minus 113 117 468 Runtime physical populations; reference private owners remain incomplete
Final total: 332 minus 158 174 696 Post-activity availability shortfall
Initial availability: 16,226 minus 16,083 143 572 Separate initial checkpoint
Main initial-to-final change: 16,083 minus 16,052 31 124 Additional retained backing after the authored activity

The initial checkpoint has a useful, narrower arithmetic decomposition: 228 KiB image difference + 136 KiB reported pool difference + 72 KiB reported stack difference + 136 KiB remaining difference = 572 KiB. The remaining 136 KiB is not automatically allocator metadata. Nor can the initial pool and stack counters be substituted into the final physical-owner ledger.

The final parity target at this retained workload is no more than 158 unavailable pages, with the same successful allocations, contents, thread lifetimes and cleanup. A lower unavailable count reached by rejecting title allocations does not satisfy the goal. Free-page parity must also preserve large contiguous-allocation capacity; equal aggregate availability can hide different fragmentation.

Why retail and Complex can have small resident images

Both retain approximately 153–157 KiB of executable-section contents, compared with SpiritBoot's 328,932 B. Retail's executable-section difference is 172,020 B / 167.988 KiB; Complex's is 175,796 B / 171.676 KiB. These are section-content differences, including embedded data and padding, rather than instruction-only totals.

Complex demonstrably retires 111 of 112 pages intersecting its discardable region. Its surviving boundary page also contains live content. Retail advertises 444,626 B of INIT contents, but all 109 INIT-intersecting pages remain mapped in the incomplete capture. That incomplete epoch explains its observed 612 KiB image backing; it does not establish its normal settled footprint.

SpiritBoot already retires INIT/resources at the completed checkpoint and has already removed several obsolete boot allocations. The remaining image gap therefore calls for smaller resident service dependencies or safe alternative placement. Repeating boot-scratch reclamation cannot recover the same pages twice.

The supplied Complex ROM's repeated quarters explain its container size. Compression explains why a 256 KiB retail file expands into a much larger initialization image. Neither explains all runtime allocations. Existing reference traces also do not establish direct ROM calls as the mechanism responsible for the small resident kernel.

Complete inventory and its limits

Inventory Coverage Where to inspect the full records Important limit
Complex inferred entry walks 1,361 roots; 329 distinct public callable entries Memory-Reference-Entry-Walk-Ledger Inferred walks, shared tails and interior entries; not original compiler functions
Retail inferred entry walks 638 roots; 329 distinct public callable entries Retail-5838-Function-Ledger Callback/indirect coverage remains incomplete
Owned compiler extents 1,438 nonzero input-section records SpiritBoot-Main-Emitted-Extent-Ledger Inlining, aliases and padding belong to containing extents
Three-way public ordinal comparison 366 rows, with 330 covered Complex callable slots Memory-Three-BIOS-Public-API-Comparison Entry bodies omit helper closures; unknown remains unknown
Retail typed data and backing Every currently proved format/ABI object and known page Retail-5838-Data-and-Physical-Atlas A public object extent does not prove its private allocation boundary
Complex tables and unknown bytes Disjoint retained-image accounting and bounded access windows Complex-BIOS-Size-and-Layout Bounded table accesses are not complete table sizes
Owned tables and objects Historical declaration/sizeof proofs and captured allocation records Memory-Function-and-Table-Ledger, Memory-Resident-Page-Atlas, Memory-Allocation-Object-Ledger Preserve each page's own build/version scope
Main physical ownership All 332 unavailable pages, with disjoint physical intervals SpiritBoot-Main-Physical-Ownership Includes title/hardware owners, not just firmware

An entry with 37 emitted bytes may only adapt arguments before calling a much larger implementation. For example, the owned ExAllocatePoolWithTag public wrapper is 37 B versus a 439 B inferred reference walk, while the real owned allocator and its metadata reside elsewhere. The same issue affects virtual-memory and file-I/O wrappers. A ratio between these entry sizes is not a subsystem saving estimate.

Bytes that remain unclassified

Image Retained executable-section contents Classified instruction bytes Other disjoint classification Unclassified remainder
Complex 153,136 94,726 prior union + 740 additional prefixes 2,238 format/ABI bytes 55,432 B
Retail 5838 156,912 78,986 union bytes 2,194 format/ABI bytes 75,732 B
SpiritBoot main 328,932 316,892 compiler-extent union, not an instruction-only count Not separately typed by this extraction 12,040 B outside that extent union

The owned row uses a different measurement method and must not be compared as decoder completeness. Unknown reference bytes can include code, tables, embedded data and padding. They are not a hidden dead-code donation. Per-entry decoded sizes overlap; the union, rather than their sum, determines covered bytes.

Where to recover physical pages

The following targets use measured owned extents and owners. They identify a mechanism and a gate; they do not promise that all listed storage can disappear.

Storage and PnP lifetime

Owned storage/filesystem code includes 44,444 B VFAT, 31,568 B IDE, 15,359 B storage-class, 6,895 B XDVDFS, and 4,590 B partition-manager compiler extents. These disjoint family labels total 102,856 B / 100.445 KiB. Another 32,547 B belongs to the I/O family, which serves more than storage. Historical source-family labels are not a current full dependency proof.

Start with #142: the 5,081 B PciIdeXDispatchPnp extent. A runtime callback route retains its body and potentially shared helpers. Replacing only a public dispatch pointer is insufficient if private copies, queued work, completions, timers, returning frames or reset/error consumers still reference it.

The bounded change must identify the final consumer, drain its work, install the resident replacement where necessary, and then discard only its exclusive page-aligned closure. Rebuild and count released pages. A 5,081-byte extent is not an automatic two-page saving when neighboring contents remain live. Keep ATA, class and partition changes independently attributable.

Required compatibility cases include ordinary read/write, retries and failures, media changes, unexpected device requests, warm reset, shutdown and emulator save/load. The broader fixed-platform storage specialization is worthwhile only after a counterfactual build proves which exclusive supporting code and persistent objects it actually removes.

Filesystem resident code

#141 begins with 10,516 B VfatDispatchRequest. Other substantial extents include 3,271 B DoQuery, 2,816 B VfatWrite, 1,761 B VfatMount and 1,752 B vfatGetFCBForFile. Their entry sizes include inlined work and do not define a removable group.

Outline one measured cold arm first, retain required FATX behavior, and compare the actual resident page layout and hot path. Merely moving code to another resident function changes no RAM ownership. A valid smaller service must preserve fragmented chains, create/open, allocation versus EOF, overwrite/reextend, rename, truncate, query behavior, persistent save/reopen, short I/O and error propagation. The previous false-disc and save failures make these boundaries essential.

Pool backing and allocation geometry

Main retains 86 pool-window pages / 344 KiB after activity. This contains live allocations, cached/reserved buffers and reusable capacity. #135 measured 70 pages, recovering 16 pages / 64 KiB through backing reuse. Its remaining cost and tail-latency qualification must remain visible; the measured saving is not a merged default or a cold-boot result.

The 2,093-event mixed lifetime replay reproduced the baseline's 86 pages. Broad finer size-class ladders modeled 117–125 pages, despite reducing some individual rounding. More classes can retain more partly occupied pages. Prioritize actual mixed populations and complete-page recovery.

A 1,024 B request plus the current 8 B header rounds to a 2,048 B block, so a carefully chosen smaller class can improve packing. A 2,384 B request still fits only once per 4 KiB page if its block shrinks to 2,400 B. That change saves physical RAM only when the remainder becomes useful or the allocation lifetime changes.

Preserve contents, addresses of live objects, alignment, query capacity, ordinary/reserved domains, failure cleanup and repeated reuse. Benchmark warm same-class allocation separately from class refills, fragmented retirements and pressure recovery. An expensive reclaimed-page scan cannot be hidden by a test that averages it with unrelated work.

Worker stacks and blocking ownership

The independently scoped thread-owner census finds four executive workers using 48 KiB, one filesystem worker using 12 KiB, and one ATA worker using 12 KiB. The title's 64 KiB stack and the idle stack already charged within image/BSS are separate. Public stack counters and image-backed idle pages must not be added twice.

#143 targets at most one unnecessary service execution context first. Removing one 12 KiB stack and its associated objects could matter; decreasing every stack constant without measuring call depth is unsafe. A single guest CPU does not eliminate blocking dependencies between queues.

Audit direct reaper submissions, queue consumers, backlog, self-waits, IRQL/APC/DPC constraints, shutdown and reset. Hypercritical reaper work can enter from KiRundownThread, outside the ordinary ExQueueWorkItem route. Dynamic escalation and the worker balance manager have already been disabled on Xbox; those are not fresh savings.

PFN and pool metadata

The current dense PFN links use 64 KiB, pin state 32 KiB, and pool metadata 12 KiB. Together they occupy 27 pages / 108 KiB. Dense pin-batch scratch was already removed. The remaining representation carries live owner, free-list, alias multiplicity, pin-count, overflow and deferred-return state.

The five-byte PFN experiment in #144 suggests 16 KiB at 64 MiB or 32 KiB at 128 MiB in storage, but has no accepted native saving. Its host pin/unpin cost worsened approximately 23–48% after refinement. It is a measured rejection of that encoding, not permission to ship for byte count alone.

A new compact representation must preserve all required state and whole-request rollback without allocating while holding the PFN lock. Compare real operation cost before a long title campaign. Installed page-table backing is a separate 17 pages / 68 KiB: reclaim only an actually unused table after all PDE/alias users and translation lifetime are retired.

Readonly tables and shared pages

Owned historical sizeof proofs include 4,870 B Unicode case data, 1,552 B export-directory storage, 1,248 B font data, and multiple status-conversion arrays. The full lists preserve their exact source and build scopes. These are placement/lifetime candidates, not proof that those ABI and runtime services can be removed.

The 12,288 B boot-named stack becomes the live idle stack. The 4,096 B double-fault stack remains a fault resource. Writable cache/allocator tables cannot simply become ROM data. An object that shares a page with other live state saves zero physical pages if only its own bytes are removed.

The ordinary 64 KiB cache and private 32 KiB write-back reserve are included in pool backing. Do not add them again to the 344 KiB total. The reserve prevents recursive FAT metadata reads from deadlocking behind dirty file-data blocks. Any smaller baseline needs a proven replacement for that dependency or allocation before dirty activity, not an emergency allocation after exhaustion.

ROM execution

#118 demonstrates 88 pages / 352 KiB of RAM recovery in the bound ROM-only capture. Mutable and self-modified state stays in RAM. The combined ROM/pool capture leaves 228 unavailable pages / 912 KiB, still 70 pages / 280 KiB behind Complex. The combined measurement exists; these figures are not a sum of unrelated historical experiments.

The combined candidate has 14 image RAM pages / 56 KiB. Its non-image backing is therefore 214 pages / 856 KiB, versus Complex's 113 pages / 452 KiB: 404 KiB more outside the image. Its image advantage of 124 KiB offsets part of that, leaving the 280 KiB total gap. This demonstrates why ROM placement alone is not the whole memory solution.

#139 and merged #151 establish that the tested lower ROM leaf has effective UC cache type, while its exact-byte RAM control is WB. The narrow warm TCG test did not show a clear penalty, but TCG's host-backed ROM reads do not measure physical Xbox flash latency. Cold-cache cost, larger working sets and instruction-boundary crossings remain separate gates.

The historical PGR2 bugcheck tracked in #120 remains unresolved. A later successful diagnostic path does not replace title/reset/save/load qualification. Upper-mirror placement, one hot RAM page and one hot/cold linker cluster have their own issues #140, #137 and #138.

Complex's sampled public/kernel entries are RAM-backed. Startup ROM execution is observed; runtime ROM execution was unobserved in the two retained public-API workloads. Those limited observations do not establish every error, shutdown or warm-reset path. ROM mappings alone do not prove functions were executed from them.

Evidence required for each removal

Each focused change needs a before/after ledger with these fields:

Field What it must establish
Artifact and source identity Exact tested executable/firmware, owned source tree and matching symbol map
Physical owner and checkpoint Initial, post-activity and peak pages; title/hardware allocations separated where known
Runtime root ABI export, saved callback, interrupt, queue, returning frame or active mapping keeping it live
Last-use boundary When every consumer is retired; how late calls are detected
Exclusive closure Code/data that can disappear without removing shared consumers
Linked and physical result Unique pages actually released; bytes moved without page recovery recorded separately
Compatibility Relevant API statuses, file/byte oracles, reset, save/load and title progression
Cost Hot-path work, stalls, total CPU, guest cadence, p95/p99 and repeated max spikes where applicable
Failure behavior Fallback, bounded cleanup, no stale pointer/mapping, and preserved allocation-failure semantics

A symbol map must match the actual tested firmware, not a mutable build-directory path retained in an old manifest. An unobserved function remains unknown until its public, callback, interrupt, failure and reset consumers are addressed. A successful build establishes neither last use nor recovered RAM.

Performance acceptance considers useful progress, CPU efficiency, latency and correctness together. A single synthetic leaf moving by roughly 1–2% is not automatically a system regression. Conversely, a memory reduction with a repeated material frame-time loss, p95/p99 increase or compatibility failure cannot be accepted solely because it frees pages. Evaluate work shifted into later calls or other threads, rather than just the changed function's stopwatch.

Work order and remaining investigation

  1. Finish the bounded pool-reuse cost repair and preserve its measured 64 KiB recovery. Keep timed idle trimming separate.
  2. Prove one storage/PnP final lifetime under #142; discard only its exclusive pages.
  3. Outline one measured filesystem cold arm under #141 and verify whether linked pages actually shrink.
  4. Retire at most one service worker under #143 after blocking ownership is established.
  5. Advance ROM placement only through its existing compatibility and cacheability gates; do not presume reference parity from the smaller image alone.
  6. Continue reference ownership discovery and retail launch qualification where those measurements change a concrete design decision.

Retail's remaining launch rejection, its final free-page census, and the reference's hidden dynamic owners are explicit gaps. The current evidence already identifies the broad difference: SpiritBoot retains a larger resident service implementation and a larger runtime page population. It does not yet prove a particular replacement reaches parity. Acceptance is actual recovered pages with unchanged successful workload behavior, rather than a smaller compressed ROM, a shorter source file, or a hidden upper-memory reserve.

Related detailed pages

Implementation workbook and lower-level overhaul

Memory-Reduction-Workbook adds an owner-by-owner challenge ledger, exact installed-RAM percentages, actual-game pressure questions and a decision format for each removal. Kernel-Overhaul-Size-and-Performance separates completed lifetime/pinch-point changes from the larger retained storage, filesystem, worker and allocator overhaul still to be implemented. Memory-Pool-Reuse-Cost-and-Attribution records the current pool candidate, including unfavorable native costs rather than treating fewer instructions as a measured speed gain.