SYSCACHE_ONLY_MODE - MarekBykowski/readme GitHub Wiki

HOME

SYSCACHE_ONLY_MODE β€” cheat sheet

What: boot U-Boot entirely from the 24 MB L3 System Cache used as RAM, no DDR init. Axxia Lionfish (Cortex-A53, Nokia 3x), bsp_u-boot_gen2, branch nokia_3x_images_SYSCACHE_outer_fixed.
Why: Customer had a requirement it is U-Boot toinit memory not SPL before DDR PHY training.

Why L3

  • L1 (per core, ~32K) / L2 (per cluster, ~1–2M) β†’ too small.
  • L3 = 24 MB System Cache on the CCN interconnect (codenames DICKENS/SHELLEY)

3 images on flash

Branch nokia_3x_images: SPL / ATF(BL31) / U-Boot as separate images (before: SPL+monitor combined). Boot ROM β†’ SPL β†’ ATF β†’ U-Boot β†’ Linux. Offsets (axc6700.h)

Boot flow: standard vs SYSCACHE

Same TBBR BL skeleton (BL1β†’BL2β†’BL31β†’BL33). The difference: BL2/SPL skips DDR and turns the L3 into RAM, so BL31+BL33 run from cache.

Standard

SYSCACHE_ONLY_MODE β€” SPL jumps to U-Boot at EL3 (runs from L3, inits DDR there); U-Boot drops to EL2 by loading the Secure Monitor (BL31) on demand and calling it β€” the eret is in the monitor, which then returns control to U-Boot now at EL2/NS.

Diffs vs standard:

  1. BL2 turns the 24 MB L3 into RAM instead of initing DDR; U-Boot is loaded into L3.
  2. U-Boot runs at EL3 from L3 and inits DDR there (init_mem_axxia). start.S enters at EL3 and stays (the armv8_switch_to_el2 in start.S is under MULTIENTRY, slaves only).
  3. EL3β†’EL2 is done by loading BL31 on demand β€” switch_to_EL2_non_secure (common/board_f.c:825) reads the Secure Monitor from flash 0x300000 into LSM+0x1000, pre-migrates SP_EL2/VBAR_EL2, then calls it (entry(NULL, cfg)). The eret is in the monitor (TF-A/BL31) binary, not in U-Boot β€” which is why it is not in the U-Boot source. The monitor erets EL3β†’EL2/NS and returns to U-Boot (comment "We are now at EL2, non-secure state" right after entry()). (The board's own armv8_switch_to_el2() at axxia.c:75 is dead #if 0.)

Caveat: PSCI is disabled; secondaries come up via spin-table (enable-method = "spin-table", 92c327575a).

ATF (BL31) side β€” how the monitor returns to where U-Boot called it

The eret back to U-Boot uses no hardcoded address β€” it reuses the normal AArch64 call return, i.e. x30 from U-Boot's call into the monitor (the instruction right after the call β€” effectively PC+4).

  1. U-Boot calls the monitor as an ordinary function β€” common/board_f.c:889: 
    entry = (void (*)(void *, void *))(LSM + 0x1000);  /* monitor loaded here */
entry(NULL, axxia_configuration);   /* compiled to `bl` β†’ x30 = next insn (board_f.c:890) */
  2. Monitor entry stashes the caller's LR β€” bl31/aarch64/bl31_entrypoint.S:47–49: 
    ldr x22, =__return_addr_for_el3
str x30, [x22]        /* __return_addr_for_el3 = x30 from U-Boot's `bl` */
    (also saves SP β†’ __sp_addr_for_el3 and VBAR_EL3 β†’ __vbar3_to_migrate.)
  3. Monitor sets that as the eret target β€” plat/intel/axxia/bl31_plat_setup.c:201: 
    bl33_ep_info.pc   = *(uintptr_t*) get_return_addr_for_el3();  /* ELR_EL3 = saved x30 */
bl33_ep_info.spsr = SPSR_64(MODE_EL2, MODE_SP_ELX, 0);        /* eret β†’ EL2/NS */
    TF-A then erets to bl33_ep_info.pc β†’ U-Boot resumes at the instruction right after entry(...) (board_f.c:890), now at EL2/NS. From U-Boot's view entry() simply "returned", only the Exception Level changed EL3β†’EL2.

entered_at is informational only: board_f.c:881 snapshots a label before the call (asm volatile ("adr %0, .\n" : "=r"(cfg->entered_at))) and the monitor only prints it ("Uboot branched from 0x%lx"). It is not the return path β€” the real return is x30 above.

Both sides must agree: axxia_configuration_t is declared identically in U-Boot (board_f.c:814) and ATF (axxia_def.h), and BL31 is linked at TZRAM_BASE = LSM + 0x1000 (platform_def.h) to match where U-Boot copies it. ATF commits: 68fdb6b142 (3 images, U-Boot-entered comment + entered_at), d23c387a96 (always-on xlat-table print).

The sequence (board_init_f, spl.c:1558–1612)

  1. L3 β†’ FULL β€” outer_cache.S sets HN-F mode FULL (__asm_enable_l3_cache). Modes: OFF / SFONLY / HALF / FULL; 7 HN-F nodes (id 0x20–0x26) on CCN.
  2. setup_security() β€” unlock CCN + GPDMA (secure/external boot). secure.c:51
  3. set_cluster_coherency(c,1) β€” add clusters to coherency domain (must precede L3 use). l3_coherency.c:37
  4. init_l3() β€” GPDMA-zero all 24 MB β†’ allocate the lines. spl.c:923
  5. mmu_configure(pgt, DISABLE_DCACHE) β€” page tables Normal, cacheable, Outer Shareable (the crux: outer-share = held/coherent at L3). MMU on, D-cache still off. cache_v8.c (PMD_SECT_OUTER_SHARE)
  6. pt_walk(...) β€” prime TLB (RAM, DICKENS, UART, LSM). spl.c:1581
  7. enable D-cache — set_sctlr(|CR_C) (cache AFTER TLB — safe MMU→TLB→cache order).
  8. load_image_mem(CONFIG_UBOOT_OFFSET) — CPU0 copies U-Boot SPI→addr 0 → lands in L3. spl.c:938
  9. flush D$ / invalidate I$ β€” .text written via D$, fetched via I$.
  10. jump β€” entry=(void*)0x0; (*entry)(NULL,NULL) β†’ U-Boot runs from L3, no DDR.

Killer bug (commit 12c9f97fc8)

PC-relative literal ldr x13,=(DICKENS+0x10) β†’ ldr x13,[pc-0x110] pulled the literal's cache line Modified; cleanup_before_linux() then cleared C/M bits β†’ dirty line dropped (lost the DICKENS address). Rule: between disabling and flushing caches, touch only local (stack) data. Fix: no PC-relative literals in that window + barriers.

Key commits

a379a81290 3 images · 96e3ab5dbd drop GPDMA, CPU-load · cf06f3dab3 SH→outer + cluster coherency · 12c9f97fc8 critical cache fix · 1ecab7fd13 working finally.