RPI 5 Zephyr - oleksiimoisieiev/rpi5-knowledge-base GitHub Wiki
repo: https://github.com/GrygiriiS/zephyr.git
branch: zephyr-v3.6.0-xt-rpi5-dom0-wip
Based on:
repo: https://github.com/Deedone/zephyr
branch: zephyr-v3.6.0-xt-v7
west build -b rpi_5 -p always samples/hello_world
There are two options to build Zephyr with XEN (Dom0) support:
- using snippet xen_dom0
west build -b rpi_5 -p always -S xen_dom0 samples/hello_world/
- using XEN specific board
west build -b rpi_5_xen_dom0 -p always samples/hello_world
To build Zephyr blinky example replace samples/hello_world vs samples/basic/blinky in the above commands.
Copy zephyr.bin binary to RPI5 bootfs boot partition.
In RPI5 config.txt file select zephyr.bin binary to boot it instead of standard RPI5 kernel.
[all]
enable_uart=1
uart_2ndstage=1
kernel=u-boot.bin
Assumption: XEN (xen) and u-boot (u-boot.bin) binaries prepared and placed in RPI5 bootfs partition.
Copy Zephyr zephyr.bin binary to RPI5 bootfs partition.
Note
Note that boot uses FDT prepared by RPI5 firmware and placed at ${fdt_addr}.
u-boot command to start XEN, which starts Zephyr:
setenv xen_addr_r 0x02700000
fatload mmc 0 ${kernel_addr_r} zephyr.bin
fdt addr ${fdt_addr} 0x40000
fdt resize
fdt chosen
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
setenv bootargs "console=dtuart dtuart=/soc/serial@7d001000 dom0_mem=128M"
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=/soc/serial@7d001000 dom0_mem=128M"
fdt rm /soc/serial@7d001400
fdt rm /soc/serial@7d001a00
fdt set /soc/gpio@7d517c00 xen,passthrough
fdt print /chosen
printenv
fatload mmc 0 ${xen_addr_r} xen
booti ${xen_addr_r} - ${fdt_addr}
XEN boot log:
Xen 4.19-unstable
(XEN) Xen version 4.19-unstable (grygorii@) (aarch64-zephyr-elf-gcc (Zephyr SDK 0.16.3) 12.2.0) debug=y Thu Apr 11 15:33:36 EEST 2024
(XEN) Latest ChangeSet: Fri Jan 5 10:46:44 2024 +0200 git:8ff1f6708a
(XEN) build-id: 81400877508840d2162ff03548bb83af12be760b
(XEN) Processor: 00000000414fd0b1: "ARM Limited", variant: 0x4, part 0xd0b,rev 0x1
(XEN) 64-bit Execution:
(XEN) Processor Features: 1100000010111112 0000000000000010
(XEN) Exception Levels: EL3:64 EL2:64 EL1:64 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD
(XEN) Debug Features: 0000000010305408 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000101122 0000000010212122
(XEN) ISA Features: 0000100010211120 0000000000100001
(XEN) 32-bit Execution:
(XEN) Processor Features: 0000000010010131:0000000000010000
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN) Extensions: GenericTimer
(XEN) Debug Features: 0000000004010088
(XEN) Auxiliary Features: 0000000000000000
(XEN) Memory Model Features: 0000000010201105 0000000040000000
(XEN) 0000000001260000 0000000002122211
(XEN) ISA Features: 0000000002101110 0000000013112111 0000000021232042
(XEN) 0000000001112131 0000000000010142 0000000001011121
(XEN) Using SMC Calling Convention v1.2
(XEN) Using PSCI v1.1
(XEN) SMP: Allowing 4 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 54000 KHz
(XEN) GICv2 initialization:
(XEN) gic_dist_addr=000000107fff9000
(XEN) gic_cpu_addr=000000107fffa000
(XEN) gic_hyp_addr=000000107fffc000
(XEN) gic_vcpu_addr=000000107fffe000
(XEN) gic_maintenance_irq=25
(XEN) GICv2: 320 lines, 4 cpus, secure (IID 0200143b).
(XEN) XSM Framework v1.0.1 initialized
(XEN) Initialising XSM SILO mode
(XEN) Initialized GSX IRQ
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Allocated console ring of 32 KiB.
(XEN) CPU0: Guest atomics will try 17 times before pausing the domain
(XEN) Bringing up CPU1
(XEN) CPU1: Guest atomics will try 15 times before pausing the domain
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
(XEN) CPU2: Guest atomics will try 15 times before pausing the domain
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
(XEN) CPU3: Guest atomics will try 15 times before pausing the domain
(XEN) Brought up 4 CPUs
(XEN) CPU 3 booted.
(XEN) I/O virtualisation disabled
(XEN) P2M: 40-bit IPA with 40-bit PA and 16-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x00000000800a3558
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) alternatives: Patching with alt table 00000a00002e5b58 -> 00000a00002e6cc8
(XEN) CPU0 will use 24 loops workaround on exception entry
(XEN) CPU3 will use 24 loops workaround on exception entry
(XEN) CPU1 will use 24 loops workaround on exception entry
(XEN) CPU2 will use 24 loops workaround on exception entry
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000000080000
(XEN) Loading ramdisk from boot module @ 000000002dfaa000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000060000000-0x00000080000000 (512MB)
(XEN) Grant table range: 0x00000002800000-0x00000002840000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) d0: extended region 0: 0x2a00000->0x3fa00000
(XEN) d0: extended region 1: 0x40000000->0x5fe00000
(XEN) d0: extended region 2: 0x80000000->0x1ffe00000
(XEN) Loading zImage from 0000000000080000 to 0000000060000000-0000000060021004
(XEN) Loading d0 initrd from 000000002dfaa000 to 0x0000000068200000-0x00000000692554fe
(XEN) Loading d0 DTB to 0x0000000068000000-0x00000000680130d2
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) PLEASE SPECIFY dom0_mem PARAMETER - USING 512M FOR NOW
(XEN) ***************************************************
(XEN) 3... 2... 1...
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 344kB init memory.
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER24
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER28
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER32
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER36
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
*** Booting Zephyr OS build v3.6.0-61-gdd5a70f99dbf ***
Hello World! Raspberry Pi 5 Xen Dom0
The below Kconfigs have to be enabled:
CONFIG_XEN_DOM0
The hypervisor node has to be added in Zephyr device tree which should use addresses allocated by XEN for Grant table. Look for below line in XEN log:
(XEN) Grant table range: 0x00000002800000-0x00000002840000
so hypervisor node for Zephyr should be defined like:
hypervisor: hypervisor@2700000 {
compatible = "xen,xen";
reg = <0x00 0x2800000 0x40000>;
interrupts = <GIC_PPI 0x0f IRQ_TYPE_EDGE IRQ_DEFAULT_PRIORITY>;
interrupt-parent = <&gic>;
};
Zephyr has to be built with memory range where XEN relocating Zephyr binary. Look for below line in XEN log:
(XEN) Loading zImage from 0000000000080000 to 0000000060000000-0000000060038004
so memory for Zephyr should be defined like:
zephyr,sram = &ram;
ram: memory@60000000 {
device_type = "mmio-sram";
reg = <0x00 0x60000000 DT_SIZE_M(16)>;
};
Console has to be defined using xen_consoleio_hvc node label as shown below (with assumption Zephyr is Dom0) and enabled as default Zephyr console:
chosen {
zephyr,console = &xen_consoleio_hvc;
zephyr,shell-uart = &xen_consoleio_hvc;
};
xen_consoleio_hvc: hvc {
compatible = "xen,hvc-consoleio";
status = "okay";
};
and HW UART used as console by native Zephyr shall be disabled:
&uart0 {
status = "disabled";
current-speed = <115200>;
};
The command sequence in https://github.com/oleksiimoisieiev/rpi5-knowledge-base/wiki/RPI-5-Zephyr/_edit#zephyr-boot-under-xen-from-u-boot can be placed in u-boot script file, compiled and then used to run Zephyr under XEN.
For example, u-boot script file is boot-xen-zephyr.txt, run below command to compile it in u-boot scr format:
mkimage -A arm -T script -d boot-xen-zephyr.txt boot-xen-zephyr.scr
Copy boot-xen-zephyr.scr binary to RPI5 bootfs partition and run below commands in u-boot to run it:
fatload mmc 0 ${scriptaddr} boot-xen-zephyr.scr
source ${scriptaddr}
PRs with initial Zephyr RPI5 support:
https://github.com/zephyrproject-rtos/zephyr/pull/70538
https://github.com/zephyrproject-rtos/zephyr/pull/70774
XEN on ARM:
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Allwinner