RPI 5 Xen dom0less exercise is intended to demonstrate how RPI 5 Linux can be started as driver domain which can access number of RPI 5 hardware.

Dom0less is a set of Xen features that enable the deployment of a Xen system without an control domain (often referred to as “dom0”). Each feature can be used independently from the others, unless otherwise stated. This feature enables Xen to create a set of DomUs at boot time.

Actually, it still creates and starts Dom0, which then can be used to start additional domains. During System boot all domains, described in DT, are started by Xen and booted in parallel which is, actually, a main purpose of Xen dom0less boot mode - speed up boot time.

But for the purpose of this exercise the Xen dom0less boot mode is selected because:

RPI 5 (bcm2712) has gicv2 (oldish);
RPI 5 (bcm2712) doesn't have Xen friendly IOMMU;
RPI 5 (bcm2712) Ethernet is behind PCIE;
RPI 5 MMC and Ethernet HW can't be simply passed through to RPI 5 Linux DomU.

As result, the RPI 5 Linux DomU should be configured to use:

direct-map (guest physical address == physical address);
static memory;
no iommu.

And all above features are now supported for ARM only in Xen dom0less boot mode.

See:

Creating Multiple Domains directly from Xen

Dom0-less Device Passthrough

                  Start                                                       
┌────────────┬───────────────────────────┬───────────────────────────┐        
│            │                           │                           │        
│            │                           │                           │        
│            │                           │                           │        
│            │                    ┌──────▼──────────────┐   ┌────────▼───────┐
│            │                    │ DomU0/DomD          │   │ DomU1          │
│   ┌────────▼────────────────┐   │ (MMC,ETH)           │   │ (no hw)        │
│   │                         │   │                     │   │                │
│   │   zephyr-hellow-wRPIO   │   │                     │   │                │
│   │                         │   │                     │   │                │
│   │                         │   │                     │   │                │
│   │                         │   │                     │   │                │
│   ├──┬────────────────┬─────┤   │                     │   │                │
│   │  │                │     │   │                     │   │  ┌───────────┐ │
│   │  │ zephyr-xenlib  │     │   │                     │   │  │           │ │
│   │  └────────────────┘     │   │                     │   │  │RamDisk    │ │
│   │                         │   │                     │   │  └───────────┘ │
│   │                         │   │                     │   │                │
│   │ Zephyr OS (dom0)        │   │RPI Linux            │   │RPI Linux       │
│   └─────────────────────────┘   └───┬───────────┬─────┘   └────────────────┘
│                                     │           │                           
│   ┌─────────────────────────────────┼───────────┼──────────────────────────┐
└───┤ XEN dom0less                    │           │                          │
    └─────────────────────────────────┼───────────┼──────────────────────────┘
                                      │           │                           
    ┌─────────────────────────────────┼───────────┼──────────────────────────┐
    │ Raspberry Pi 5                  │           │                          │
    │                          ┌──────▼─────┐ ┌───▼──────────────┐           │
    │                          │sdio1: mmc  │ │pcie2:rp1:rp1_eth │           │
    │                          └────────────┘ └──────────────────┘           │
    │                                                                        │
    └────────────────────────────────────────────────────────────────────────┘

Boot process:

RPI5 bootloder started u-boot
u-boot executes boot script:
- loads Zephyr Dom0 binary
- loads RPI 5 Linux DomU binary for DomU0
- loads RPI 5 Linux DomU binary for DomU1
- loads Xen Passthrough device tree binary for DomU0
- loads ramdisk binary for DomU1
- loads Linux System device tree device tree binary
- applies changes to Linux System device tree device tree for Xen dom0less boot mode
- loads Xen binary and starts Xen
Xen is booted and starts Zephyr Dom0, RPI 5 Linux DomU0 and DomU1

Expected result:

RPI 5 Linux DomU0 (DOM1) console is accessible;
RPI 5 Linux DomU1 (DOM2) console is accessible
- MMC rootfs mounted
- RPI login can be executed
- Ethernet eth0 is available and working.

Note

During dom0less boot the DomU numbering can change - for example, DomU0 in device tree can be referenced as DOM1 or as DOM2 in Xen console output.

Note

The pinctrl is not passed through to `RPI 5 Linux kernel' and MMC replies on pinctrl configuration done by u-boot.

For this exercise to run following item should be created/prepared:

u-boot: can be used the same as described here RPI 5 u‐boot;
Xen: with dom0less and static memory support;
RPI 5 Linux kernel: with Xen support, the same binary will be use for both DomU0/DomU1;
Zephyr as Dom0: the simple RPI 5 Zephyr (https://github.com/xen-troops/meta-xt-rpi5/wiki/RPI-5-Zephyr#zephyr-dom0-sampleshello_world-build)
Xen Passthrough device tree: with data to passthrough MMC sdio1: mmc@fff000 and Ethernet pcie@120000->rp1->ethernet@100000;
ramdisk: minirootfs like alpine-minirootfs-3.18.5-aarch64.tar.gz
boot script: the ImageBuilder will be used to create boot script.

Prepare ImageBuilder project

Get ImageBuilder will be used to create boot script.

Download ImageBuilder configuration file ib_linux_domd.config in ImageBuilder folder.

Download ImageBuilder ib_boot_extra.source file containing extra u-boot commands to be added to the boot script before the boot command in ImageBuilder folder.

U-boot

Follow instructions in RPI 5 u‐boot to get u-boot binary and boot it.

Xen

Follow general instructions in RPI 5 Xen to build Xen with below additional step.

Once build is finished copy xen binary to ImageBuilder folder.

The Xen should be configured to support static memory, to enable it apply below diff:

diff --git a/xen/arch/arm/configs/arm64_defconfig b/xen/arch/arm/configs/arm64_defconfig
index e69de29bb2..66e45a743f 100644
--- a/xen/arch/arm/configs/arm64_defconfig
+++ b/xen/arch/arm/configs/arm64_defconfig
@@ -0,0 +1,2 @@
+CONFIG_STATIC_MEMORY=y
+CONFIG_UNSUPPORTED=y

RPI 5 Linux kernel

Sources

repo: https://github.com/raspberrypi/linux.git

branch: rpi-6.6.y

commit: 37573248884ec252165a246df21642c52573f013

Adaptation

Apply below changes before building RPI 5 Linux kernel:

Enable XEN support

diff --git a/arch/arm64/configs/bcm2712_defconfig b/arch/arm64/configs/bcm2712_defconfig
index 875b905be102..3351a494e4e7 100644
--- a/arch/arm64/configs/bcm2712_defconfig
+++ b/arch/arm64/configs/bcm2712_defconfig
@@ -1673,3 +1673,5 @@ CONFIG_SCHED_TRACER=y
 CONFIG_BLK_DEV_IO_TRACE=y
 # CONFIG_UPROBE_EVENTS is not set
 # CONFIG_STRICT_DEVMEM is not set
+
+CONFIG_XEN=y

Apply fix to sdhci-brcmstb MMC driver to fix boot without pinctrls:

diff --git a/drivers/mmc/host/sdhci-brcmstb.c b/drivers/mmc/host/sdhci-brcmstb.c
index 61cea31cc891..13b522031fb2 100644
--- a/drivers/mmc/host/sdhci-brcmstb.c
+++ b/drivers/mmc/host/sdhci-brcmstb.c
@@ -553,16 +553,17 @@ static int sdhci_brcmstb_probe(struct platform_device *pdev)
        priv->pinctrl = devm_pinctrl_get(&pdev->dev);
        if (IS_ERR(priv->pinctrl)) {
                        no_pinctrl = true;
-       }
-       priv->pins_default = pinctrl_lookup_state(priv->pinctrl, "default");
-       if (IS_ERR(priv->pins_default)) {
-                       dev_dbg(&pdev->dev, "No pinctrl default state\n");
-                       no_pinctrl = true;
-       }
-       priv->pins_sdex = pinctrl_lookup_state(priv->pinctrl, "sd-express");
-       if (IS_ERR(priv->pins_sdex)) {
-                       dev_dbg(&pdev->dev, "No pinctrl sd-express state\n");
-                       no_pinctrl = true;
+       } else {
+               priv->pins_default = pinctrl_lookup_state(priv->pinctrl, "default");
+               if (IS_ERR(priv->pins_default)) {
+                               dev_dbg(&pdev->dev, "No pinctrl default state\n");
+                               no_pinctrl = true;
+               }
+               priv->pins_sdex = pinctrl_lookup_state(priv->pinctrl, "sd-express");
+               if (IS_ERR(priv->pins_sdex)) {
+                               dev_dbg(&pdev->dev, "No pinctrl sd-express state\n");
+                               no_pinctrl = true;
+               }
        }
        if (no_pinctrl || !priv->sde_ioaddr || !priv->sde_ioaddr2) {
                priv->pinctrl = NULL;

Build

Run below commands to build RPI 5 Linux kernel:

make ARCH=arm64 CROSS_COMPILE=<path>/zephyr-sdk-0.16.3/aarch64-zephyr-elf/bin/aarch64-zephyr-elf- bcm2712_defconfig
make ARCH=arm64 CROSS_COMPILE=<path>/zephyr-sdk-0.16.3/aarch64-zephyr-elf/bin/aarch64-zephyr-elf-

Deploy

Once build is finished copy:

arch/arm64/boot/Image.gz kernel binary to ImageBuilder folder and rename it to `kernel_2712.img';
arch/arm64/boot/dts/broadcom/bcm2712-rpi-5-b.dtb device tree binary to ImageBuilder folder.

Zephyr Dom0

Follow instructions in RPI 5 Zephyr to build Zephyr Dom0 samples/hello_world application.

Once build is finished copy zephyr.bin binary to ImageBuilder folder.

In this exercise Zephyr Dom0 is actually a NOP - it only started and prints "Hello world!".

Xen Passthrough device tree

Download ImageBuilder Xen Passthrough device tree file linux_mmc_eth_passthrough.dts in ImageBuilder folder. Ensure file name is linux_mmc_eth_passthrough.dts.

Use DTC compiler to build 'Xen Passthrough device tree'

dtc linux_mmc_eth_passthrough.dts -o linux_mmc_eth_passthrough.dtb

ramdisk

Use miniramfs, run below:

wget https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/aarch64/alpine-minirootfs-3.18.5-aarch64.tar.gz

mkdir -p rootfs
cd rootfs
tar xzf ../alpine-minirootfs-3.18.5-aarch64.tar.gz

Create init file with content:

#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev
# Set loglevel to debug to see the messages from the module on the console
echo \"7\" > /proc/sys/kernel/printk
/bin/sh

Then run commands:

chmod +x init
find . | cpio --create --format='newc' | gzip > ../initrd_domU1

Copy initrd_domU1 binary to ImageBuilder folder.

Prepare Xen dom0less u-boot script

Once you are here there ImageBuilder folder should contain following files:

bcm2712-rpi-5-b.dtb
ib_boot_extra.source
ib_linux_domd.config
initrd_domU1
kernel_2712.img
linux_mmc_eth_passthrough.dtb
linux_mmc_eth_passthrough.dts
xen
zephyr.bin

Run below command to generate Xen dom0less u-boot script:

bash ./scripts/uboot-script-gen -c ib_linux_domd.config -d .

It should generate two files boot.scr and boot.source, and produce output like below:

Generated uboot script boot.scr, to be loaded at address 0xC00000:
fatload mmc 0 0xC00000 boot.scr; source 0xC00000

RPI 5 Xen dom0less boot

It is expected that RPI 5 already configured to boot u-boot.

To perform `Xen dom0less boot' copy below files to RPI 5 bootfs boot partition:

boot.scr
bcm2712-rpi-5-b.dtb
initrd_domU1
kernel_2712.img
linux_mmc_eth_passthrough.dtb
xen
zephyr.bin

Boot RPI 5 - it may pick up and start boot script automatically or above command fatload mmc 0 0xC00000 boot.scr; source 0xC00000 can be used from u-boot command line.

boot log

** Booting bootflow '[email protected]_1' with script
3698692 bytes read in 156 ms (22.6 MiB/s)
9393545 bytes read in 393 ms (22.8 MiB/s)
7193 bytes read in 3 ms (2.3 MiB/s)
9393545 bytes read in 393 ms (22.8 MiB/s)
3214800 bytes read in 136 ms (22.5 MiB/s)
1114120 bytes read in 48 ms (22.1 MiB/s)
80198 bytes read in 5 ms (15.3 MiB/s)
Working FDT set to 2e00000
chosen {
        xen,dom0-bootargs = "console=hvc0 earlycon=xen earlyprintk=xen clk_ignore_unused root=/dev/ram0";
        xen,xen-bootargs = "console=dtuart dtuart=/soc/serial@7d001000 dom0_mem=128M dom0_max_vcpus=1";
        #size-cells = <0x00000002>;
        #address-cells = <0x00000002>;
        bootargs = "reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1";
        stdout-path = "serial10:115200n8";
        phandle = <0x00000118>;
        domU1 {
                xen,enhanced = "enabled";
                vpl011 = <0x00000001>;
                cpus = <0x00000001>;
                memory = <0x00000000 0x00080000>;
                #size-cells = <0x00000002>;
                #address-cells = <0x00000002>;
                compatible = "xen,domain";
                module0x2800000 {
                        reg = <0x00000000 0x02800000 0x00000000 0x00310dd0>;
                        compatible = "multiboot,ramdisk", "multiboot,module";
                };
                module0x1E00000 {
                        bootargs = "console=ttyAMA0";
                        reg = <0x00000000 0x01e00000 0x00000000 0x008f5589>;
                        compatible = "multiboot,kernel", "multiboot,module";
                };
        };
        domU0 {
                xen,enhanced = "enabled";
                vpl011 = <0x00000001>;
                direct-map;
                xen,static-mem = <0x00000000 0x40000000 0x00000000 0x20000000>;
                cpus = <0x00000001>;
                memory = <0x00000000 0x00080000>;
                #size-cells = <0x00000002>;
                #address-cells = <0x00000002>;
                compatible = "xen,domain";
                module0x1C00000 {
                        reg = <0x00000000 0x01c00000 0x00000000 0x00001c19>;
                        compatible = "multiboot,device-tree", "multiboot,module";
                };
                module0x1200000 {
                        bootargs = "console=ttyAMA0 earlycon=xen earlyprintk=xen clk_ignore_unused root=PARTUUID=1c507a66-02 rootfstype=ext4 rootwait";
                        reg = <0x00000000 0x01200000 0x00000000 0x008f5589>;
                        compatible = "multiboot,kernel", "multiboot,module";
                };
        };
        dom0 {
                reg = <0x00000000 0x00e00000 0x00000000 0x00387004>;
                compatible = "xen,linux-zimage", "xen,multiboot-module", "multiboot,module";
        };
};
arch=arm
baudrate=115200
board=rpi
board_name=5 Model B
board_rev=0x17
board_rev_scheme=1
board_revision=0xD04170
boot_targets=mmc usb pxe dhcp
bootcmd=bootflow scan
bootdelay=2
cpu=armv8
devnum=0
devtype=mmc
dhcpuboot=usb start; dhcp u-boot.uimg; bootm
distro_bootpart=1
fdt_addr=2df96100
fdt_addr_r=0x02600000
fdt_high=0xffffffffffffffff
fdtaddr=2e00000
fdtcontroladdr=3f732ec0
fdtfile=broadcom/bcm2712-rpi-5-b.dtb
fileaddr=2e00000
filesize=13946
initrd_high=ffffffffffffffff
kernel_addr_r=0x00080000
loadaddr=0x1000000
preboot=pci enum; usb start;
prefix=/
pxefile_addr_r=0x02500000
ramdisk_addr_r=0x02700000
scriptaddr=0x02400000
serial#=41c149f4c727eccd
soc=bcm283x
stderr=serial,vidconsole
stdin=serial,usbkbd
stdout=serial,vidconsole
vendor=raspberrypi

Environment size: 761/16380 bytes
## Flattened Device Tree blob at 02e00000
   Booting using the fdt blob at 0x2e00000
Working FDT set to 2e00000
   Using Device Tree in place at 0000000002e00000, end 0000000002e17fff
Working FDT set to 2e00000

Starting kernel ...

 Xen 4.19-unstable
(XEN) Xen version 4.19-unstable (grygorii@) (aarch64-zephyr-elf-gcc (Zephyr SDK 0.16.3) 12.2.0) debug=y Thu Apr 25 18:58:17 EEST 2024
(XEN) Latest ChangeSet: Wed Apr 24 15:01:56 2024 +0300 git:293cd92b4b
(XEN) build-id: 0c7feb37415251051c82a473f0eb25d7af66df5a
(XEN) Processor: 00000000414fd0b1: "ARM Limited", variant: 0x4, part 0xd0b,rev 0x1
(XEN) 64-bit Execution:
(XEN)   Processor Features: 1100000010111112 0000000000000010
(XEN)     Exception Levels: EL3:64 EL2:64 EL1:64 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305408 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000101122 0000000010212122
(XEN)   ISA Features:  0000100010211120 0000000000100001
(XEN) 32-bit Execution:
(XEN)   Processor Features: 0000000010010131:0000000000010000
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer
(XEN)   Debug Features: 0000000004010088
(XEN)   Auxiliary Features: 0000000000000000
(XEN)   Memory Model Features: 0000000010201105 0000000040000000
(XEN)                          0000000001260000 0000000002122211
(XEN)   ISA Features: 0000000002101110 0000000013112111 0000000021232042
(XEN)                 0000000001112131 0000000000010142 0000000001011121
(XEN) Using SMC Calling Convention v1.2
(XEN) Using PSCI v1.1
(XEN) SMP: Allowing 4 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 54000 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=000000107fff9000
(XEN)         gic_cpu_addr=000000107fffa000
(XEN)         gic_hyp_addr=000000107fffc000
(XEN)         gic_vcpu_addr=000000107fffe000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: 320 lines, 4 cpus, secure (IID 0200143b).
(XEN) XSM Framework v1.0.1 initialized
(XEN) Initialising XSM SILO mode
(XEN) Initialized GSX IRQ
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Allocated console ring of 32 KiB.
(XEN) CPU0: Guest atomics will try 16 times before pausing the domain
(XEN) Bringing up CPU1
(XEN) CPU1: Guest atomics will try 15 times before pausing the domain
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
(XEN) CPU2: Guest atomics will try 12 times before pausing the domain
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
(XEN) CPU3: Guest atomics will try 1 times before pausing the domain
(XEN) Brought up 4 CPUs
(XEN) CPU 3 booted.
(XEN) I/O virtualisation disabled
(XEN) P2M: 40-bit IPA with 40-bit PA and 16-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x00000000800a3558
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) alternatives: Patching with alt table 00000a00002e5e28 -> 00000a00002e6f98
(XEN) CPU0 will use 24 loops workaround on exception entry
(XEN) CPU3 will use 24 loops workaround on exception entry
(XEN) CPU1 will use 24 loops workaround on exception entry
(XEN) CPU2 will use 24 loops workaround on exception entry
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000000e00000
(XEN) Allocating 1:1 mappings totalling 128MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x00000010000000 (128MB)
(XEN) Grant table range: 0x00000002c00000-0x00000002c40000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) d0: extended region 0: 0x2e00000->0x7e00000
(XEN) d0: extended region 1: 0x10000000->0x3fa00000
(XEN) d0: extended region 2: 0x60000000->0x1ffe00000
(XEN) Loading zImage from 0000000000e00000 to 0000000008000000-0000000008387004
(XEN) Loading d0 DTB to 0x000000000fe00000-0x000000000fe1300f
(XEN) *** LOADING DOMU cpus=1 memory=0x80000KB ***
(XEN) Loading d1 kernel from boot module @ 0000000001e00000
(XEN) Loading ramdisk from boot module @ 0000000002800000
(XEN) Allocating mappings totalling 512MB for d1:
(XEN) d1 BANK[0] 0x00000040000000-0x00000060000000 (512MB)
(XEN) d1: extended region 0: 0x60000000->0xc0000000
(XEN) d1: extended region 1: 0x200000000->0x10000000000
(XEN) Loading zImage from 000000001c000000 to 0000000040000000-00000000418dea00
(XEN) Loading d1 initrd from 0000000002800000 to 0x0000000048200000-0x0000000048510dd0
(XEN) Loading d1 DTB to 0x0000000048000000-0x000000004800053d
(XEN) *** LOADING DOMU cpus=1 memory=0x80000KB ***
(XEN) Loading d2 kernel from boot module @ 0000000001200000
(XEN) d2: STATIC BANK[0] 0x00000040000000-0x00000060000000
(XEN) d2: extended region 0: 0x200000->0x37e00000
(XEN) d2: extended region 1: 0x39000000->0x3fa00000
(XEN) d2: extended region 2: 0x60000000->0x1ffe00000
(XEN) Loading zImage from 000000007c000000 to 0000000040000000-00000000418dea00
(XEN) Loading d2 DTB to 0x0000000048000000-0x0000000048001c82
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 348kB init memory.
(XEN) d1v0 Unhandled SMC/HVC: 0x84000050
(XEN) d2v0 Unhandled SMC/HVC: 0x84000050
(XEN) d1v0 Unhandled SMC/HVC: 0x8600ff01
(XEN) d2v0 Unhandled SMC/HVC: 0x8600ff01
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER24
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER24
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER28
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER28
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER32
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER32
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER36
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER36
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) d2v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) DOM1: [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]
(XEN) DOM1: [    0.000000] Linux version 6.6.25-v8-16k+ (grygorii@epuakyiw0a98) (aarch64-zephyr-elf-gcc (Zephyr SDK 0.16.3) 12.2.0, GNU ld
(XEN) DOM1:  (Zephyr SDK 0.16.3) 2.38) #14 SMP PREEMPT Thu Apr 25 18:55:16 EEST 2024
(XEN) DOM2: [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]
(XEN) DOM1: [    0.000000] KASLR disabled due to lack of seed
(XEN) DOM2: [    0.000000] Linux version 6.6.25-v8-16k+ (grygorii@epuakyiw0a98) (aarch64-zephyr-elf-gcc (Zephyr SDK 0.16.3) 12.2.0, GNU ld
(XEN) DOM1: [    0.000000] Xen 4.19 support found

...

uart:~$ (XEN) *** Serial input to DOM1 (type 'CTRL-a' three times to switch input)
(XEN) ~ # ls
(XEN) bin    etc    init   media  opt    root   sbin   sys    usr
(XEN) dev    home   lib    mnt    proc   run    srv    tmp    var
(XEN) ~ # (XEN) DOM2: [FAILED] Failed to start NetworkMan…[0m - Network Manager Wait Online.
(XEN) DOM2: 
(XEN) DOM2: Debian GNU/Linux 12 raspberrypi ttyAMA0
(XEN) DOM2: 
(XEN) ~ # (XEN) *** Serial input to DOM2 (type 'CTRL-a' three times to switch input)
(XEN) raspberrypi login: pi
(XEN) Password: 
(XEN) Linux raspberrypi 6.6.25-v8-16k+ #14 SMP PREEMPT Thu Apr 25 18:55:16 EEST 2024 aarch64
(XEN) 
(XEN) The programs included with the Debian GNU/Linux system are free software;
(XEN) the exact distribution terms for each program are described in the
(XEN) individual files in /usr/share/doc/*/copyright.
(XEN) 
(XEN) Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
(XEN) permitted by applicable law.
(XEN) Last login: Sun Mar 17 00:49:51 GMT 2024 on tty1
(XEN) pi@raspberrypi:~$ 
pi@raspberrypi:~$ 
pi@raspberrypi:~$ sudo su
root@raspberrypi:/home/pi# ifconfig eth0 192.168.0.1
root@raspberrypi:/home/pi# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
(XEN) 64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=5.27 ms
(XEN) 64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=0.368 ms
(XEN) ^C
(XEN) --- 192.168.0.2 ping statistics ---
(XEN) 2 packets transmitted, 2 received, 0% packet loss, time 1002ms
(XEN) rtt min/avg/max/mdev = 0.368/2.819/5.270/2.451 ms
(XEN) root@raspberrypi:/home/pi#

RPI 5 XEN specific issues & workarounds

Shared "arm,pl011" IRQ vs Xen console

There are 3 UART devices in RPI5 DT bcm2712.dtsi defined with the same IRQ=121 (uart0: serial@7d001000, uart2: serial@7d001400 and uart5: serial@7d001a00). The Xen doesn't understand that when uart0 is used as Xen console and fails with indication of "already in use irq".

Workaround is to delete uart2 and uart5 nodes from DT. Applied in ib_boot_extra.source as:

fdt rm /soc/serial@7d001400 
fdt rm /soc/serial@7d001a00

"arm,pl011" regs not Xen page aligned

This issue is triggered in Xen dom0less boot mode when all domains are configured to use emulated VUART vpl011. During boot Xen complains with message "vpl011: Can't re-use the Xen UART MMIO region as it is too small.\n"

Workaround is to adjust uart0 regs size to XEN_PAGE. Applied in ib_boot_extra.source as:

fdt set /soc/serial@7d001000 reg <0x7d001000 0x1000 >

Wrong regs in RPI 5 DT for nvmem_* nodes

This issue introduced by commit ec2216edd465 ("nvmem: raspberrypi: Enable nvmem otp driver through DT") in RPI 5 Kernel. nvmem_* DT nodes are added under DT /soc path, which use wrong regs format, so XEN fails with "Unable to retrieve address 0 for /soc/nvmem_otp".

Workaround is to delete buggy nvmem_* DT nodes. Applied in ib_boot_extra.source as:

fdt rm /soc/nvmem_otp
fdt rm /soc/nvmem_cust
fdt rm /soc/nvmem_priv
fdt rm /soc/nvmem_mac

The correct approach for nvmem_* is to NOT add them in DT, first of all, as those are FW based software "devices", so shell be added through MFD path. Even if they are added in DT - they shell be placed under firmware node with of_platform_populate() call. This is actually valid for ALL FW based software "devices".

Ref: https://github.com/raspberrypi/linux/issues/6196

PCI2 MSI is not XEN compatible

The MSI parent for pcie2: pcie@120000 is set to point at some mip0: msi-controller@130000:

		pcie2: pcie@120000 {
			compatible = "brcm,bcm2712-pcie";
...
			msi-controller;
			msi-parent = <&mip0>;
                }

		mip0: msi-controller@130000 {
			compatible = "brcm,bcm2712-mip-intc";
			reg = <0x10 0x00130000  0x0 0xc0>;
			msi-controller;
			interrupt-controller;
			#interrupt-cells = <2>;
			brcm,msi-base-spi = <128>;
			brcm,msi-num-spis = <64>;
			brcm,msi-offset = <0>;
			brcm,msi-pci-addr = <0xff 0xfffff000>;
		};

But the mip0: msi-controller@130000 doesn't have IRQs defined in the standard way and instead uses custom properties: brcm,msi-base-spi to define GIC IRQ offset, and brcm,msi-num-spis to define number of GIC IRQs to use. The Xen doesn't understand this and so can't map interrupts to DomU in case of 'xen,passthrough'.

The attempt to add correct interrupts property to the mip0: msi-controller@130000 has failed - no IRQs received by macb 1f00100000.ethernet. Reason unknown.

Workaround: luckily the msi-parent' for pcie2: pcie@120000can be set to point atpcie2node itself. With such configurationmacb 1f00100000.ethernetIRQ are working properly and ping passed. Applied inlinux_mmc_eth_passthrough.dts` as:

		pciex: pcie@1000120000 {
...
//			msi-parent = <&mip0>;
			msi-parent = <&pciex>;
...
                };

RPI 5 Xen Dom0less boot exercise - xen-troops/meta-xt-rpi5 GitHub Wiki

Prepare ImageBuilder project

U-boot

Xen

RPI 5 Linux kernel

Sources

Adaptation

Build

Deploy

Zephyr Dom0

Xen Passthrough device tree

ramdisk

Prepare Xen dom0less u-boot script

RPI 5 Xen dom0less boot

boot log

RPI 5 XEN specific issues & workarounds

Shared "arm,pl011" IRQ vs Xen console

"arm,pl011" regs not Xen page aligned

Wrong regs in RPI 5 DT for nvmem_* nodes

PCI2 MSI is not XEN compatible

⚠️ GitHub.com Fallback ⚠️

RPI 5 Xen Dom0less boot exercise - xen-troops/meta-xt-rpi5 GitHub Wiki

Prepare ImageBuilder project

U-boot

Xen

RPI 5 Linux kernel

Sources

Adaptation

Build

Deploy

Zephyr Dom0

Xen Passthrough device tree

ramdisk

Prepare Xen dom0less u-boot script

RPI 5 Xen dom0less boot

boot log

RPI 5 XEN specific issues & workarounds

Shared "arm,pl011" IRQ vs Xen console

"arm,pl011" regs not Xen page aligned

Wrong regs in RPI 5 DT for nvmem_* nodes

PCI2 MSI is not XEN compatible

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️