How to build and setup for Raspberry PI QEMU user.md - tx00100xt/SeriousSamClassic GitHub Wiki
First we need to install the QEMU user static and binfmt packages on HOST system.
sudo apt-get update && sudo apt-get install -y qemu qemu-user-static binfmt-support parted wget dosfstools zip
sudo adduser $USER kvm
Checking the status of the binfmt daemon.
sudo systemctl status systemd-binfmt.service
● systemd-binfmt.service - Set Up Additional Binary Formats Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Mon 2022-06-06 17:19:44 MSK; 10 months 12 days ago Docs: man:systemd-binfmt.service(8) man:binfmt.d(5) https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
Prepare binfmt config for Rasberry PI.
sudo nano -w /etc/binfmt.d/qemu-arm-rpi.conf
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-rpi:
sudo nano -w /etc/binfmt.d/qemu-aarch64-rpi.conf
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-rpi:
Restart the binfmt service.
sudo systemctl restart systemd-binfmt.service
Checking the status of the binfmt daemon again.
sudo systemctl status systemd-binfmt.service
● systemd-binfmt.service - Set Up Additional Binary Formats Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static; vendor preset: enabled) Active: active (exited) since Wed 2023-04-19 19:28:36 MSK; 9s ago Docs: man:systemd-binfmt.service(8) man:binfmt.d(5) https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Process: 6580 ExecStart=/lib/systemd/systemd-binfmt (code=exited, status=0/SUCCESS) Main PID: 6580 (code=exited, status=0/SUCCESS) апр 19 19:28:36 debian systemd[1]: Starting Set Up Additional Binary Formats... апр 19 19:28:36 debian systemd[1]: Started Set Up Additional Binary Formats.
Checking the presence of the binfmt magic:
sudo cat /proc/sys/fs/binfmt_misc/arm
enabled interpreter /usr/bin/qemu-arm-rpi flags: offset 0 magic 7f454c4601010100000000000000000002002800 mask ffffffffffffff00fffffffffffffffffeffffff
sudo cat /proc/sys/fs/binfmt_misc/aarch64
enabled interpreter /usr/bin/qemu-aarch64-rpi flags: offset 0 magic 7f454c460201010000000000000000000200b700 mask fffffffffffffffcfffffffffffffffffeffffff
Let's create a wrapper to switch between configuration for Raspberry PI 3B and Raspberry PI 4
sudo mkdir /home/qemu
sudo mkdir /home/qemu/qemu-user
cd /home/qemu/qemu-user
nano -w qemu-wrapper.c
// From https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Compiling_with_qemu_user_chroot
/*
* pass arguments to qemu binary
* QEMU_CPU environment variable can be unset
*/
#include <string.h>
#include <unistd.h>
#define XSTR(cpu) STR(cpu)
#define STR(cpu) #cpu
int main(int argc, char **argv, char **envp) {
int this_len = strlen(argv[0]);
char staticpath[this_len];
char *newargv[argc + 3];
newargv[0] = argv[0];
newargv[1] = "-cpu";
newargv[2] = XSTR(QEMU_CPU);
memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
newargv[argc + 2] = NULL;
memcpy(staticpath, argv[0], this_len - 1);
staticpath[this_len - 1] = 0;
return execve(staticpath, newargv, envp);
}
Compiling wrappers.
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a7 -O3 -s -o /usr/bin/qemu-arm-rpi3
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a7 -O3 -s -o /usr/bin/qemu-arm-rpi4
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a53 -O3 -s -o /usr/bin/qemu-aarch64-rpi3
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a72 -O3 -s -o /usr/bin/qemu-aarch64-rpi4
Switching wrappers between configurations for Raspberry PI 3B and Raspberry PI 4 in the future will be using the creation of symbolic links.
Create directories for future guests.
sudo mkdir /home/qemu/qemu-user/raspios-armhf-lite
sudo mkdir /home/qemu/qemu-user/raspios-arm64-lite
Clone the repository containing the CPU information needed to emulate the Raspberry PI 3B and 4B
sudo git clone https://github.com/pguyot/arm-runner-action.git
Let's create scripts for logging into the guest system using chroot.
sudo nano -w /home/qemu/qemu-user/run_armhf_rpi3.sh
#!/bin/bash
cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi
cd /home/qemu/qemu-user
cd raspios-armhf-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_3b proc/cpuinfo
chroot . /bin/bash --login
sudo nano -w /home/qemu/qemu-user/run_armhf_rpi4.sh
#!/bin/bash
cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi
cd /home/qemu/qemu-user
cd raspios-armhf-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_4b proc/cpuinfo
chroot . /bin/bash --login
sudo nano -w /home/qemu/qemu-user/run_arm64_rpi3.sh
#!/bin/bash
cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi
cd /home/qemu/qemu-user
cd raspios-arm64-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_3b proc/cpuinfo
chroot . /bin/bash --login
sudo nano -w /home/qemu/qemu-user/run_arm64_rpi4.sh
#!/bin/bash
cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi
cd /home/qemu/qemu-user
cd raspios-arm64-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_4b proc/cpuinfo
chroot . /bin/bash --login
Now our HOST is completely ready.
Next, you need to download and unpack the images of the Raspberry PI operating system into the directories "/home/qemu/qemu-user/raspios-armhf-lite" and "/home/qemu/qemu-user/raspios-arm64-lite"
We go to the page https://www.raspberrypi.com/software/operating-systems/ and select the desired system. In Let's take Raspbian bullseye as an example, the current version is 2023-02-22-raspios-bullseye-armhf.img and 2023-02-22-raspios-bullseye-arm64.img.
Download and unpack the compressed image:
sudo wget http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-02-22/2023-02-21-raspios-bullseye-armhf-lite.img.xz
sudo xz -d 2023-02-21-raspios-bullseye-armhf-lite.img.xz
sudo wget http://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2023-02-22/2023-02-21-raspios-bullseye-arm64-lite.img.xz
sudo xz -d 2023-02-21-raspios-bullseye-armhf-lite.img.xz
First, get information about the image:
LC_ALL=en_US.utf8 fdisk -lu 2023-02-21-raspios-bullseye-armhf-lite.img
Disk 2023-02-21-raspios-bullseye-armhf-lite.img: 1.83 GiB, 1962934272 bytes, 3833856 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x34f4435e Device Boot Start End Sectors Size Id Type 2023-02-21-raspios-bullseye-armhf-lite.img1 8192 532479 524288 256M c W95 FAT32 (LBA) 2023-02-21-raspios-bullseye-armhf-lite.img2 532480 3833855 3301376 1.6G 83 Linux
LC_ALL=en_US.utf8 fdisk -lu 2023-02-21-raspios-bullseye-arm64-lite.img
Disk 2023-02-21-raspios-bullseye-arm64-lite.img: 1.95 GiB, 2097152000 bytes, 4096000 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xe088fd39 Device Boot Start End Sectors Size Id Type 2023-02-21-raspios-bullseye-arm64-lite.img1 8192 532479 524288 256M c W95 FAT32 (LBA) 2023-02-21-raspios-bullseye-arm64-lite.img2 532480 4095999 3563520 1.7G 83 Linux
We hook the entire first section (starts from sector 8192,each sector of 512 bytes) to the loop0 device, and the second section (starts from sector 532480,each sector of 512 bytes) to loop1.
sudo losetup -f --show -o $((8192*512)) 2023-02-21-raspios-bullseye-armhf-lite.img
sudo losetup -f --show -o $((532480*512)) 2023-02-21-raspios-bullseye-armhf-lite.img
sudo mkdir /mnt/rpi_boot
sudo mkdir /mnt/rpi_system
sudo mount /dev/loop0 /mnt/rpi_boot
sudo mount /dev/loop1 /mnt/rpi_system
Check the content of mounted directories.
ls /mnt/rpi_boot
bcm2708-rpi-b.dtb bcm2710-rpi-3-b.dtb bootcode.bin fixup.dat overlays bcm2708-rpi-b-plus.dtb bcm2710-rpi-3-b-plus.dtb cmdline.txt fixup_db.dat start4cd.elf bcm2708-rpi-b-rev1.dtb bcm2710-rpi-cm3.dtb config.txt fixup_x.dat start4db.elf bcm2708-rpi-cm.dtb bcm2710-rpi-zero-2.dtb COPYING.linux issue.txt start4.elf bcm2708-rpi-zero.dtb bcm2710-rpi-zero-2-w.dtb fixup4cd.dat kernel7.img start4x.elf bcm2708-rpi-zero-w.dtb bcm2711-rpi-400.dtb fixup4.dat kernel7l.img start_cd.elf bcm2709-rpi-2-b.dtb bcm2711-rpi-4-b.dtb fixup4db.dat kernel8.img start_db.elf bcm2709-rpi-cm2.dtb bcm2711-rpi-cm4.dtb fixup4x.dat kernel.img start.elf bcm2710-rpi-2-b.dtb bcm2711-rpi-cm4s.dtb fixup_cd.dat LICENCE.broadcom start_x.el
ls /mnt/rpi_system
bin boot dev etc home lib lost+found media mnt opt proc root run sbin srv sys tmp usr var
Copy the contents of the mounted directories to our guest directory.
cp -ax /mnt/rpi_system/* /home/qemu/qemu-user/raspios-armhf-lite
cp -ax /mnt/rpi_boot/* /home/qemu/qemu-user/raspios-armhf-lite/boot
Unmount the directories and free the loop devices. Let's do the same operation with the second image of the system.
sudo umount /mnt/rpi_boot
sudo umount /mnt/rpi_system
sudo losetup -d /dev/loop0 /dev/loop1
sudo losetup -f --show -o $((8192*512)) 2023-02-21-raspios-bullseye-arm64-lite.img
sudo losetup -f --show -o $((532480*512)) 2023-02-21-raspios-bullseye-arm64-lite.img
sudo mount /dev/loop0 /mnt/rpi_boot
sudo mount /dev/loop1 /mnt/rpi_system
cp -ax /mnt/rpi_system/* /home/qemu/qemu-user/raspios-armhf-lite
cp -ax /mnt/rpi_boot/* /home/qemu/qemu-user/raspios-armhf-lite/boot
sudo umount /mnt/rpi_boot
sudo umount /mnt/rpi_system
sudo losetup -d /dev/loop0 /dev/loop1
Let's copy the qemu to the guest systems.
cp -vfr /usr/bin/qemu* /home/qemu/qemu-user/raspios-armhf-lite/usr/bin
cp -vfr /usr/bin/qemu* /home/qemu/qemu-user/raspios-arm64-lite/usr/bin
And finally, we will create scripts in the guest systems that should be run after the chroot.
sudo nano -w /home/qemu/qemu-user/raspios-armhf-lite/rpi3.sh
#!/bin/bash
cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi
cd /
sudo nano -w /home/qemu/qemu-user/raspios-armhf-lite/rpi4.sh
#!/bin/bash
cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi
cd /
sudo nano -w /home/qemu/qemu-user/raspios-arm64-lite/rpi3.sh
#!/bin/bash
cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi
cd /
sudo nano -w /home/qemu/qemu-user/raspios-arm64-lite/rpi4.sh
#!/bin/bash
cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi
cd /
Now our guest systems are fully prepared for chroot. Now it's time to check what we've got.
cd /home/qemu/qemu-user/
lscpu
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. Model name: AMD Ryzen 5 5600G with Radeon Graphics BIOS Model name: AMD Ryzen 5 5600G with Radeon Graphics CPU family: 25 Model: 80 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 0 Frequency boost: enabled CPU max MHz: 4464.0000 CPU min MHz: 400.0000 BogoMIPS: 7803.38 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr _core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold av ic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm Virtualization features: Virtualization: AMD-V Caches (sum of all): L1d: 192 KiB (6 instances) L1i: 192 KiB (6 instances) L2: 3 MiB (6 instances) L3: 16 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Srbds: Not affected Tsx async abort: Not affected
gcc -v -E -x c /dev/null -o /dev/null -march=native 2>&1 | grep /cc1
/usr/libexec/gcc/x86_64-pc-linux-gnu/11.3.0/cc1 -E -quiet -v /dev/null -o /dev/null -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi -mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mpku -mno-prefetchwt1 -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=znver3 -dumpbase null
Run script "run_arm64_rpi4.sh" for example
sh run_arm64_rpi4.sh
In the guest system, run the script "rpi4"
sh rpi4.sh
Now, being in the guest system, we give the same commands.
Belka /home/qemu/qemu-user # ls arm-runner-action comands download_image.sh qemu-wrapper.c raspios-arm64-lite raspios-armhf-lite run_arm64_rpi3.sh run_arm64_rpi4.sh run_armhf_rpi3.sh run_armhf_rpi4.sh umount Belka /home/qemu/qemu-user # sh run_arm64_rpi4.sh root@Belka:/# ls bin boot dev etc home lib lost+found media mnt opt proc root rpi3.sh rpi4.sh run sbin srv sys tmp usr var root@Belka:/# sh rpi4.sh root@Belka:/# lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 NUMA node(s): 1 Vendor ID: ARM Model: 3 Model name: Cortex-A72 Stepping: r0p3 Frequency boost: enabled CPU max MHz: 4464.0000 CPU min MHz: 400.0000 BogoMIPS: 108.00 L1d cache: 192 KiB L1i cache: 192 KiB L2 cache: 3 MiB L3 cache: 16 MiB NUMA node0 CPU(s): 0-11 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fp asimd evtstrm crc32 cpuid root@Belka:/# gcc -v -E -x c /dev/null -o /dev/null -march=native 2>&1 | grep /cc1 /usr/lib/gcc/aarch64-linux-gnu/10/cc1 -E -quiet -v -imultiarch aarch64-linux-gnu /dev/null -o /dev/null -mlittle-endian -mabi=lp64 -march=armv8-a+crc -fasynchronous-unwind-tables root@Belka:/# su - pi pi@Belka:~ $
Further steps you can see in the building and installation instructions for Raspberry PI OS.
How to build and setup on Raspberry PI
How to build and setup for Raspberry PI QEMU native
Unmount the various file systems when they are no longer in use:
root #umount sys proc proc/cpuinfo dev/pts dev