A few scripts are appended at the end of this page for reference.

Create a rootfs

• Create a disk image (qemu-img: arch.qcow2)
• (Optional, recommended) create a COW image (qemu-img: cow.qcow2 over arch.qcow2).
• Access the disk image as a partition (qemu-nbd: ?.qcow2 -> /dev/nbd0)
• Format filesystem (mkfs.xfs /dev/nbd0) Don't need a partition table.
• Mount the filesystem (mount /dev/nbd0 /mnt)
• Install Archlinux as usual, add all necessary packages.

We don't need a bootloader. A regular linux kernel package is recommended. The rootfs can boot using the host linux kernel (/boot/vmlinuz-linux and initramfs-linux-fallback.img).

Obtain a minimal config file

• Boot the VM, setup network (a simple DHCP suffices) so you can easily transfer files or install new packages. The following steps should be done inside the VM.

• Download a recent linux src package. Git-clone is too slow so an xz package at https://mirrors.edge.kernel.org/pub/linux/kernel/ is the preferred. This kernel version don't have to be the one you're about to use or debug. We only use it to generate a good config file for use in this VM.

• Unpack and enter the kernel code directory. Now, download the Archlinux linux config from https://github.com/archlinux/svntogit-packages/tree/packages/linux/trunk, save the config file as ".config".

• make olddefconfig This refreshes the config file and add missing entries if any.

• make localyesconfig This removes the currently unused modules to save time and space, and set to build all loaded modules into the kernel, so we won't need to load modules with the debugging kernel.

• Transfer the .config file to the host for future use. This config file should be good even if you change kernel versions in the future.

Compile the Linux kernel for debugging

• On the host, copy the .config file to the kernel source code directory that you're about to build.

• make olddefconfig This can be omitted. But it's recommended to use it when you switch to different kernel versions.

• make all This will create vmlinux and bzImage. We will need both of them for booting the VM and debugging.

• During the build process, you may find a few modules are still created (e.g., XEN). It's ok to ignore them. You can also clean them by manually disabling those modules with make menuconfig

We will use the bzImage to boot the VM, and use the vmlinux for debugging with GDB. This kernel do not depend modules so we don't need a initrd.

boot qemu and debug with GDB

Make a few changes to the qemu commands:

• -s -S for waiting GDB at localhost:1234
• -kernel PATH_TO_bzImage
• -append nokaslr (and other existing arguments)

Now, when the qemu starts, you can attach to it from another shell (better use a .gdbinit)

• gdb> target remote localhost:1234
• sym PATH_TO_vmlinux
• b vfs_read
• c
• ...

scripts

mount the rootfs for maintenance

sudo modprobe nbd
sudo qemu-nbd -c /dev/nbd0 cow.qcow2
sudo mount /dev/nbd0 /mnt
sudo arch-chroot /mnt
sudo sync

unmount the rootfs

sudo sync
sudo umount /mnt
sudo qemu-nbd -d /dev/nbd0

commit the COW image to the original (after successful maintenance)

qemu-img commit cow.qcow2

discard changes in cow.qcow2 (since last commit)

rm cow.qcow2
qemu-img create -f qcow2 -b arch.qcow2 -F qcow2  cow.qcow2

start a regular VM

Note: the linux package in the vm must be up-to-date with that on the host. If not, mount the rootfs and upgrade manually.

nohup \
qemu-system-x86_64 \
  -cpu host -accel kvm -smp 2 -m 8G \
  -drive file=cow.qcow2,cache=none,if=virtio \
  -nic user,hostfwd=tcp::9122-:22 \
  -serial  telnet:localhost:9111,server,nowait \
  -monitor telnet:localhost:9112,server,nowait \
  -kernel /boot/vmlinuz-linux -initrd /boot/initramfs-linux-fallback.img \
  -append 'root=/dev/vda console=ttyS0 quiet loglevel=3 audit=0' \
  -nographic &>/dev/null &
#

• To connect to the console: telnet localhost 9111
• To connect to the qemu monitor: telnet localhost 9112
• To login over ssh if network is already setup: ssh localhost -p 9122

start a VM for debugging

nohup \
qemu-system-x86_64 \
  -s -S \
  -cpu host -accel kvm -smp 1 -m 1G \
  -drive file=cow.qcow2,cache=none,if=virtio \
  -nic user,hostfwd=tcp::9122-:22 \
  -serial  telnet:localhost:9111,server,nowait \
  -monitor telnet:localhost:9112,server,nowait \
  -kernel ${PATH_TO_bzImage} \
  -append 'root=/dev/vda console=ttyS0 quiet nokaslr loglevel=3 audit=0' \
  -nographic &>/dev/null &
#

PM emulation

Enable PMEM and DAX (And also select file systems that you want to debug)

	Device Drivers ---> 
		{*} NVDIMM (Non-Volatile Memory Device) Support --->
			<*>   PMEM: Persistent memory block device support
			<*>   BLK: Block data window (aperture) device support
	Processor type and features --->
		[*] Support non-standard NVDIMMs and ADR protected memory
	File systems --->
		[*] Direct Access (DAX) support

Build the kernel and boot with qemu: -m 20G ... -append 'memmap=4G!16G'. the memmap can be specified multiple times. The example marks the memory from 16G to 16+4G as persistent.

References

Enable PM support and emulate pmem https://pmem.io/2016/02/22/pm-emulation.html