Kernel Hacking - nicktehrany/notes GitHub Wiki

These are just some notes on kernel configs and qemu setup to get things running and how to do debugging. See also kernel build guide and stackoverflow debug kernel build.

kernel config

I have a kernel config file here that I typically use and adapt to support additional modules, etc. needed. I have another config with the addition of BPF. Note It has KASAN already enabled. Just copy this into the kernel directory and name it .config, then compile the bzImage with make -j6 bzImage (or however many threads) and make sure config contains =y for all that is needed. If an error for invalid certificate/keyring shows up,

scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS

disable the keys, and during the make hit enter on all configurations, such that it generates a private key to use.

You can also find your current system's config file in the /usr/src/linux-headers-$(uname -r)/.config or simply run make oldconfig. However, if you're using a different config file make sure to have CONFIG_DEBUG_INFO=y to compile the kernel with debugging information.

qemu

My qemu command is typically in a qemu_start script since the command is quite long and can get messy. The script contains

#!/bin/bash

set -e

sudo $QEMU_HOME/build/qemu-system-x86_64 -name ZNS -m 4G -smp 4 -enable-kvm -nographic \
    -hda $HOME/src/storage/images/ubuntu-20.04.qcow2 \
    -kernel $HOME/src/storage/ZNS/linux/arch/x86/boot/bzImage \
    -append "root=/dev/sdb2 rootfstype=ext4 console=ttyS0 nokaslr" \
    -net nic,model=virtio -net user,hostfwd=tcp::8888-:22,hostfwd=tcp::3333-:3000 -device virtio-tablet-pci,id=tablet0,serial=virtio-tablet \
    -S -s

The -hda uses an image where ubuntu server 20.04 (available here) is already installed and setup. Note when installing it the create logical volume when making the partitions has to be disabled, as it's enabled by default and just makes life more difficult for running with qemu.

The -append contains an additional nokaslr to disable kernel randomization such that we can set clean breakpoints with gdb. For gdb we additionally have -S to have qemu wait for gdb to start running and -s for enabling to attach gdb to qemu running on localhost:1234.

Remember, the -kernel will load the kernel bzImage without any modules. It just loads the bare bones kernel, therefore make sure that whatever you are debugging is included in it.

Note that with this setup, the kernel must be compiled without modules, as we only include the bzImage in the qemu, and do not write the modules into the root file system. Hence when running the qemu, the modules do not exist. The kernel config I use has the necessary parts included in the kernel, not as modules, such that the kernel works. This is particularly for networking, which can be tricky to debug with modules.

I have a bigger script to provide me with emulated nvme devices (SSD and ZNS) with passthrough. The images backing the devices can easily be generated with this script (Note I manually created additional images not in that script, so if a command fails for lack of existing images, simply manually generate that image), and then I use this qemu script to start up the qemu with all the emulated devices.

gdb debugging

Debugging can then be done as usual, setting breakpoints, etc. once it's attached to qemu remote. This is done with:

# Create symbol file from the compiled kernel
objcopy --only-keep-debug vmlinux kernel.sym

# Run gdb
gdb
(gdb) file ./kernel.sym
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) c
(gdb) s

See also debugging guide.

Building of just a single in-tree module

At some point you might want to load the module into a running kernel. The kernel has to be the same version, otherwise modprobe will complain of either wrong version magic (which can by bypassed sometimes and the module can still be run by using --force with modprobe) or an invalid Exec format which modprobe will not load.

# cd into the Kernel build!
cd linux

# Use current .config and add required modules
make oldconfig
make prepare
make modules_prepare

# Compile the specific module dir
make modules M=/drivers/md
# OR compile with your system's build to know if you're missing header files or so
make -C /lib/modules/`uname -r`/build M=drivers/md

# Copy the specific module to the system modules
cp drivers/md/custom_module.ko /lib/modules/`uname -r`/kernel/drivers/md/
depmod

# Check if module is there
lsmod

# Load it
modprobe custom_module

There are more ways of building external modules, see documentation here

Updating Kernel from manual build

If we want to install the deb-kg from the Kernel build, which is how I install Kernels on my local server, we can build the packages and install them. To just update the Kernel, which I use if I just need to bump up the Kernel, we can just pull the packages from the remote.

Manual Build

Manual build looks like, make sure the Kernel is configured with the config file:

make -j4 bindeb-pkg LOCALVERSION=-stosys 

# Will result in a single .deb package in the root dir of the Kernel
# If only one .deb package in there, this is fine, otherwise specify name
sudo dpkg -i ../*.deb
sudo upgrade-grub2
sudo sync
sudo reboot

This includes all the headers, kernel image, etc., and generates a new initrd, and auto configures bootloader if needed.

Building on host - copy to VM

We can also build on a host machine and copy the .deb files to the VM (in order to save space on the VM image)

# Build the same way as before

# Copy to host on open port
scp -P 8888 *.deb user@localhost:~/

# Install the files as before

Downloading Built Packages

Downloading package builds from the remote:

mkdir pkgs && cd pkgs

# wget pkgs - adapt Kernel version
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-headers-5.12.0-051200_5.12.0-051200.202104252130_all.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-headers-5.12.0-051200-generic_5.12.0-051200.202104252130_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-image-unsigned-5.12.0-051200-generic_5.12.0-051200.202104252130_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-modules-5.12.0-051200-generic_5.12.0-051200.202104252130_amd64.deb

# Install all deb packages
sudo dpkg -i *.deb