Kernel Hacking - nicktehrany/notes Wiki

These are just some notes on kernel configs and qemu setup to get things running and how to do debugging.

kernel config

I have a kernel config file here that I typically use and adapt to support additional modules, etc. needed. Note It has KASAN already enabled. Just copy this into the kernel directory and name it .config, then compile the bzImage with make -j6 bzImage (or however many threads) and make sure config contains =y for all that is needed.

You can also find your current system's config file in the /usr/src/linux-headers-$(uname -r)/.config or simply run make oldconfig. However, if you're using a different config file make sure to have CONFIG_DEBUG_INFO=y to compile the kernel with debugging information.

qemu

My qemu command is typically in a qemu_start script since the command is quite long and can get messy. The script contains

#!/bin/bash

set -e

sudo $QEMU_HOME/build/qemu-system-x86_64 -name ZNS -m 4G -smp 4 -enable-kvm -nographic \
    -hda $HOME/src/storage/images/ubuntu-20.04.qcow2 \
    -kernel $HOME/src/storage/ZNS/linux/arch/x86/boot/bzImage \
    -append "root=/dev/sdb2 rootfstype=ext4 console=ttyS0 nokaslr" \
    -net nic,model=virtio -net user -device virtio-tablet-pci,id=tablet0,serial=virtio-tablet \
    -S -s

The -hda uses an image where ubuntu server 20.04 (available here) is already installed and setup. Note when installing it the create logical volume when making the partitions has to be disabled, as it's enabled by default and just makes life more difficult for running with qemu.

The -append contains an additional nokaslr to disable kernel randomization such that we can set clean breakpoints with gdb. For gdb we additionally have -S to have qemu wait for gdb to start running and -s for enabling to attach gdb to qemu running on localhost:1234.

Remember, the -kernel will load the kernel bzImage without any modules. It just loads the bare bones kernel, therefore make sure that whatever you are debugging is included in it.

gdb debugging

Debugging can then be done as usual, setting breakpoints, etc. once it's attached to qemu remote. This is done with:

# Create symbol file from the compiled kernel
objcopy --only-keep-debug vmlinux kernel.sym

# Run gdb
gdb
(gdb) file ./kernel.sym
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) c
(gdb) s

Building of just a single in-tree module

At some point you might want to load the module into a running kernel. The kernel has to be the same version, otherwise modprobe will complain of either wrong version magic (which can by bypassed sometimes and the module can still be run by using --force with modprobe) or an invalid Exec format which modprobe will not load.

# cd into the Kernel build!
cd linux

# Use current .config and add required modules
make oldconfig
make prepare
make modules_prepare

# Compile the specific module dir
make modules M=/drivers/md
# OR compile with your system's build to know if you're missing header files or so
make -C /lib/modules/`uname -r`/build M=drivers/md

# Copy the specific module to the system modules
cp drivers/md/custom_module.ko /lib/modules/`uname -r`/kernel/drivers/md/
depmod

# Check if module is there
lsmod

# Load it
modprobe custom_module

There are more ways of building external modules, see documentation here