Kernel Hacking - nicktehrany/notes GitHub Wiki
These are just some notes on kernel configs and qemu setup to get things running and how to do debugging. See also kernel build guide and stackoverflow debug kernel build.
kernel config
I have a kernel config file here that I typically use and adapt to support additional modules, etc. needed. I have another config with the addition of BPF. Note It has KASAN already enabled. Just copy this into the kernel directory and name it .config
, then compile the bzImage with make -j6 bzImage
(or however many threads) and make sure config contains =y
for all that is needed. If an error for invalid certificate/keyring shows up,
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS
disable the keys, and during the make
hit enter on all configurations, such that it generates a private key to use.
You can also find your current system's config file in the /usr/src/linux-headers-$(uname -r)/.config
or simply run make oldconfig
. However, if you're using a different config file make sure to have CONFIG_DEBUG_INFO=y
to compile the kernel with debugging information.
qemu
My qemu command is typically in a qemu_start
script since the command is quite long and can get messy. The script contains
#!/bin/bash
set -e
sudo $QEMU_HOME/build/qemu-system-x86_64 -name ZNS -m 4G -smp 4 -enable-kvm -nographic \
-hda $HOME/src/storage/images/ubuntu-20.04.qcow2 \
-kernel $HOME/src/storage/ZNS/linux/arch/x86/boot/bzImage \
-append "root=/dev/sdb2 rootfstype=ext4 console=ttyS0 nokaslr" \
-net nic,model=virtio -net user,hostfwd=tcp::8888-:22,hostfwd=tcp::3333-:3000 -device virtio-tablet-pci,id=tablet0,serial=virtio-tablet \
-S -s
The -hda
uses an image where ubuntu server 20.04 (available here) is already installed and setup. Note when installing it the create logical volume when making the partitions has to be disabled, as it's enabled by default and just makes life more difficult for running with qemu.
The -append
contains an additional nokaslr
to disable kernel randomization such that we can set clean breakpoints with gdb. For gdb we additionally have -S
to have qemu wait for gdb to start running and -s
for enabling to attach gdb to qemu running on localhost:1234
.
Remember, the -kernel
will load the kernel bzImage without any modules. It just loads the bare bones kernel, therefore make sure that whatever you are debugging is included in it.
Note that with this setup, the kernel must be compiled without modules, as we only include the bzImage in the qemu, and do not write the modules into the root file system. Hence when running the qemu, the modules do not exist. The kernel config I use has the necessary parts included in the kernel, not as modules, such that the kernel works. This is particularly for networking, which can be tricky to debug with modules.
I have a bigger script to provide me with emulated nvme devices (SSD and ZNS) with passthrough. The images backing the devices can easily be generated with this script (Note I manually created additional images not in that script, so if a command fails for lack of existing images, simply manually generate that image), and then I use this qemu script to start up the qemu with all the emulated devices.
gdb debugging
Debugging can then be done as usual, setting breakpoints, etc. once it's attached to qemu remote. This is done with:
# Create symbol file from the compiled kernel
objcopy --only-keep-debug vmlinux kernel.sym
# Run gdb
gdb
(gdb) file ./kernel.sym
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) c
(gdb) s
See also debugging guide.
Building of just a single in-tree module
At some point you might want to load the module into a running kernel. The kernel has to be the same version, otherwise modprobe will complain of either wrong version magic (which can by bypassed sometimes and the module can still be run by using --force
with modprobe
) or an invalid Exec format which modprobe will not load.
# cd into the Kernel build!
cd linux
# Use current .config and add required modules
make oldconfig
make prepare
make modules_prepare
# Compile the specific module dir
make modules M=/drivers/md
# OR compile with your system's build to know if you're missing header files or so
make -C /lib/modules/`uname -r`/build M=drivers/md
# Copy the specific module to the system modules
cp drivers/md/custom_module.ko /lib/modules/`uname -r`/kernel/drivers/md/
depmod
# Check if module is there
lsmod
# Load it
modprobe custom_module
There are more ways of building external modules, see documentation here
Updating Kernel from manual build
If we want to install the deb-kg from the Kernel build, which is how I install Kernels on my local server, we can build the packages and install them. To just update the Kernel, which I use if I just need to bump up the Kernel, we can just pull the packages from the remote.
Manual Build
Manual build looks like, make sure the Kernel is configured with the config
file:
make -j4 bindeb-pkg LOCALVERSION=-stosys
# Will result in a single .deb package in the root dir of the Kernel
# If only one .deb package in there, this is fine, otherwise specify name
sudo dpkg -i ../*.deb
sudo upgrade-grub2
sudo sync
sudo reboot
This includes all the headers, kernel image, etc., and generates a new initrd
, and auto configures bootloader if needed.
Building on host - copy to VM
We can also build on a host machine and copy the .deb
files to the VM (in order to save space on the VM image)
# Build the same way as before
# Copy to host on open port
scp -P 8888 *.deb user@localhost:~/
# Install the files as before
Downloading Built Packages
Downloading package builds from the remote:
mkdir pkgs && cd pkgs
# wget pkgs - adapt Kernel version
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-headers-5.12.0-051200_5.12.0-051200.202104252130_all.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-headers-5.12.0-051200-generic_5.12.0-051200.202104252130_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-image-unsigned-5.12.0-051200-generic_5.12.0-051200.202104252130_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.12/amd64/linux-modules-5.12.0-051200-generic_5.12.0-051200.202104252130_amd64.deb
# Install all deb packages
sudo dpkg -i *.deb