6. Realtime Kernel, CPU Isolation, and Interrupts - josh-blake/pixie GitHub Wiki

Once your time service is up and running, you'll note that your timing accuracy is good; not great, but good. Chrony is wholly dependent on the kernel and how it behaves to control timing latency, interrupt handling, and other low-level processing. This section focuses on optimising the system to be as responsive as possible. By no means is this comprehensive, and I am certain there are more performance tweaks to be had.

Realtime Kernel

There are a couple of options for realtime kernels here. One is prebuilt and the easiest to start tinkering with.

The Linux kernel relies on a unit of time known as a 'jiffy'. This unit of time is how long a process is allowed to retain control before it is asked to relinquish control of the CPU to another process. This, coupled to the priority of a process, (in simplistic terms) dictates how the CPU(s) prioritise what processes to run.

Lower level attributes can be set on the kernel command line at boot, and others specified in the kernel build process. These are elaborated on throughout.

The default realtime kernel builds by the Raspberry Pi development team compile their kernels with a jiffy period of 250Hz. The Linux kernel is able to support up to 1000Hz. Higher frequency jiffies are not necessarily of benefit here, in particular with a realtime kernel. Take for example an interrupt routine for the serial port that gets compelled to relinquish control part way through receiving a NMEA sentence; the kernel will not return to this process until it has given all other processes with the same or higher priority a look in. A standard NMEA sentence from the GPS unit can be up to 81 characters in length; at 8 bits per character, and a baud rate of 460800bps, a single message from the GPS unit takes just over 1.4ms to transmit. A jiffy rate of 250Hz offers an interrupt period of 4ms and thus will never interrupt the serial interrupt handler, whereas a jiffy period of 1000Hz (1ms) is guaranteed to interrupt the serial port interrupt handler. Fortunately there are a few tricks that can also be employed here so that a high jiffy rate can be used without disrupting critical interrupt handlers.

A quick sudo dmesg | grep sched_clock should reveal the following: sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns. This implies that the best precision we can achieve is 18ns. This equates to roughly the speed of signal conduction in the wire of your GPS antenna if it were 5m long. In practicality, a system that can maintain precision at this level is well and truly suitable for my needs. If it does not suit yours, I would not invest in this setup.

Obtain a Prebuilt Realtime Kernel

This will bump you to the current kernel tree: 6.17. Please visit kernel.org to confirm what current branch you are on.

sudo WANT_64BIT=0 WANT_64BIT_RT=1 WANT_PI5=1 WANT_PI4=0 rpi-update rpi-6.17.y

Make sure you select yes to these. This will install the Raspberry Pi realtime kernel released by the devs. Note that it will not boot it by default! You will need to adjust your config.txt file to specify the kernel you want to run. Note that the bootloader will load the first valid kernel (in alphabetical order) by default (this is the kernel_2712.img on contemporary systems after running rpi-update).

sudo nano /boot/firmware/config.txt

Add the following line at the top of the file, save and exit:

kernel=/kernel8_rt.img

It is good practice to confirm that you actually have a kernel by that name in the /boot/firmware folder before rebooting otherwise you will be in for a headache.

When the system comes back up, check that you are running a realtime kernel with uname -r. There should be a RT somewhere in the kernel name.

Build a Realtime Kernel

If you like to tinker, you can build your own kernel. It takes about 40 minutes on a contemporary RPi 5. I have tinkered enough with the current configuration options and have them shared in this repo. I will update the .config file with relevant incremental builds here. A quick summary of my build process is inspired by this git project: by/RT-Kernel.

Salient points:

  • The IPv4 and IPv6 stacks are compiled into the kernel by default, and not as modules. While this increases kernel size, the address space is slightly more accessible than in a module.
  • Additional NIC devices are not compiled; only the macb and gem drivers are compiled (and a some of the intel phys that support timestamping). You will need to change this if you plan on using my config file with other PHY devices!
  • Fully preemptable realtime kernel
  • Dynamic preemption enabled (you can specify the type of preemption on the kernel command line)
  • Lazy preemption by default (give the process 1 extra jiffy to wrap things up)
  • Direct DMA passthrough rather than lazy randomisation (this does in theory reduce overhead runtime at the expense of hardware security)
  • Default 'performance' based rather than 'power' based CPU Frequency governor
  • GPS, PPS, and PTP device drivers are built
  • Hardware timestamping support

Make sure you have a relevant kernel build environment:

sudo apt install git bc bison flex libssl-dev libncurses5-dev make

Download the kernel source:

git clone --depth 1 --branch rpi-6.18.y https://github.com/raspberrypi/linux

Adjust the configuration (if required):

make menuconfig

Otherwise make sure a relevant .config file is present in the folder. If you need a good starting point, try make bcm2712_defconfig. Alternatively, you can download the .config file that I use.

Specify KBUILD parameters and a kernel filename to facilitate the rest of these build instructions:

export KBUILD_BUILD_TIMESTAMP="2025-12-15" KBUILD_BUILD_VERSION="1" KERNEL_NAME="6.18.1-v8-16k-rt"

Now build your kernel (make some popcorn, watch some tv etc.; just don't watch the kettle boil):

make prepare

make CFLAGS='-O3 -march=native -mtune=native' -j6 Image.gz modules dtbs

Install the modules:

sudo make -j6 modules_install

Make a directory for the kernel and overlays:

sudo mkdir -p /boot/firmware/$KERNEL_NAME/overlays

Copy the kernel and modules into the boot directory:

sudo cp arch/arm64/boot/Image.gz /boot/firmware/kernel-$KERNEL_NAME.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/$KERNEL_NAME/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/$KERNEL_NAME/overlays
sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/$KERNEL_NAME/overlays/

If you have not used this method before, update your config.txt to add the following lines to your config.txt, save and exit:

sudo nano /boot/firmware/config.txt

os_prefix=
overlay_prefix=overlays/
kernel=

Run this snippet to update your config.txt to specify the new kernel filename:

sudo sed -i "s@^os_prefix=.*@os_prefix=$KERNEL_NAME/@g" /boot/firmware/config.txt
sudo sed -i "s@^kernel=.*@kernel=/kernel-$KERNEL_NAME.img@g" /boot/firmware/config.txt

Restart your RPi. Once you are back up, check that the kernel hasn't borked and that devices are loading as expected:

sudo dmesg

Finally, ensure that your RPi is running contemporary firmware suitable for the kernel version:

sudo WANT_64BIT=0 WANT_64BIT_RT=0 WANT_PI5=0 WANT_PI4=0 SKIP_KERNEL=1 PRUNE_MODULES=1 SKIP_VCLIBS=0 rpi-update rpi-6.18.y

CPU Isolation & Kernel Preemption

Modify cmdline.txt to Reserve Specific CPUs

Ensure the following parameters are in your cmdline.txt file. Note that some of these parameters may not work until you are using a RT kernel.

Ensure the following are present, save and exit: sudo nano /boot/firmware/cmdline.txt

preempt=full skew_tick=1 isolcpus=2,3 irqaffinity=0,1 nohz=off cpuidle.off=1

The above tells the kernel to: use realtime preemption, skews the tick interrupts so that only 1 core is ever being preempted at a time, isolates CPUs 2 and 3 from general use processes, and restricts general IRQ interrupts to CPUs 0 and 1 by default. Specific processes and interrupts will be earmarked to each core once the system is up and running. Please review the previous systemd unit files for Chrony and GPSD and ensure that the CPUAffinity directive is specified. Chrony for example runs with an extremely high context switching rate (~2000 switches per second). GPSD runs at 1-2 switches per second (on par with the NMEA message rates). The premise is to isolate timing critical processes from eachother; ie Chrony timekeeping should not interrupt GPSD and hardware timestamping. Moreover, logging into the device, updating the system, compiling a kernel etc should never interfere with the timing precision either.

Interrupt Routine Isolation

There is no way to specify what CPU each interrupt handler runs on unlike a process. Thus, interrupts can be prone to scheduling related jitter within the kernel space. This code isolates the serial device and PPS interrupt handler to a specific CPU. When other processes are excluded from the CPU, latency is reduced as interrupt handling can occur independently. This becomes very apparent when Chrony is operating as a NTP Server and may otherwise have overhead network latency. The premise here is to blacklist specific CPUs from use, and then only add specific processes once everything is online. The irqafffinity directive described in cmdline.txt in the previous section accounts for this. Here, hardware level interrupts can only run on CPU0 and 1. The following section moves the IRQ Handler to an isolated CPU once it has spun up.

Create an IRQ Rescheduler Script

This script looks for pps and uart interrupt handlers, gets their task number, and then executes a taskset command to restrict these interrupts to CPU 3 (Note CPU numbering is 0-3 rather than 1-4 unless you build your kernel otherwise). It also changes the process scheduler to round robin style; this allows the process to take advantage of RT kernel scheduling ticks.

Add the following, save and exit: sudo nano /usr/bin/pps-irq

#!/bin/bash
PPS_IRQ=$(ps -e | grep pps@12 | awk '{print $1}')
taskset -p -c 3 $PPS_IRQ
chrt -rr -p 50 $PPS_IRQ

SER_IRQ=$(ps -e | grep uart-pl011 | awk '{print $1}')
taskset -p -c 3 $SER_IRQ
chrt -rr -p 50 $SER_IRQ

Make the script executable: sudo chmod +x /usr/bin/pps-irq

Create Systemd Unit File

Add the following, save and exit: sudo nano /etc/systemd/system/setpps.service

[Unit]
Description=Sets IRQ Scheduler Priority and CPU Isolation
After=gpsd.service
After=chronyd.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/pps-irq

[Install]
WantedBy=multi-user.target

Then do the following:

sudo systemctl daemon-reload

sudo systemctl enable setpps.service

sudo systemctl start setpps

Confirm that your interrupt handlers are now running on the correct CPU with htop (don't forget to edit your htop setup: make sure that 'Hide Kernel Threads' is deselected and that you have added the CPU column).