Prepare Debian

Prepare a standard Headless Raspberry Pi Debian install. If you have used your CM5 before, you may have already configured the boot order on the CM5 and can jump straight into a SSD install. If not, you may need to boot from SD and then migrate your system to an SSD. Again, I won't belabour much here as there are many guides now on how to get started.

Otherwise, use an SSD of your choice. Install it in a NVME Enclosure so that you can use the Raspberry Pi Imager. Install Debian 64 bit WITHOUT a GUI (ie headless). Ensure that you configure SSH in the additional configuration so that you can access the device on your network. I would STRONGLY recommend against using Wifi here as it will get disabled later on once the RPi Config file is updated. Once you have finished here, put your SSD in the CM5 Carrier Board, ensure that it is connected over ethernet, and boot up.

Once you are set up, log into your server and run sudo apt update and sudo apt upgrade to make sure you are running the most contemporary versions of everything. We will get to different kernel and firmware versions later on once we have confirmed that the hardware is working correctly.

You will need to remove some packages, and install some others. The main Trixie repo does not contain contemporary packages for chrony or linuxptp, so I have added both the backports and testing repos. Follow these steps to do the same for yourself:

Add backports and testing repos

Do not forget to run a sudo apt update once you have done the following. Do not forget to pin the testing repo (see below) so that you do not bork your system.

Backports

sudo nano /etc/apt/sources.list.d/debian-backports.sources

Add the following, save and exit:

Types: deb deb-src
URIs: http://deb.debian.org/debian
Suites: trixie-backports
Components: main
Enabled: yes
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Testing

sudo nano /etc/apt/sources.list.d/debian-testing.sources

Add the following, save and exit:

Types: deb deb-src
URIs: http://deb.debian.org/debian
Suites: forky
Components: main
Enabled: yes
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Pin testing repo

sudo nano /etc/apt/preferences.d/50testing

Add the following, save and exit:

Package: *
Pin: release o=Debian,a=testing,n=forky
Pin-Priority: 1

Remove Conflicting Services

Remove Fake HW Clock

sudo apt remove fake-hwclock
sudo update-rc.d -f fake-hwclock remove
sudo systemctl disable fake-hwclock

Then run sudo nano /lib/udev/hwclock-set

Be sure to comment out:

#if [ -e /run/systemd/system ] ; then
# exit 0
#fi
#/sbin/hwclock --rtc=$dev --systz

Save and exit.

Disable systemd-timesyncd

sudo systemctl disable --now systemd-timesyncd

Disable Serial TTY on UART0

sudo systemctl disable --now [email protected]

sudo systemctl disable --now hciuart

Disable Kernel Serial TTY on UART0

sudo nano /boot/firmware/cmdline.txt

Remove console=serial0,115200 and save.

Disable additional services

Note that removing alsa-utils and other packages which may not seem immediately relevant, are depends for other core utilities like raspi-config. It is easier to simply disable the offending services so that systemd does not load them on boot.

sudo systemctl disable --now alsa-restore.service
sudo systemctl disable --now alsa-state.service
sudo systemctl disable --now alsa-utils.service
sudo systemctl disable --now apt-daily-upgrade.timer
sudo systemctl disable --now apt-daily.timer
sudo systemctl mask apt-daily-upgrade.service
sudo systemctl mask apt-daily.service
sudo systemctl disable --now avahi-daemon.service
sudo systemctl disable --now bluetooth.service
sudo systemctl disable --now [email protected]
sudo systemctl disable --now triggerhappy.service
sudo systemctl disable --now rpi-display-backlight.service
sudo systemctl disable --now wpa_supplicant.service
sudo systemctl disable --now x11-common.service

Once you have done the above, reboot. When you are back online, have a look with htop or equivalent to see what processes are currently running. You should not see many, and those that are running should appear somewhat relevant for the normal functioning of a system.

Install Required Packages

sudo apt install gpsd gpsd-tools gpsd-clients pps-tools chrony linuxptp setserial i2c-tools ser2net ethtool

We will configure these tools in later sections.

Update config.txt

Once you have booted in, and are up to date, you can modify your config.txt file. You can copy/paste mine, or you can modify your existing one.

Run sudo nano /boot/firmware/config.txt and change your file to resemble the following. Note that this configuration disables the video core, SPI bus, SD card, Wifi and BT modules, as well as sets the base serial baud and DMA control.

# For more options and information see
# http://rptl.io/configtxt
# Some settings may impact device functionality. See link above for details
dtoverlay=disable-wifi-pi5
dtoverlay=disable-bt-pi5
# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c=on,i2c_baudrate=400000
dtparam=i2s=off
dtparam=spi=off

# Disable SD Card
dtparam=sd=off

# Enable audio (loads snd_bcm2835)
#dtparam=audio=on

# Additional overlays and parameters are documented
# /boot/firmware/overlays/README

# Automatically load overlays for detected cameras
#camera_auto_detect=1

# Automatically load overlays for detected DSI displays
#display_auto_detect=1

# Automatically load initramfs files, if found
auto_initramfs=1

# Enable DRM VC4 V3D driver
#dtoverlay=vc4-kms-v3d
#max_framebuffers=2

# Don't have the firmware create an initial video= setting in cmdline.txt.
# Use the kernel's default instead.
disable_fw_kms_setup=1

# Run in 64-bit mode
arm_64bit=1

# Run as fast as firmware / board allows, and with a constant clock speed
arm_boost=1
force_turbo=1

# Disable compensation for displays with overscan
#disable_overscan=1

#Disable Power and NIC Lights
dtparam=pwr_led_activelow=off
dtparam=pwr_led_trigger=default-on
dtparam=act_led_activelow=off
dtparam=act_led_trigger=none
dtparam=eth_led0=4
dtparam=eth_led1=4

[cm5]
dtoverlay=dwc2,dr_mode=host

[all]
#Enable performance options
usb_max_current_enable=1
dtparam=pciex1=on,pciex1_gen=3

#RTC Module
dtparam=rtc=off
dtoverlay=i2c-rtc,rv3028,addr=0x52,wakeup-source,backup-switchover-mode=3

#GPS Module
dtoverlay=pps-gpio,gpiopin=18,pull=down

#Change Serial Parameters
dtoverlay=uart0-pi5
dtparam=uart0=on,uart0_console=off,uart0_dma=on
#dtoverlay=miniuart-bt #Not necessary since BT is disabled as above
init_uart_baud=460800

#Disable NIC EEE
dtparam=eee=off

Once you have changed your config.txt file, save and reboot.

Reduce Ethernet Latency for PTP and NTP Services

Change Ethernet Packet Coalescence

The onboard NIC only supports time-based packet coalescence; that is to say that sent and received packets are aggregated by the NIC for a certain amount of time before creating a system interrupt. Changing the coalescence interval improves latency at the expense of higher system interrupt frequency. The lowest timing the NIC supports is 4us so we will use that. The default is 49us. You will need to modify the unit file if you want to play around with tx-usec and rx-usec values.

We will create a systemd unit file to run the required ethtool command on boot. The order that this unit file loads is important; the NIC must be online for this coalesce time to be relevant so we will wait until after the NetworkManager-wait-online.service has started. Note that you can use this unit file for other NICs. I assume that your NIC is named eth0 for the purposes of enabling the service.

sudo nano /etc/systemd/system/[email protected]

Add the following, save and exit:

[Unit]
Description=Minimize Ethernet Time Coalescence
After=NetworkManager-wait-online.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ethtool -C %i tx-usecs 4 rx-usecs 4
ExecStop=/usr/sbin/ethtool -C %i tx-usecs 49 rx-usecs 49

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload

sudo systemctl enable coalesce@eth0

sudo systemctl start coalesce@eth0

Reduce Ethernet Ring Buffer Size

This has a theoretical advantage here in compelling the kernel to poke and prod the NIC more frequently by limiting how many packets the kernel buffers in memory. Although I am not sure that it makes a practical difference. With a smaller ring buffer size, kernel interrupts should occur more frequently. The theoretical limit here is that interrupts do not happen quick enough and packets are dropped. You can monitor NIC stats using the following: sudo watch -n 1 ethtool -S eth0. If you start to see dropped packets, your ring buffer is too small. Instead, most server NICs will run with a ring buffer of 256 packets. This is sensible, and should in theory keep the packet data well cached in fast memory. Create another systemd unit file to make these changes permanent.

sudo nano /etc/systemd/system/[email protected]

Add the following, save and exit:

[Unit]
Description=Reduce Ethernet Ring Buffer Size
After=NetworkManager-wait-online.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ethtool -G %i rx 256 tx 256
ExecStop=/usr/sbin/ethtool -G %i rx 512 tx 512

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload

sudo systemctl enable ring@eth0

sudo systemctl start ring@eth0

Disable Energy-Efficient Ethernet (EEE)

The onboard NIC supports EEE networking by default; for most systems this is fine, and can reduce power consumption by allowing the NIC to sleep for periods of time. This however conflicts with the tenant of creating a responsive and accurate time server. The easiest way to disable EEE (at the hardware level) is to simply add the following dtparam in your config.txt file: dtparam=eee=off. Please see above for more information. A sudo ethtool --show-eee eth0 should return netlink error: Operation not supported once EEE has been disabled.

Reduce Serial Latency

Change Serial Port Priority

By default, serial devices are considered slow by a contemporary kernel. Whereby the serial port operates in thousands of bits per second, each core of the RPi 5 operates at billions of operations per second. Let's change the kernels perception of how it should prioritise the serial port.

Ensure that setserial has been installed as above.

Add the following, then save and exit: sudo nano /etc/serial.conf

/dev/ttyAMA0 low_latency close_delay 0 closing_wait none

Then restart setserial: sudo systemctl restart etc-setserial

This should now reduce the kernel latency. Restart your system, and confirm that this is the case: sudo setserial -g /dev/ttyAMA0