Building a Linux kernel for the Xilinx PYNQ-Z1 Arm board

2018-10-20

The original Linux kernel on my strikingly coloured PYNQ-Z1 board has a bug which makes the networking crash every now and then. The error message in the system log is as follows:

  macb e000b000.ethernet eth0: DMA bus error: HRESP not OK

This is fixed in newer kernels (see this patch for details) so I decided to build a new Linux kernel for my board.

Much of this tutorial deals with workarounds required to cross-compile a Linux kernel on MacOS, but I have provided the instructions for cross-compiling on a Linux system as well, that is much simpler.

Assumptions

This tutorial assumes that you are familiar with working in a terminal with e.g. bash, with building code with make and the related tools, and know how to use tools like git, ssh, wget or curl etc. If you are on MacOS you'll need Macports or Homebrew.

Preparations

Get the cross-compilation toolchain

To compile source code into Arm binaries on a non-Arm architecture, we need a cross-compilation toolchain.

• On Linux:

    $ git clone https://github.com/raspberrypi/tools ~/rpi-tools
  

• On MacOS I used Jared Wolff's cross compilation toolchain

    $ wget https://s3.amazonaws.com/jaredwolff/xtools-2016-09-01.dmg
  

  After mounting the tools are available on /Volumes/xtools.

Create a working directory

    $ mkdir PYNQ-Z1
    $ cd PYNQ-Z1/

Configure the cross-compilation toolchain

• On MacOS:

    $ export CROSS_COMPILE=armv8-rpi3-linux-gnueabihf-
    $ export ARCH=arm
    $ export PATH=/Volumes/xtools/armv8-rpi3-linux-gnueabihf/bin/:$PATH
  

• On Linux it would be (I did not verify this):

    $ export CROSS_COMPILE=arm-linux-gnueabihf-
    $ export ARCH=arm
    $ export PATH=~/rpi-tools/arm-bcm2708/arm-linux-gnueabihf/bin/:$PATH
  

Get the Xilinx Linux kernel

    $ git clone https://github.com/Xilinx/linux-xlnx.git

I selected the latest series 4 kernel:

    $ git checkout -b xlnx_rebase_v4.14
    $ cd linux-xlnx/

Install missing stuff on MacOS

Because MacOS is not GNU/Linux, some header files are missing and some tools are different between Darwin and Linux. I use MacPorts which installs in /opt/local, for Homebrew adjust accordingly.

GNU/Linux tools

I installed the GNU versions of the tools:

    $ sudo port install coreutils
    $ sudo port install gsed
    $ sudo port install findutils

Then I symlinked /opt/local/bin/gxargs to /opt/local/bin/xargs and the same for gawk to awk and gstat to stat, and put /opt/local/bin first in the $PATH.

Missing header files

I copied a number of files from my PYNQ Linux distro's /usr/include directory into linux-xlnx/usr/include. I used scp as I had a running system available.

    $ mkdir usr/include
    $ cd usr/include

    $ scp pynq-1:/usr/include/elf.h .
    $ scp pynq-1:/usr/include/features.h .
    $ scp -r pynq-1:/usr/include/arm-linux-gnueabihf .

The latter has rather more than needed, but I was lazy:

usr/include/
└── arm-linux-gnueabihf
    ├── asm
    ├── bits
    ├── c++
    │   ├── 5
    │   │   ├── bits
    │   │   └── ext
    │   └── 5.2.1
    │       ├── bits
    │       └── ext
    ├── freetype2
    │   └── config
    ├── gnu
    ├── layout
    ├── libavcodec
    ├── libavformat
    ├── libavutil
    ├── libswresample
    ├── libswscale
    ├── openssl
    ├── python2.7
    ├── python3.4m
    ├── sys_LINUX
    └── unicode

Note that sys should be renamed or removed as it clashes with the MacOS sys.

Soft floating-point stub

The PYNQ-Z1 board has an Arm Cortex-A9 processor which has a hard floating point unit, so the distro does not have the file gnu/stubs-soft.h (it assumes that code is compiled with -mfloat-abi=hard). For some reason, the Linux kernel does not use hard floating point instructions, likely because it is more portable and there are very few floating point operations in the kernel code anyway.

I copied the gnu/stubs-soft.h from Matt Filetto's Linaro repo:

    $ wget https://github.com/Albinoman887/linaro/blob/master/arm-cortex_a8-linux-gnueabi/sysroot/usr/include/gnu/stubs-soft.h

Configuring the Linux kernel

Building the Linux kernel has two stages. In the first stage, the build is configured. Then the actual kernel image is built using the generated configuration. This step does not use the cross compiler so a simple make is enough. As I wanted the default configuration for the Xilinx Zynq board, I used:

    $ make xilinx_zynq_defconfig

Other configurations for Arm boards are available in arch/arm/configs. If you want to explore the configuration or customize it in detail, you can do:

    $ make menuconfig

Building the kernel zImage

If what you need is the zImage, then your job is done. Just build the kernel as follows:

    $ PATH=/opt/local/bin:$PATH \
     CPATH=~/PYNQ-Z1/linux-xlnx/usr/include/:\
     ~/PYNQ-Z1/linux-xlnx/usr/include/arm-linux-gnueabihf/ \
     make -j8 zImage

The $PATH contains the path to the cross-compilation toolchain; the $CPATH contains the additional include paths where the compiler should look for header files. The option -j8 tells make to build in parallel for 8 hardware threads. The zImage target builds the compressed kernel image.

Building the kernel uImage

However, my PYNQ-Z1 uses the U-Boot booloader which needs a uImage kernel image, which in turn needs the tool mkimage to create it, so I had to build U-Boot first.

Get the Device Tree Compiler dtc

Building U-Boot requires dtc. Apparently there is a Homebrew package but there is no MacPorts package. I (finally) found the source at https://git.kernel.org/pub/scm/utils/dtc/dtc.git and built it:

    $ wget https://git.kernel.org/pub/scm/utils/dtc/dtc.git/snapshot/dtc-1.4.7.tar.gz
    $ tar -zxvf dtc-1.4.7.tar.gz
    $ cd dtc-1.4.7
    $ make

Now copy the dtc somewhere in your path, e.g. I put it in $HOME/bin.

Get the Xilinx U-Boot sources

    $ git clone https://github.com/Xilinx/u-boot-xlnx.git
    $ cd u-boot-xlnx
    $ git branch
    $ git checkout xilinx-v2016.4
    $ make zynq_zed_config        

For the actual build we need some openssl header as well, from /opt/local/include, so:

    $ PATH=/opt/local/bin:/Volumes/xtools/armv8-rpi3-linux-gnueabihf/bin/:$PATH CPATH=/opt/local/include/ make

And then the newly built U-Boot tools need to be added to the $PATH, so finally:

    $ cd ../linux-xlnx 
    $ PATH=~/PYNQ-Z1/u-boot-xlnx/tools/:/opt/local/bin:PATH CPATH=~/PYNQ-Z1/linux-xlnx/usr/include/:~/PYNQ-Z1/linux-xlnx/usr/include/arm-linux-gnueabihf make -j8 UIMAGE_LOADADDR=0x8000 uImage

I changed the U-Boot config file uEnv.txt to point at the new kernel, which I named uImage-4.14.

  kernel=uImage-4.14

Then I mounted the PYNQ disk image, and copied the newly compiled kernel:

$ cp arch/arm/boot/uImage /Volumes/NO\ NAME/uImage-4.14

And that's it, I put the micro-SD card back into the PYNQ board, switched it on and watched it boot. Then finally:

$ ssh pynq-1

And it worked, we're running kernel v4.14: