Building a cross compile toolchain for later netbsd6 devices, creating and installing a full netbsd 7.2 distrib and compile pkgsrc packages

• Building a cross compile toolchain for ARMv6EL
• Running a full Netbsd 7.2 distro on airport alongside with the current stripped down Netbsd 6.0
• Compiled full Netbsd 7.2 binaries -- for those who want eager to hack their airports
• TODOs

This tutorial has been tested on last generation Airport Time Capsule. It runs on a dual core Cortex-A9 and suports Armv7 instruction set.

However, as Apple decided to run it with Netbsd 6.0 (which lacks support for multi-core ARM CPU and Armv7 instruction set) the toolchain must be configured to Armv6 (yes, your device runs on only one core).

Even though it is possible to build the netbsd tools on Linux -- there are a plenty of how-to's on the net. I used a Netbsd 8.1 64bit virtual machine running on virtual box.

On the other hand it is possible (and I recommend it) to use a newer version of Netbsd source code. I am using the 7.2 release -- its binaries run just fine alongside with Apple's Netbsd 6 kernel.

The script bellows download and statically cross-compiles Netbsd 7.2 to Armv6 Little Endian used on later Airport Time Capsule and Airport Express/Extreme devices.

# I found out as useful to have some additional environment variable to have a better control on the tools building.
export RELEASE=7.2
export ARCH=earmv6hf
BUILD_OPTS="-U -u -j 20 -a ${ARCH} -m evbarm"
BUILD_DIRS="-O ../evbarm/obj -D ../evbarm/dest -R ../evbarm/release -T ../evbarm/tools"

export LDSTATIC="-static"
export MAKEVERBOSE=2
export MKPIC=no
export MKPICINSTALL=no
export MKDTRACE=no
export MKCTF=no
export CPUFLAGS="-mno-unaligned-access"
export LDFLAGS="-z muldefs"

echo Check if pkg sets are present
mkdir -p netbsd/${RELEASE}/pkgs
for i in gnusrc sharesrc src syssrc xsrc; do
        if [ ! -f netbsd/${RELEASE}/pkgs/${i}.tgz ] ; then
                echo Downloading ${i}.tgz from http://archive.netbsd.org/pub/NetBSD-archive/NetBSD-${RELEASE}/source/sets
                (cd netbsd/${RELEASE}/pkgs && lftp -c "pget http://archive.netbsd.org/pub/NetBSD-archive/NetBSD-${RELEASE}/source/sets/${i}.tgz")
        fi
done

if [ ! -d netbsd/${RELEASE}/src ] ; then
        for i in gnusrc sharesrc src syssrc; do
                echo Extracting  ${i}.tgz
                if [ `uname` = "NetBSD" ] ; then
                        (cd netbsd/${RELEASE} && tar -xvzf pkgs/${i}.tgz -s /^usr/./s )
                else
                        (cd netbsd/${RELEASE} && tar xvzf pkgs/${i}.tgz --strip-components=1 )
                fi
        done
fi


# Some tweaks to fix the impossibility of having DTRACE routines and due to the static build
echo Disabling output check file list
sed -e 's/exit \${es}/exit 0/g' netbsd/${RELEASE}/src/distrib/sets/checkflist >  netbsd/${RELEASE}/src/distrib/sets/checkflist.tmp
mv netbsd/${RELEASE}/src/distrib/sets/checkflist.tmp netbsd/${RELEASE}/src/distrib/sets/checkflist

echo Disabling compat libs
sed -e "s/usr.sbin share sys etc tests compat/usr.sbin share sys etc tests/" netbsd/${RELEASE}/src/Makefile | sed -e "s/BUILDTARGETS.*do-compat-lib//" > netbsd/${RELEASE}/src/Makefile.tmp
mv netbsd/${RELEASE}/src/Makefile.tmp  netbsd/${RELEASE}/src/Makefile

for i in tools build distribution sets ; do
        if [ ! -f netbsd/${RELEASE}/evbarm/.${i}_ok ] ; then
                (cd netbsd/$RELEASE/src && ./build.sh $BUILD_OPTS $BUILD_DIRS ${i}) && touch netbsd/${RELEASE}/evbarm/.${i}_ok || exit 1
        fi
done

Update December 3 2019

It should end with:

--- sets ---
===> Built sets to /home/User/netbsd/7.2/src/../evbarm/release/evbarm/binary/sets
===> build.sh ended:      Tue Dec  3 18:31:20 UTC 2019
===> Summary of results:
         build.sh command:    ./build.sh -U -u -j 20 -a earmv6hf -m evbarm -O ../evbarm/obj -D ../evbarm/dest -R ../evbarm/release -T ../evbarm/tools sets
         build.sh started:    Tue Dec  3 18:29:44 UTC 2019
         NetBSD version:      7.2
         MACHINE:             evbarm
         MACHINE_ARCH:        earmv6hf
         Build platform:      NetBSD 8.1 amd64
         HOST_SH:             /bin/sh
         MAKECONF file:       /etc/mk.conf
         TOOLDIR path:        /home/User/netbsd/7.2/src/../evbarm/tools
         DESTDIR path:        /home/User/netbsd/7.2/src/../evbarm/dest
         RELEASEDIR path:     /home/User/netbsd/7.2/src/../evbarm/release
         Updated makewrapper: /home/User/netbsd/7.2/src/../evbarm/tools/bin/nbmake-evbarm
         Building sets from pre-populated /home/User/netbsd/7.2/src/../evbarm/dest
         Built sets to /home/User/netbsd/7.2/src/../evbarm/release/evbarm/binary/sets
         build.sh ended:      Tue Dec  3 18:31:20 UTC 2019
===> .

The main problem with Netbsd build system is to cross-compile its pkgsrc distributed packages. Although there are some instructions on the net, they seems to be incomplete and buggy (perhaps this might explain why its firmware gets thinner from version to version).

So, the best way to add new software so far to the airport currently seems to compile it on the unit it self.

To do so, one must be able to install the netbsd distribution set on the device. One could use the internal hard drive on the timemachine device -- but it is not recommended as it uses a HFS partition that do not support device files in netbsd. The lack of device files implies on a much less compatibility for running chrooted applications.

The alternative is to have install an additional USB drive, if you have a spare USB port on the unit, or to shrink the HFS partition to open up space for an additional one.

Using the host machine the compiled it create a new ffs partition and untar the distribution files, they should be on: /home/User/netbsd/7.2/src/../evbarm/release/evbarm/binary/sets.

To be able to compile packages, download the current pkgsrc source distribution and untar it to the usr/pkgsrc directory on the target usb disk.

And, btw, it is not recommended to use SSD drives on Netbsd 6 as it do not support TRIM.

After you have root access to you device, some changes are needed to mount the USB file, on location that is convenient is to mount it at the boot on the /Volumes folder.

Doing so will let both the standard Apple binaries as well as custom chrooted ones to have access to your shared files (useful if you want to replace the current cifs stack with a fully working samba4, for instance).

In my case, I wanted to have rsyncd running to be able to differential back up files directly from some linux boxes without imposing the huge network overhead I would have with cifs.

SSH to your airport and mount the disk:

airport# mount /dev/sd0e /Volumes/

Create the devices:

airport# mkdir -p /Volumes/dev
airport# cp /dev/MAKEDEV /Volumes/dev
airport# (cd /Volumes/dev && ./MAKEDEV all)

Create a /mnt/Flash/rc.local similar to this. Remember to replace /dev/sd0e with the address of your USB device and /dev/dk2 with the address of your internal hard disk data partition and /Volumes/dk2 with its respective mounting point.

Keep in mind that changing the rc.local of the machine might lead to boot problems and should be done with extreme care as it could render the device inoperable or even brick it, so use it at your own risk.

#!/bin/sh

(
        echo Linking config files
        for i in rc.local man.conf; do
                if [ -f /mnt/Flash/${i} ] ; then
                        #if [ ! -l /etc/${i} ] ; then
                                ln -s /mnt/Flash/${i} /etc/${i}
                        #fi
                fi
        done
) &

(
        echo Link autoexec files and referencing in .profile
        if [ -f /.profile ] ; then
                chmod 755 /.profile
                echo >> /.profile
                for i in environment profile; do
                        if [ -f /mnt/Flash/${i} ] ; then
                                echo ". /mnt/Flash/${i}">>/.profile
                                #if [ ! -l /etc/${i} ] ; then
                                        ln -s /mnt/Flash/${i} /etc/${i}
                                #fi
                        fi
                done
        fi
)&

(
        echo Adding SSH keys for password less login
        if [ -f /mnt/authorized_keys ]; then
                mkdir -p /.ssh && chmod 700 /.ssh && cp /mnt/flash/authorized_keys /.ssh
        fi
) &

(
        echo Checking and mounting volumes
        if [ -d /Volumes ] ; then
                fsck -y /dev/sd0e ;
                case $? in
                        2)
                                reboot
                                ;;
                        *)
                                ;;
                esac

                if [ -d /Volumes/dk2/SharedRoot ] ; then
                        umount -f /Volumes/dk2 || reboot
                        REMOUNT_DK=yes
                fi

                mount /dev/sd0e /Volumes
                if [ $REMOUNT_DK ] ; then
                        mount -t hfs /dev/dk2 /Volumes/dk2
                fi

                echo Waiting mount to complete
                while [ ! -d /Volumes/etc ] ; do sleep 3; done

                echo Executing rc.chroot
                if [ -x /mnt/Flash/rc.chroot ] ; then
                        /bin/sh /mnt/Flash/rc.chroot
                fi
        fi
) &

(
        echo Linking /usr/pkg
        if [ ! -s /usr/pkg ] ; then
                ln -s /Volumes/usr/pkg /usr/pkg
        fi
) &

Create a new file /mnt/Flash/environment that will add /usr/pkg/bin to the path as well as add the environment variables needed to native compile anything.

export PATH=/Volumes/usr/local/bin:/Volumes/usr/local/sbin:/Volumes/sbin:/Volumes/bin:/Volumes/usr/sbin:/Volumes/usr/bin:/Volumes/usr/libexec:/usr/pkg/sbin:/usr/pkg/bin:/usr/local/sbin:/usr/local/bin:$PATH

export LANG="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
export LC_ALL=""

export CC1=/Volumes/usr/libexec/cc1
export C_INCLUDE_PATH=/Volumes/usr/include/
export LIBRARY_PATH=/Volumes/lib:/Volumes/dk2/root/usr/lib
export CPUFLAGS="-mfloat-abi=hard -mfpu=vfp3 -march=armv6zk -mtune=arm1176jzf-s"
export CFLAGS="-mno-unaligned-access -fno-use-linker-plugin -static -zmuldefs"
export CPPFLAGS="-mno-unaligned-access -fno-use-linker-plugin -static -zmuldefs"
export CXXFLAGS="-mno-unaligned-access -fno-use-linker-plugin -static -zmuldefs"
export LDFLAGS="-mno-unaligned-access -fno-use-linker-plugin -static -zmuldefs"
export LDSTATIC='-static'
export ALLOW_VULNERABLE_PACKAGES=yes

alias chr='/Volumes/usr/sbin/chroot /Volumes/ /bin/sh '

It also ads an alias to easily enter in the chrooted full netbsd using the chr command.

Finally you can add a man.conf suited to use when you are not chrooted the man command.

airport# sed -e 's/\/usr/\/Volumes\/usr/g' /Volumes/etc/man.conf > /mnt/Flash/man.conf

To compile a package. In a nutshell:

airport# chr
airport# cd /usr/pkgsrc
airport# cd net/zsync
airport# make all install

In practice is more complicated than that, all builds need to be static -- as the Apple Netbsd 6 lacks the ability to dynamic load libraries. That is particularly troublesome with many languages. Many packages require tweaks on their configurations.

I have managed, so far, to compile with some success the following: bash 2.05, bison-3.2.4, m4, openssl-1.0.2p, openvpn-2.4.7nb1, rsync-3.1.3nb1 (need to compile a simple perl working), unzip-6.0nb9, zsync-0.6.2. And I have partially working builds of perl-5.28.2, ocaml-4.0.7, python2 and python3.

The resulting binaries are now available in the repository:

NetBSD 7.2 sets for airport 5th gen

• Add table of contents this markdown document
• Fix the links to point to NetBSD archive
• Document how to compile and install pkgsrc specific packages.
• Update to current version of PKGSRC
• Contribute with running packages.
• Maybe create one page for each step, but with specific instructions for all device kinds...