BuildServer - gemian/gemian GitHub Wiki
Build Server
We are using Jenkins with the jenkins-debian-glue configuration that automatically sets up qemu to build for arm64.
Note: GKA server was setup on a Debian 9 Host, using ipv6, which already had lxc installed and configured hosting other things. LXC info is probably not relevant to most so that is put second, though you would want to start with that if your using containers.
Jenkins
https://jenkins-debian-glue.org/
Using NAT64 gets access to IPv4 internet. GitHub unfortunately is not IPv6 enabled, they have added it to their wall of tasks.
Unfortunately Arm64 builds fail with the default settings, have to use apt not aptitude, this can be fixed by adding a file /root/.pbuilderrc containing:
PBUILDERSATISFYDEPENDSCMD=/usr/lib/pbuilder/pbuilder-satisfydepends-apt
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816097 for more info.
For builds that depend upon other builds we need to add the local repository as an input to other builds, create a file /usr/share/jenkins-debian-glue/pbuilder-hookdir/D00localrepo containing (swap paths for your own ones):
apt-get -y install wget gnupg
if [ -z $DIST ];
then
echo "deb http://ftp.debian.org/debian stretch-backports main\ndeb http://gemian.thinkglobally.org/stretch stretch main" > /etc/apt/sources.list.d/dh-virtualenv.list
elif [ $DIST = "stretch" ];
then
echo "deb http://ftp.debian.org/debian stretch-backports main\ndeb http://gemian.thinkglobally.org/$DIST $DIST main" > /etc/apt/sources.list.d/dh-virtualenv.list
else
echo "deb http://gemian.thinkglobally.org/$DIST $DIST main" > /etc/apt/sources.list.d/dh-virtualenv.list
fi
wget -O /tmp/archive-key.asc http://gemian.thinkglobally.org/archive-key.asc
apt-key add /tmp/archive-key.asc
apt-get update
cat /etc/apt/sources.list.d/dh-virtualenv.list
Your builds in jenkins will also need to set your build execute shell to something like (swap the key ID for your own one):
export POST_BUILD_HOOK=/usr/bin/jdg-debc
export TRUNK_RELEASE="stretch"
export ARCHITECTURES='amd64 arm64 source'
export KEY_ID="AD1CD6D4BA888E60"
/usr/bin/build-and-provide-package
And your check execute shell to something like:
export REPOS="stretch"
echo "Listing packages inside the ${REPOS} repository:"
/usr/bin/repository_checker --list-repos "${REPOS}"
Speeding up the builds a little bit by including some of the things that need to be installed for all builds, actually we need at least wget for the key downloading:
sudo chroot /var/cache/pbuilder/base-stretch-amd64.cow/
apt-get install wget ca-certificates gnupg debhelper apt-utils
Repeat for arm64.
Builds will need signing, which will fail if you've not recently entered your gpg key password. So you'll need to login to the jenkins user and perform some action that will request the key, I tend to find that export works well as this fixes a previous build that failed at this point:
reprepro -b /srv/repository/stretch -v export
Extend the timeout so you don't have to keep logging in again all the time, add large ttls to the file: .gnupg/gpg-agent.conf
default-cache-ttl 34560000
max-cache-ttl 34560000
I had to restart the server to get it to recognise the new ttls, logging in/out wasn't sufficient.
Build order
Packages need building in order of dependency.
- xkeyboard-config - to enable our keyboard
- glibc - to cope with old kernels (not needed with 4.14)
- yamui - to show gemian logo whilst booting
- mkbootimg
- android-headers
- pulseaudio - to allow external to pulseaudio module compilation
- libhybris - depends (android-headers)
- libglibutil
- libgbinder - depends (libglibutil)
- audiosystem-passthrough - depends (libglibutil, libgbinder)
- pulseaudio-modules-droid-[modern|old] - depends (pulseaudio, android-headers, libhybris)
- pulseaudio-modules-droid-hidl - depends (audiosystem-passthrough, pulseaudio-modules-droid-[modern|old])
- initramfs-tools-halium - depends (yamui)
- hybris-usb - to allow usb tethering
- gemian-system - to add the keyboard id etc
- lxc-android - to start an lxc containerised android to talk to the hardware
lxc specific info
This is how we setup the lxc container that we then installed jenkins onto. For pbuilder/debootstrap to work it needs to be a privileged one:
sudo lxc-create -n jenkins -t download
Distribution: debian
Release: stretch
Architecture: amd64
You should then assign your ip address by editing the config:
/var/lib/lxc/jenkins/config
add:
lxc.network.ipv6 = 2001:.../64
Older Debian's:
/var/lib/lxc/jenkins/rootfs/etc/network/interfaces
change:
iface eth0 inet dhcp
to:
iface eth0 inet6 static
address 2001:...
network 64
Newer Debian's: can use DHCP to detect from the config as set in: /etc/systemd/network/eth0.network
/var/lib/lxc/jenkins/rootfs/etc/resolv.conf
swap nameserver to one that exists (ignore warning the resolvconf package isn't installed):
nameserver 2001:...
This can then be started with:
sudo lxc-start -n jenkins
Or if you want to see the logs:
sudo lxc-start -n jenkins --logpriority DEBUG --logfile /home/[user]/jenkinslog
All being well you can now dive into it:
sudo lxc-attach -n jenkins
The attach method is a bit of a brute force jump in as root, its generally best to setup user accounts and the like.
Sometimes lxc attach will have lost the path, if so configure a path:
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
set the passwd:
passwd
add a user account:
adduser foobar
allow remote access:
apt-get install openssh-server
add sudo:
apt-get install sudo
add your user to the group sudo so they can do sudo things:
adduser foobar sudo
install other useful things:
apt-get install nano iputils-ping git
leave:
exit
Now you can ssh into your container.
Now follow the http://jenkins-debian-glue.org instructions (automatic).
Then apply the fixes above.
Debos peculiarities for building rootfs
The rootfs building done by debos requires the build container to have a kernel installed even if its not the one running on the host system, when you have different kernels you'll need to create a config symlink to ghosting the running kernel back to the installed one so that things don't fail.
a@b:/boot$ sudo ln -s config-6.1.0-7-amd64 config-5.10.0-21-amd64
Debugging build problems with some packages
AutoPkgTest fails
Some packages fail the autopkgtest stage, haven't figured out the root of the problem so when we get this:
+ [ -x /usr/bin/autopkgtest ]
+ newpid autopkgtest --output-dir /tmp/buildd/autopkgtest.out --summary /tmp/buildd/autopkgtest.summary /tmp/buildd/glibc*_arm64.deb -- null
clone: Invalid argument
+ EXIT=1
We just disable the test (ADT=skip) in the environment variables passed to debian-jenkins-glue's build-and-provide-package:
export MAIN_ARCHITECTURE='arm64'
export ADT="skip"
/usr/bin/build-and-provide-package