Continuous Integration - openbmc/openbmc GitHub Wiki

Jenkins

OpenBMC project has a Jenkins Instance for continuous integration.

Build and configuration

The machine uses the official Jenkins image running in a docker container.

We want to run OpenBMC builds under docker also, so that we can easily test various distributions. To avoid docker in docker, we will add the host as a jenkins node and run the docker build instances there.

Therefore the host:

  • runs the latest upstream docker
  • has jenkins master running in official docker container
  • is running as a jenkins slave build node, with a local jenkins user
  • runs jenkins builds in docker containers on the host, kicked off by the jenkins master

Docker prep

As the machine is using Ubuntu 14.04 (which comes with Docker 1.01) we will use the upstream docker repo (supports exec and overlay, etc).

apt-key adv --keyserver hkp:_pgp.mit.edu:80 --recv-keys \
58118E89F3A912897C070ADBF76221572C52609D
cat > /etc/apt/sources.list.d/docker.list << EOF
# Ubuntu Trusty
deb https:_apt.dockerproject.org/repo ubuntu-trusty main
EOF
apt-get update && apt-get install docker

Set docker to use overlay fs instead of aufs.

echo 'DOCKER_OPTS="--storage-driver=overlay"' >> /etc/default/docker
mkdir -p /scratch/var/lib/docker/
echo "/var/lib/docker /scratch/var/lib/docker none bind 0 0" >> /etc/fstab
mount -a
service docker restart

Jenkins prep

Create the home dir for the host jenkins user.

mkdir -p /scratch/var/lib/jenkins/
echo "/var/lib/jenkins /scratch/var/lib/jenkins none bind 0 0" >> /etc/fstab
mount -a

The jenkins user inside the docker container has a home dir at /var/jenkins_home so we will mount this directory from the host (/scratch/var/lib/jenkins/) rather than using a docker image container, for easy access to shared data.

The jenkins user in the jenkins docker container has uid and gid of 1000, which on the ubuntu host is the ubuntu user. Therefore this directory will be owned on the host by the ubuntu user (which will be the jenkins user inside the container).

chmod 777 /scratch
chmod +t /scratch
mkdir -p /scratch/docker-jenkins/jenkins_home
chown ubuntu:ubuntu /scratch/docker-jenkins

Install dependencies for running as a jenkins slave and docker

apt-get install openjdk-7-jre expect libsdl1.2debian

Create a local jenkins user on the host and add to docker group to run containers for jenkins slave builds.

adduser jenkins --home /var/lib/jenkins
passwd jenkins
gpasswd -a jenkins docker
chown jenkins:jenkins -Rf /scratch/var/lib/jenkins

Switch to the jenkins user, create ssh keys (to log in from jenkins host) and test.

su - jenkins
ssh-keygen
ssh-copy-id localhost
exit
Create local reference git repos

Switch to jenkins user and pull some git repos.

su - jenkins
mkdir git && cd git
git clone https:_github.com/openbmc/openbmc.git

Create Jenkins docker container

Now that we have docker running, let's create the jenkins container, passing through our mount point for shared data. We will configure it to restart automatically so that the docker service starts it up again on a reboot. We pass through ports 8080 for Jenkins and 50000 for slave communication.

docker run --name jenkins --restart=always -p 50000:50000 -p 8080:8080 -v \
/scratch/docker-jenkins/jenkins_home:/var/jenkins_home -it jenkins

Jenkins configuration

Make sure the following plugins are installed and enabled:

  • conditional-buildstep
  • Dynamic Axis
  • embeddable-build-status
  • GitHub plugin
  • GitHub API Plugin
  • GitHub Pull Request Builder
  • JUnit Plugin
  • Matrix Authorization Strategy Plugin
  • Matrix Project Plugin
  • Multiple SCMs plugin
  • Parameterized Trigger plugin
  • Plain Credentials Plugin
  • Run Condition Plugin
  • Run Condition Extras Plugin
  • SSH Slaves plugin
  • Token Macro Plugin
  • Workspace Cleanup Plugin

Browse to Manage Jenkins -> Configure System and set:

  • Shell executable default shell to /bin/bash
  • Quiet period to 10 or 20

Set job subject line to HTML, so that links to PR work. Browse to Manage Jenkins -> Configure Global Security and set:

  • Markup Formatter to Safe HTML

Add an ssh key to allow jenkins to ssh into the host: Browse to Credentials -> Global credentials (unrestricted)

  • Click Add Credentials
  • Under Kind drop down, select SSH Username with private key
  • Username to "jenkins"
  • Under Private Key select From the Jenkins master ~/.ssh
  • Under Passphrase type in the passphrase for the key
  • Under Description type something meaningful

Browse to Manage Jenkins -> Manage Nodes and select:

  • New Node

Fill in the details for the host:

  • Node name = Slave
  • Dumb Slave
  • Select OK
    • # of executors = 10
    • Remote root directory = /var/lib/jenkins
    • Labels = docker
    • Launch method = Unix machines via SSH
      • Host = openpower.xyz
      • Credentials = the ssh one you added previously
    • Save

Back at Nodes, select the master and click Configure, set:

  • # of executors = 0
  • Save
Generate a GitHub token

Browse to Manage Jenkins -> Configure System

Under GitHub Pull Request Builder click Create API Token and enter the username and password of the GitHub account who Jenkins will be managing the pull requests on behalf of (perhaps a jenkins bot).

Select the newly created credentials from the Credentials drop down.

Note: If you are using two-factor auth on GitHub, then you might need to create the token manually instead, with permissions repo and repo:status. Then in Jenkins, create a new api.github.com credential domain with Add domain and use those details from GitHub to add a credential account of Shared Text type.

Now set the following:

/images/01-github-pr.png /images/02-github-pr.png

GitHub hook build job

At Jenkins login screen click New Item to create our job: /images/02-jenkins-job-creation.png

Now, set the following: /images/03-jenkins-job-details.png /images/04-jenkins-job.png /images/05-jenkins-job.png /images/06-jenkins-job.png /images/07-jenkins-job.png /images/08-jenkins-job.png /images/09-jenkins-job.png

Pull Request Job

For the Pull request job, create a New Item but copy the above job openbmc-build.

/images/12-jenkins-pr-job.png

Then before saving, make the following changes:

Tick This build is parameterized and set:

/images/12-jenkins-pr-job.png

Set the Refspec and branch to build under the git job:

/images/14-jenkins-pr-job-git.png

Under Triggers, remove Build when a change is pushed to GitHub and tick GitHub Pull Request Builder and set:

/images/15-jenkins-pr-job-builder.png

Remove post build steps:

/images/15-jenkins-pr-job-remove.png

Save the job.

Adding external slaves

We have some external slaves hooked up to openpower.xyz. They should have a fast internet connection for downloading and uploading packages, at least 30GB of disk space, and more than 4 CPU cores to be useful.

This method is for machines with a public facing IP address.

  1. Fire up a machine with a modern Linux distribution. Ubuntu 16.04 is the most tested option
  2. Install java, docker and git
  3. Add a jenkins user with the home directory /var/lib/jenkins-slave
  4. Add https://ozlabs.org/~joel/openpower.xyz-jenkins-pubkey.txt to jenkins' ~/.ssh/authorized_keys
  5. Contact [email protected] (shenki on #openbmc) and ask for the slave to be added to openpower.xyz, providing the machine name and IP address or DNS name