Building Calico 3.x - linux-on-ibm-z/docs GitHub Wiki

Building Calico

The instructions specify the steps to build Calico version v3.27.0 on Linux on IBM Z for following distributions:

  • RHEL (7.8, 7.9, 8.6, 8.8, 8.9, 9.0, 9.2, 9.3)
  • SLES (12 SP5, 15 SP5)
  • Ubuntu (20.04, 22.04, 23.10)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
  • <host_run_etcd_ip> and <host-ip> will be referred to the IP address of the host where etcd service is run.
  • Please note that in case if any calico module build gets failed, make sure to run make clean before rerunning the build command.

Step 1: Build environment set up

  • Ensure the current user belongs to group docker:

Use the below command to add group docker if it does not exist:

sudo groupadd docker

Use the below command to add current user to group docker if it has not been done:

sudo usermod -aG docker $USER && newgrp docker

Step 2: Build using script

If you want to build Calico using manual steps, go to STEP 3.

Use the following commands to build Calico using the build script. Please make sure you have wget installed.

wget https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Calico/3.27.0/build_calico.sh

# Build Calico
bash build_calico.sh   [Provide -t option for executing build with tests]

If the build and tests complete successfully, go to STEP 7. In case of error, check logs at <source_root>/logs/ for more details or go to STEP 3 to follow manual build steps.

In case build is successful but tests fail, run the following command to set Calico environment and go to STEP 6.3.

export SOURCE_ROOT=/<source_root>/
source $SOURCE_ROOT/setenv.sh    #Source environment file

Step 3: Install the system dependencies

  • RHEL (7.8, 7.9)

    sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
sudo yum-config-manager --enable docker-ce-stable
sudo yum install -y curl git wget tar gcc glibc-static.s390x docker-ce make which patch
sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x
sudo chmod 755 /usr/local/bin/yq.v2
export CC=gcc

  • RHEL (8.6, 8.8, 8.9, 9.0, 9.2, 9.3)

    sudo yum remove -y podman buildah
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
sudo yum install -y curl git wget tar gcc glibc.s390x docker-ce docker-ce-cli containerd.io make which patch
sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x
sudo chmod 755 /usr/local/bin/yq.v2
export CC=gcc

  • SLES 12 SP5

    sudo zypper install -y curl git wget tar gcc glibc-devel-static make which patch docker libnghttp2-devel
wget --no-check-certificate https://github.com/docker/buildx/releases/download/v0.6.1/buildx-v0.6.1.linux-s390x
mkdir -p ~/.docker/cli-plugins
mv buildx-v0.6.1.linux-s390x ~/.docker/cli-plugins/docker-buildx
chmod a+x ~/.docker/cli-plugins/docker-buildx
sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x
sudo chmod 755 /usr/local/bin/yq.v2
export CC=gcc

  • SLES 15 SP5

    sudo zypper install -y curl git wget tar gcc glibc-devel-static make which patch docker containerd docker-buildx
sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x
sudo chmod 755 /usr/local/bin/yq.v2
export CC=gcc

  • Ubuntu (20.04, 22.04, 23.10)

    sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
    "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |
   sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt-get update
sudo apt-get install -y patch git curl tar gcc wget make docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin clang
sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x
sudo chmod 755 /usr/local/bin/yq.v2

Step 4: Install prerequisites

4.1) Export Environment Variable

export SOURCE_ROOT=/<source_root>/
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Calico/3.27.0/patch

4.2) Install Go 1.21.5

Instructions for installing Go can be found here

mkdir $SOURCE_ROOT/go
export GOPATH=$SOURCE_ROOT/go

4.3) Download ETCD v3.5.1 binary

cd $SOURCE_ROOT
wget --no-check-certificate https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-s390x.tar.gz
tar xvf etcd-v3.5.1-linux-s390x.tar.gz
sudo cp -f etcd-v3.5.1-linux-s390x/etcd /usr/local/bin

Step 5: Build required components and images

5.1) Start docker service

sudo service docker start

Next, install buildx for docker (Only on SLES 12 SP5):

cd $SOURCE_ROOT
wget https://github.com/docker/buildx/releases/download/v0.6.1/buildx-v0.6.1.linux-s390x
mkdir -p ~/.docker/cli-plugins
mv buildx-v0.6.1.linux-s390x ~/.docker/cli-plugins/docker-buildx
chmod a+x ~/.docker/cli-plugins/docker-buildx

5.2) Build bpftool image

This builds a docker image calico/bpftool:v5.3-s390x. Certain components require the binary provided in this image.

Clone the projectcalico/bpftool github repository:

git clone https://github.com/projectcalico/bpftool $GOPATH/src/github.com/projectcalico/bpftool
cd $GOPATH/src/github.com/projectcalico/bpftool
sed -i 's,buster-slim,bullseye-slim,g' Dockerfile.s390x
sed -i 's,libgcc-8-dev,libgcc-10-dev,g' Dockerfile.s390x

Then build the calico/bpftool:v5.3-s390x image:

ARCH=s390x VERSION=v5.3 make image

5.3) Build go-build

This builds a docker image calico/go-build that is used to build other components

Clone the projectcalico/go-build github repository.

rm -rf $GOPATH/src/github.com/projectcalico/go-build
git clone -b v0.89 https://github.com/projectcalico/go-build $GOPATH/src/github.com/projectcalico/go-build
cd $GOPATH/src/github.com/projectcalico/go-build
curl -s $PATCH_URL/go-build.patch | patch -p1

Then buildcalico/go-build:v0.89 image:

ARCH=s390x VERSION=v0.89 ARCHIMAGE='$(DEFAULTIMAGE)' make image
docker tag calico/go-build:v0.89 calico/go-build:v0.89-s390x

5.4) Build BIRD

mkdir $GOPATH/tmp
cd $GOPATH/tmp
wget -O bird.tar.gz https://github.com/projectcalico/bird/tarball/v0.3.3-202-g7a77fb7
tar xvfz bird.tar.gz
cd projectcalico-bird-7a77fb7/
./build.sh
make image -f Makefile.calico
docker tag calico/bird:latest-s390x calico/bird:v0.3.3-202-g7a77fb73-s390x
cd $GOPATH
sudo rm -rf $GOPATH/tmp

5.5) Download Calico source code and apply patches

git clone -b v3.27.0 https://github.com/projectcalico/calico $GOPATH/src/github.com/projectcalico/calico
cd $GOPATH/src/github.com/projectcalico/calico
curl -s $PATCH_URL/calico.patch | patch -p1

5.6) Build Calico images and binaries

cd $GOPATH/src/github.com/projectcalico/calico
ARCH=s390x make image
ARCH=s390x make -C felix image
ARCH=s390x make -C api build
ARCH=s390x make bin/helm

5.7) Apply docker tags

Apply docker tags to label images with the version.

docker tag calico/node:latest-s390x calico/node:v3.27.0
docker tag calico/felix:latest-s390x calico/felix:v3.27.0
docker tag calico/typha:latest-s390x calico/typha:master-s390x
docker tag calico/typha:latest-s390x calico/typha:v3.27.0
docker tag calico/ctl:latest-s390x calico/ctl:v3.27.0
docker tag calico/cni:latest-s390x calico/cni:v3.27.0
docker tag calico/apiserver:latest-s390x docker.io/calico/apiserver:v3.27.0
docker tag calico/pod2daemon-flexvol:latest-s390x calico/pod2daemon:latest-s390x
docker tag calico/pod2daemon:latest-s390x calico/pod2daemon:v3.27.0
docker tag calico/pod2daemon-flexvol:latest-s390x calico/pod2daemon-flexvol:v3.27.0
docker tag calico/kube-controllers:latest-s390x calico/kube-controllers:v3.27.0
docker tag calico/dikastes:latest-s390x calico/dikastes:v3.27.0
docker tag calico/flannel-migration-controller:latest-s390x calico/flannel-migration-controller:v3.27.0

Verify that following docker images with respective tags are created:

REPOSITORY                            TAG
calico/felix                          latest-s390x                
calico/felix                          v3.27.0                     
felix                                 latest-s390x                
calico/node                           latest-s390x                
calico/node                           v3.27.0                     
node                                  latest-s390x                
calico/typha                          latest-s390x                
calico/typha                          master-s390x                
calico/typha                          v3.27.0                     
typha                                 latest-s390x                
calico/dikastes                       latest-s390x                
calico/dikastes                       v3.27.0                     
dikastes                              latest-s390x                
calico/flannel-migration-controller   latest-s390x                
calico/flannel-migration-controller   v3.27.0                     
flannel-migration-controller          latest-s390x                
calico/kube-controllers               latest-s390x                
calico/kube-controllers               v3.27.0                     
kube-controllers                      latest-s390x                
calico/apiserver                      latest-s390x                
calico/apiserver                      v3.27.0                     
apiserver                             latest-s390x                
calico/cni                            latest-s390x                
calico/cni                            v3.27.0                     
cni                                   latest-s390x                
calico/ctl                            latest-s390x                
calico/ctl                            v3.27.0                     
ctl                                   latest-s390x                
calico/node-driver-registrar          latest-s390x                
node-driver-registrar                 latest-s390x                
calico/csi                            latest-s390x                
csi                                   latest-s390x                
calico/pod2daemon-flexvol             latest-s390x                
calico/pod2daemon-flexvol             v3.27.0                     
calico/pod2daemon                     latest-s390x                
calico/pod2daemon                     v3.27.0                     
pod2daemon-flexvol                    latest-s390x                
calico/bird                           latest-s390x                
calico/bird                           v0.3.3-202-g7a77fb73-s390x  
birdbuild-s390x                       latest                      
calico/go-build                       v0.89                       
calico/go-build                       v0.89-s390x                 
calico/bpftool                        v5.3-s390x

5.8) Test Calico node

  • First start etcd in background that is required for running calico/node

    etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 &

  • Start calico/node

    cd $GOPATH/src/github.com/projectcalico/calico/calicoctl
sudo ETCD_ENDPOINTS=http://127.0.0.1:2379 bin/calicoctl-linux-s390x node run --node-image=calico/node:latest-s390x

Check the output and confirm that calico/node is successfully started.

Step 6: Calico testcases (Optional)

6.1) Build calico/dind

  • Clone the projectcalico/dind github repository

    git clone https://github.com/projectcalico/dind $GOPATH/src/github.com/projectcalico/dind
cd $GOPATH/src/github.com/projectcalico/dind

  • Build the dind image

    docker build -t calico/dind -f Dockerfile-s390x .

6.2) Copy ETCD artifact

cd $GOPATH/src/github.com/projectcalico/calico/node
mkdir -p calico_test/pkg
cp $SOURCE_ROOT/etcd-v3.5.1-linux-s390x.tar.gz calico_test/pkg

6.3) Execute Calico Tests

Note: While running below command for running tests, make sure etcd and calico-node are not running already. Kill etcd process and stop calico-node container using following commands:

sudo pkill etcd
docker rm -f calico-node

Execute test cases for Calico components

cd $GOPATH/src/github.com/projectcalico/calico/node
ARCH=s390x CALICOCTL_VER=latest CNI_VER=latest-s390x make test_image
docker tag calico/test:latest-s390x calico/test:latest
ARCH=s390x CALICOCTL_VER=latest CNI_VER=latest-s390x make st

cd $GOPATH/src/github.com/projectcalico/calico/felix
ARCH=s390x make ut

cd $GOPATH/src/github.com/projectcalico/calico/kube-controllers
ARCH=s390x make fv

cd $GOPATH/src/github.com/projectcalico/calico/calicoctl
ARCH=s390x make test

cd $GOPATH/src/github.com/projectcalico/calico/cni-plugin
ARCH=s390x make ut

cd $GOPATH/src/github.com/projectcalico/calico/confd
ARCH=s390x make ut

cd $GOPATH/src/github.com/projectcalico/calico/app-policy
ARCH=s390x make ut

cd $GOPATH/src/github.com/projectcalico/calico/apiserver
ARCH=s390x make test

cd $GOPATH/src/github.com/projectcalico/calico/api
ARCH=s390x make test

cd $GOPATH/src/github.com/projectcalico/calico/typha
ARCH=s390x make ut

cd $GOPATH/src/github.com/projectcalico/calico/pod2daemon
ARCH=s390x make test

cd $GOPATH/src/github.com/projectcalico/calico/libcalico-go
ARCH=s390x make ut

6.4) Verify test results

Test results for individual components can be found in their respective repository under report folder.

Following example shows how to rerun a failing test cases in Node module:

cd $GOPATH/src/github.com/projectcalico/calico/node
ARCH=s390x CALICOCTL_VER=latest CNI_VER=latest-s390x make st ST_TO_RUN="tests/st/bgp/test_ipip.py:TestIPIP.test_gce_rr_1"

For example, expected output for a successful Node test case result:

----------------------------------------------------------------------
XML: /code/report/nosetests.xml
[success] 100.00% tests.st.bgp.test_single_route_reflector.TestSingleRouteReflector.test_bird_single_route_reflector: 90.3319s
----------------------------------------------------------------------
Ran 1 test in 90.601s

OK
make stop-etcd
make[1]: Entering directory '/home/test/go/src/github.com/projectcalico/calico/node'
calico-etcd
make[1]: Leaving directory '/home/test/go/src/github.com/projectcalico/calico/node'

Note:

  • tests.st.bgp.test_ipip.TestIPIP.test_issue_1584_0_bird of Node component fails for both s390x and x86 on some distros.
  • Many of tests.st.bgp.test_* and nose.suite tests of Node component fails for both s390x and x86 on RHEL 7.x, 9.x and Ubuntu 22.04, 23.10. There are because of cgroup not being mounted.
  • There are intermittent test failures in Node component. To run a subset of failed tests, you can refer here for more information.
  • Some intermittent test failures in Typha component are observed in fv-tests/server_test.go which can be addressed by re-running the testcase and/or using systemd Cgroup Driver for Docker.
  • Increase open files ulimit setting if test fails with too many open files error.

Step 7. Calico Integration

Please refer to Calico with Kubernetes for more information.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

⚠️ **GitHub.com Fallback** ⚠️