Building Calico 3.x - linux-on-ibm-z/docs GitHub Wiki
The instructions specify the steps to build Calico version v3.27.0 on Linux on IBM Z for following distributions:
- RHEL (8.6, 8.8, 8.9, 9.0, 9.2, 9.3)
- SLES (12 SP5, 15 SP5)
- Ubuntu (20.04, 22.04, 23.10)
General Notes:
- When following the steps below please use a standard permission user unless otherwise specified.
- A directory
/<source_root>/
will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it. <host_run_etcd_ip>
and<host-ip>
will be referred to the IP address of the host where etcd service is run.- Please note that in case if any calico module build gets failed, make sure to run
make clean
before rerunning the build command.
- Ensure the current user belongs to group
docker
:
Use the below command to add group docker
if it does not exist:
sudo groupadd docker
Use the below command to add current user to group docker
if it has not been done:
sudo usermod -aG docker $USER && newgrp docker
If you want to build Calico using manual steps, go to STEP 3.
Use the following commands to build Calico using the build script. Please make sure you have wget installed.
wget https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Calico/3.27.0/build_calico.sh
# Build Calico
bash build_calico.sh [Provide -t option for executing build with tests]
If the build and tests complete successfully, go to STEP 7. In case of error, check logs at <source_root>/logs/
for more details or go to STEP 3 to follow manual build steps.
In case build is successful but tests fail, run the following command to set Calico environment and go to STEP 6.3.
export SOURCE_ROOT=/<source_root>/
source $SOURCE_ROOT/setenv.sh #Source environment file
-
RHEL (8.6, 8.8, 8.9, 9.0, 9.2, 9.3)
sudo yum remove -y podman buildah sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo sudo yum install -y curl git wget tar gcc glibc.s390x docker-ce docker-ce-cli containerd.io make which patch sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x sudo chmod 755 /usr/local/bin/yq.v2 export CC=gcc
-
SLES 12 SP5
sudo zypper install -y curl git wget tar gcc glibc-devel-static make which patch docker libnghttp2-devel wget --no-check-certificate https://github.com/docker/buildx/releases/download/v0.6.1/buildx-v0.6.1.linux-s390x mkdir -p ~/.docker/cli-plugins mv buildx-v0.6.1.linux-s390x ~/.docker/cli-plugins/docker-buildx chmod a+x ~/.docker/cli-plugins/docker-buildx sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x sudo chmod 755 /usr/local/bin/yq.v2 export CC=gcc
-
SLES 15 SP5
sudo zypper install -y curl git wget tar gcc glibc-devel-static make which patch docker containerd docker-buildx sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x sudo chmod 755 /usr/local/bin/yq.v2 export CC=gcc
-
Ubuntu (20.04, 22.04, 23.10)
sudo apt-get update sudo apt-get install -y ca-certificates curl gnupg sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg sudo chmod a+r /etc/apt/keyrings/docker.gpg echo \ "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null sudo apt-get update sudo apt-get install -y patch git curl tar gcc wget make docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin clang sudo wget -O /usr/local/bin/yq.v2 https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_s390x sudo chmod 755 /usr/local/bin/yq.v2
export SOURCE_ROOT=/<source_root>/
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Calico/3.27.0/patch
Instructions for installing Go can be found here
mkdir $SOURCE_ROOT/go
export GOPATH=$SOURCE_ROOT/go
cd $SOURCE_ROOT
wget --no-check-certificate https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-s390x.tar.gz
tar xvf etcd-v3.5.1-linux-s390x.tar.gz
sudo cp -f etcd-v3.5.1-linux-s390x/etcd /usr/local/bin
sudo service docker start
Next, install buildx
for docker (Only on SLES 12 SP5):
cd $SOURCE_ROOT
wget https://github.com/docker/buildx/releases/download/v0.6.1/buildx-v0.6.1.linux-s390x
mkdir -p ~/.docker/cli-plugins
mv buildx-v0.6.1.linux-s390x ~/.docker/cli-plugins/docker-buildx
chmod a+x ~/.docker/cli-plugins/docker-buildx
docker pull calico/bpftool:v7.4.0-s390x
This builds a docker image calico/go-build
that is used to build other components
Clone the projectcalico/go-build
github repository.
rm -rf $GOPATH/src/github.com/projectcalico/go-build
git clone -b v0.89 https://github.com/projectcalico/go-build $GOPATH/src/github.com/projectcalico/go-build
cd $GOPATH/src/github.com/projectcalico/go-build
curl -sSL $PATCH_URL/go-build.patch | git apply --ignore-whitespace -
Then buildcalico/go-build:v0.89
image:
ARCH=s390x VERSION=v0.89 ARCHIMAGE='$(DEFAULTIMAGE)' make image
docker tag calico/go-build:v0.89 calico/go-build:v0.89-s390x
mkdir $GOPATH/tmp
cd $GOPATH/tmp
wget -O bird.tar.gz https://github.com/projectcalico/bird/tarball/v0.3.3-202-g7a77fb7
tar xvfz bird.tar.gz
cd projectcalico-bird-7a77fb7/
./build.sh
make image -f Makefile.calico
docker tag calico/bird:latest-s390x calico/bird:v0.3.3-202-g7a77fb73-s390x
cd $GOPATH
sudo rm -rf $GOPATH/tmp
git clone -b v3.27.0 https://github.com/projectcalico/calico $GOPATH/src/github.com/projectcalico/calico
cd $GOPATH/src/github.com/projectcalico/calico
curl -s $PATCH_URL/calico.patch | patch -p1
cd $GOPATH/src/github.com/projectcalico/calico
ARCH=s390x make image
ARCH=s390x make -C felix image
ARCH=s390x make -C api build
ARCH=s390x make bin/helm
Apply docker tags to label images with the version.
docker tag calico/node:latest-s390x calico/node:v3.27.0
docker tag calico/felix:latest-s390x calico/felix:v3.27.0
docker tag calico/typha:latest-s390x calico/typha:master-s390x
docker tag calico/typha:latest-s390x calico/typha:v3.27.0
docker tag calico/ctl:latest-s390x calico/ctl:v3.27.0
docker tag calico/cni:latest-s390x calico/cni:v3.27.0
docker tag calico/apiserver:latest-s390x docker.io/calico/apiserver:v3.27.0
docker tag calico/pod2daemon-flexvol:latest-s390x calico/pod2daemon:latest-s390x
docker tag calico/pod2daemon:latest-s390x calico/pod2daemon:v3.27.0
docker tag calico/pod2daemon-flexvol:latest-s390x calico/pod2daemon-flexvol:v3.27.0
docker tag calico/kube-controllers:latest-s390x calico/kube-controllers:v3.27.0
docker tag calico/dikastes:latest-s390x calico/dikastes:v3.27.0
docker tag calico/flannel-migration-controller:latest-s390x calico/flannel-migration-controller:v3.27.0
Verify that following docker images with respective tags are created:
REPOSITORY TAG
calico/felix latest-s390x
calico/felix v3.27.0
felix latest-s390x
calico/node latest-s390x
calico/node v3.27.0
node latest-s390x
calico/typha latest-s390x
calico/typha master-s390x
calico/typha v3.27.0
typha latest-s390x
calico/dikastes latest-s390x
calico/dikastes v3.27.0
dikastes latest-s390x
calico/flannel-migration-controller latest-s390x
calico/flannel-migration-controller v3.27.0
flannel-migration-controller latest-s390x
calico/kube-controllers latest-s390x
calico/kube-controllers v3.27.0
kube-controllers latest-s390x
calico/apiserver latest-s390x
calico/apiserver v3.27.0
apiserver latest-s390x
calico/cni latest-s390x
calico/cni v3.27.0
cni latest-s390x
calico/ctl latest-s390x
calico/ctl v3.27.0
ctl latest-s390x
calico/node-driver-registrar latest-s390x
node-driver-registrar latest-s390x
calico/csi latest-s390x
csi latest-s390x
calico/pod2daemon-flexvol latest-s390x
calico/pod2daemon-flexvol v3.27.0
calico/pod2daemon latest-s390x
calico/pod2daemon v3.27.0
pod2daemon-flexvol latest-s390x
calico/bird latest-s390x
calico/bird v0.3.3-202-g7a77fb73-s390x
birdbuild-s390x latest
calico/go-build v0.89
calico/go-build v0.89-s390x
calico/bpftool v7.4.0-s390x
-
First start
etcd
in background that is required for runningcalico/node
etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 &
-
Start
calico/node
cd $GOPATH/src/github.com/projectcalico/calico/calicoctl sudo ETCD_ENDPOINTS=http://127.0.0.1:2379 bin/calicoctl-linux-s390x node run --node-image=calico/node:latest-s390x
Check the output and confirm that calico/node
is successfully started.
-
Clone the
projectcalico/dind
github repositorygit clone https://github.com/projectcalico/dind $GOPATH/src/github.com/projectcalico/dind cd $GOPATH/src/github.com/projectcalico/dind
-
Build the dind image
docker build -t calico/dind -f Dockerfile-s390x .
cd $GOPATH/src/github.com/projectcalico/calico/node
mkdir -p calico_test/pkg
cp $SOURCE_ROOT/etcd-v3.5.1-linux-s390x.tar.gz calico_test/pkg
Note: While running below command for running tests, make sure etcd
and calico-node
are not running already. Kill etcd
process and stop calico-node
container using following commands:
sudo pkill etcd
docker rm -f calico-node
Execute test cases for Calico components
cd $GOPATH/src/github.com/projectcalico/calico/node
ARCH=s390x CALICOCTL_VER=latest CNI_VER=latest-s390x make test_image
docker tag calico/test:latest-s390x calico/test:latest
ARCH=s390x CALICOCTL_VER=latest CNI_VER=latest-s390x make st
cd $GOPATH/src/github.com/projectcalico/calico/felix
ARCH=s390x make ut
cd $GOPATH/src/github.com/projectcalico/calico/kube-controllers
ARCH=s390x make fv
cd $GOPATH/src/github.com/projectcalico/calico/calicoctl
ARCH=s390x make test
cd $GOPATH/src/github.com/projectcalico/calico/cni-plugin
ARCH=s390x make ut
cd $GOPATH/src/github.com/projectcalico/calico/confd
ARCH=s390x make ut
cd $GOPATH/src/github.com/projectcalico/calico/app-policy
ARCH=s390x make ut
cd $GOPATH/src/github.com/projectcalico/calico/apiserver
ARCH=s390x make test
cd $GOPATH/src/github.com/projectcalico/calico/api
ARCH=s390x make test
cd $GOPATH/src/github.com/projectcalico/calico/typha
ARCH=s390x make ut
cd $GOPATH/src/github.com/projectcalico/calico/pod2daemon
ARCH=s390x make test
cd $GOPATH/src/github.com/projectcalico/calico/libcalico-go
ARCH=s390x make ut
Test results for individual components can be found in their respective repository under report
folder.
Following example shows how to rerun a failing test cases in Node
module:
cd $GOPATH/src/github.com/projectcalico/calico/node
ARCH=s390x CALICOCTL_VER=latest CNI_VER=latest-s390x make st ST_TO_RUN="tests/st/bgp/test_ipip.py:TestIPIP.test_gce_rr_1"
For example, expected output for a successful Node
test case result:
----------------------------------------------------------------------
XML: /code/report/nosetests.xml
[success] 100.00% tests.st.bgp.test_single_route_reflector.TestSingleRouteReflector.test_bird_single_route_reflector: 90.3319s
----------------------------------------------------------------------
Ran 1 test in 90.601s
OK
make stop-etcd
make[1]: Entering directory '/home/test/go/src/github.com/projectcalico/calico/node'
calico-etcd
make[1]: Leaving directory '/home/test/go/src/github.com/projectcalico/calico/node'
Note:
-
tests.st.bgp.test_ipip.TestIPIP.test_issue_1584_0_bird
ofNode
component fails for boths390x
andx86
on some distros. - Many of
tests.st.bgp.test_*
andnose.suite
tests ofNode
component fails for boths390x
andx86
onRHEL 9.x and Ubuntu 22.04, 23.10
. There are because of cgroup not being mounted. - There are intermittent test failures in
Node
component. To run a subset of failed tests, you can refer here for more information. - Some intermittent test failures in
Typha
component are observed infv-tests/server_test.go
which can be addressed by re-running the testcase and/or usingsystemd
Cgroup Driver
for Docker. - Increase
open files
ulimit
setting if test fails withtoo many open files
error.
Please refer to Calico with Kubernetes for more information.