sam calico - levivic/zEcoTest GitHub Wiki

Building Calicoctl

The instructions provided below specify the steps to build Calico for Docker version v1.1.1 on Linux on the IBM z Systems for following distributions:

RHEL (7.1, 7.2, 7.3)
SLES (12 SP1, 12 SP2)
Ubuntu (16.04, 16.10)

General Notes:

When following the steps below please use a standard permission user unless otherwise specified.
A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Prerequisites

Go -- Instructions for building Go can be found here.
Docker -- Instructions for install Docker can be found here
etcd -- Instructions for building etcd can be found here

Building Calicoctl

1. Install the system dependencies

For RHEL (7.1, 7.2, 7.3)

   sudo yum install curl git wget tar gcc glibc-static.s390x make

For SLES (12 SP1, 12 SP2)

   sudo zypper install curl git wget tar gcc glibc-static.s390x make

For Ubuntu (16.04, 16.10)

   sudo apt-get update
   sudo apt-get install git curl tar gcc wget make

2. Build dependent 3rd part packages

2.1 Build `runit`

   cd /<source_root>/
   wget http://smarden.org/runit/runit-2.1.2.tar.gz
   tar xvf runit-2.1.2.tar.gz
   cd admin/runit-2.1.2/src
   make load
   cd ../

Update src/load to add -static option to make link statically

   #!/bin/sh
   # WARNING: This file was auto-generated. Do not edit!
   main="$1"; shift
   exec gcc -s -o "$main" "$main".o -static ${1+"$@"}

Then build and pack executable files

   package/compile
   tar cvf runit.tar command/
   gzip runit.tar

This runit.tar.gz is used in building calico/node docker images

2.2 Build `glibc`

   cd /<source_root>/
   git clone https://github.com/sgerrand/docker-glibc-builder
   cd docker-glibc-builder/

Update Dockerfile to use s390x/ubuntu base image:

@@ -1,4 +1,4 @@
-FROM ubuntu-debootstrap:14.04
+FROM s390x/ubuntu
MAINTAINER Sasha Gerrand <[email protected]>

Build a docker image and then use it to build glibc

   docker build -t glibc-builder .
   docker run --rm -e STDOUT=1 glibc-builder 2.24 /usr/glibc-compat > glibc-bin-2.24.tar.gz

glibc package is used in building docker images.

3. Build `calicoctl` and other components of `calico`

3.1 Build `go-build`

This builds a docker image calico/go-build that is used to build other components by containerized build

   cd /<source_root>/
   mkdir -p calico/src/github.com/projectcalico
   cd calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/go-build
   cd go-build
   git checkout v0.5

Modify Dockerfile for s390x

@@ -1,4 +1,4 @@
-FROM golang:1.8.1-alpine
+FROM tmh1999/alpine-s390x
MAINTAINER Tom Denham <[email protected]>

# Install su-exec for use in the entrypoint.sh (so processes run as the right user)
@@ -9,22 +9,25 @@ MAINTAINER Tom Denham <[email protected]>
# Install wget for fetching glibc
# Install make for building things
# Install util-linux for column command (used for output formatting).
+RUN apk update
RUN apk add --no-cache su-exec curl bash git mercurial make wget util-linux

+RUN apk add go
# Install glibc
RUN wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub
-RUN wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk
-RUN apk add glibc-2.23-r3.apk
+COPY pkg /pkg/
+RUN  cd / && \
+     tar xvf pkg/glibc-bin-2.24.tar.gz && \
+     mv /pkg/ld.so.conf /usr/glibc-compat/etc/   &&  \
+     /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc/usr/lib && \
+     rm -rf /pkg

# Disable cgo so that binaries we build will be fully static.
ENV CGO_ENABLED=0
+ENV GOROOT=/usr/lib/go
+ENV GOPATH=/go
+ENV PATH=$PATH:$GOROOT/bin:$GOPATH/bin

-# Apply patches to Go runtime and recompile.
-# See https://github.com/golang/go/issues/5838 for defails of vfork patch.
-COPY patches/use-clone-vfork-b7edfba429d982e3e065d637334bcc63ad49f8f9.patch \
-     /tmp/use-clone-vfork-b7edfba429d982e3e065d637334bcc63ad49f8f9.patch
-RUN cd /usr/local/go && \
-    patch -p 1 < /tmp/use-clone-vfork-b7edfba429d982e3e065d637334bcc63ad49f8f9.patch
RUN go install -v -a syscall

   mkdir pkg
   cp /<source_root>/docker-glibc-builder/glibc-bin-2.24.tar.gz pkg/

Create a file pkg/ld.so.conf with follow content:

     /usr/local/lib
     /usr/glibc-compat/lib
     /usr/lib
     /lib

Then start to build calico/go-build image

   make

3.2 Build `calicoctl`

This component builds interactive command interface calicoctl and docker image calico/node

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/calicoctl
   cd calicoctl
   git checkout v1.1.1

Modify Makefile.calicoctl for s390x:

@@ -6,7 +6,7 @@
###############################################################################
# Determine which OS / ARCH.
OS := $(shell uname -s | tr A-Z a-z)
-ARCH := amd64
+ARCH := s390x
###############################################################################
# Subcomponent versions:
GO_BUILD_VER:=latest
@@ -75,11 +75,11 @@ binary: $(CALICOCTL_FILES) vendor
        GOOS=$(OS) GOARCH=$(ARCH) CGO_ENABLED=0 go build -v $$INSTALL_FLAG -o dist/calicoctl-$(OS)-$(ARCH) $(LDFLAGS) "./calicoctl/calicoctl.go"

dist/calicoctl: $(CALICOCTL_FILES) vendor
-       $(MAKE) dist/calicoctl-linux-amd64
-       mv dist/calicoctl-linux-amd64 dist/calicoctl
+       $(MAKE) dist/calicoctl-linux-s390x
+       mv dist/calicoctl-linux-s390x dist/calicoctl

-dist/calicoctl-linux-amd64: $(CALICOCTL_FILES) vendor
-       $(MAKE) OS=linux ARCH=amd64 binary-containerized
+dist/calicoctl-linux-s390x: $(CALICOCTL_FILES) vendor
+       $(MAKE) OS=linux ARCH=s390x binary-containerized

Then make dist/calicoctl to build, the executable file is in dist/calicoctl Because calico/node needs the executable files built from other components, let's build other components first, then come back.

   make  dist/calicoctl
   mkdir -p /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/    ## store executable files built from other components

3.3 Build `bird`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/bird
   cd bird
   git checkout v0.3.0

Modify Dockerfile for s390x

@@ -1,4 +1,4 @@
-FROM alpine:latest
+FROM tmh1999/alpine-s390x
MAINTAINER Tom Denham <[email protected]>

Then run build.sh to build 3 executable files (dist/) that are required by building calico/node image.

   ./build.sh
   cp dist/* /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

3.4 Build `confd`

   cd /<source_root>
   git clone https://github.com/kelseyhightower/confd.git calico/src/github.com/kelseyhightower/confd
   cd calico/src/github.com/kelseyhightower/confd
   export GOROOT=<go_instll_dir>
   export PATH=$GOROOT/bin:$PATH
   export GOPATH=/<source_root>/calico
   go build -ldflags "-v -linkmode=external '-extldflags=-v -static'"
   cp confd /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

The built executable file confd is required by calico/node.

3.4 Build `libnetwork-plugin`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/libnetwork-plugin
   cd libnetwork-plugin
   git checkout  v1.1.0

Modify Dockerfile to change FROM alpine to FROM tmh1999/alpine-s390x

   make calico/libnetwork-plugin
   cp dist/libnetwork-plugin /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

This builds executable file dist/libnetwork-plugin required by calico/node as well as docker image calico/libnetwork-plugin

3.5 Build `felix`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/felix
   cd felix
   git checkout 2.1.1

Modify Makefile to use calico/go-build:latest

@@ -80,7 +80,7 @@ help:

-GO_BUILD_CONTAINER?=calico/go-build:v0.4
+GO_BUILD_CONTAINER?=calico/go-build

To build felix it needs felixbackend.pb.go that is generated by a docker image calico/protoc. Let's first built this image.

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/tigera/docker-protobuf
   cd  docker-protobuf

Modify Dockerfile as follows:

@@ -1,4 +1,4 @@
-FROM golang:1.6
+FROM s390x/ubuntu

MAINTAINER Shaun Crampton <[email protected]>

@@ -8,6 +8,10 @@ WORKDIR /src
# We want v3.0.0 but the tag for v3.0.0 has broken download links.  Pin to
# master for now :-(
ENV PROTOBUF_TAG master
+RUN apt-get -y  update \
+     && apt-get -y  install golang git curl
+ENV GOROOT=/usr/lib/go
+ENV GOPATH=/usr/local

Then build docker images calico/protoc

   docker build -t calico/protoc .

Let's come back to felix and build

   cd /<source_root>/calico/src/github.com/projectcalico/felix
   make bin/calico-felix
   cp bin/calico-felix /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

The built calico-felix is in bin/

3.6 Build `calico-bgp-daemon`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/calico-bgp-daemon
   cd calico-bgp-daemon
   git checkout v0.2.0

Modify Dockerfile to change to FROM tmh1999/alpine-s390x and then build

@@ -1,4 +1,4 @@
-FROM alpine:3.4
+FROM tmh1999/alpine-s390x

   make build-containerized
   cp dist/* /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

Two executable files calico-bgp-daemon and gobgp are in dist/, which are required by calico/node

3.7 Build image `calico/node`

Now let's come beck to calicoctl and build image calico/node

   cd  /<source_root>/calico/src/github.com/projectcalico/calicoctl
   cp -r /<source_root>/calico/src/github.com/projectcalico/go-build/pkg  calico_node/
   cp /<source_root>/admin/runit-2.1.2/runit.tar.gz calico_node/pkg/

Modify Makefile.calico-node:

@@ -123,7 +123,7 @@ $(NODE_CONTAINER_BIN_DIR)/allocate-ipip-addr: dist/allocate-ipip-addr

## Build startup.go
startup:
-       GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -v -i -o dist/startup $(LDFLAGS) "./calico_node/startup/startup.go"
+       GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -v -i -o dist/startup $(LDFLAGS) "./calico_node/startup/startup.go"

dist/startup: $(STARTUP_FILES) vendor
        mkdir -p dist
@@ -139,7 +139,7 @@ dist/startup: $(STARTUP_FILES) vendor

## Build allocate_ipip_addr.go
allocate-ipip-addr:
-       GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -v -i -o dist/allocate-ipip-addr $(LDFLAGS) "./calico_node/allocateipip/allocate_ipip_addr.go"
+       GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -v -i -o dist/allocate-ipip-addr $(LDFLAGS) "./calico_node/allocateipip/allocate_ipip_addr.go"

dist/allocate-ipip-addr: $(ALLOCATE_IPIP_FILES) vendor
        mkdir -p dist
@@ -198,11 +198,11 @@ certs/.certificates.created:
        touch certs/.certificates.created

busybox.tar:
-       docker pull busybox:latest
-       docker save --output busybox.tar busybox:latest
+       docker pull s390x/busybox:latest
+       docker save --output busybox.tar s390x/busybox:latest

routereflector.tar:
-       docker pull calico/routereflector:latest
+       #docker pull calico/routereflector:latest
        docker save --output routereflector.tar calico/routereflector:latest

and modify docker file calico_node/Dockerfile

@@ -11,25 +11,24 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM alpine:3.4
+FROM  tmh1999/alpine-s390x
MAINTAINER Tom Denham <[email protected]>

# Set the minimum Docker API version required for libnetwork.
ENV DOCKER_API_VERSION 1.21

+COPY pkg /pkg/
+RUN apk update
# Download and install glibc for use by non-static binaries that require it.
RUN apk --no-cache add wget ca-certificates libgcc && \
     wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub && \
-    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && \
-    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-bin-2.23-r3.apk && \
-    apk add glibc-2.23-r3.apk glibc-bin-2.23-r3.apk && \
-    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc/usr/lib && \
-    apk del wget && \
-    rm -f glibc-2.23-r3.apk glibc-bin-2.23-r3.apk
-
+    tar xvf pkg/glibc-bin-2.24.tar.gz && \
+    mv pkg/ld.so.conf /usr/glibc-compat/etc/   &&  \
+    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc/usr/lib
# Install runit from the community repository, as its not yet available in global
-RUN apk add --no-cache --repository "http://alpine.gliderlabs.com/alpine/edge/community" runit
-
+RUN  tar xvf pkg/runit.tar.gz  && \
+     cp command/* /sbin/

Modify calico_node/filesystem/etc/service/available/confd/run

@@ -3,6 +3,6 @@ exec 2>&1

-exec confd -confdir=/etc/calico/confd -interval=5 -watch -no-discover --log-level=debug \
+exec confd -confdir=/etc/calico/confd -interval=5 -watch  --log-level=debug \

Start to build calico/node:

   make calico/node

4. Test `calicoctl` (optional)

To run the test, it requires a few docker images. They are as follows:

calico/node // already built
calico/test, // test modular
calico/dind // docker in docker
quay.io/coreos/etcd // tecd
nginx // load balance(?)
calico/routereflector // routing
workload // test load/client

We will build them one by one

4.1 Build image `quay.io/coreos/etcd`

   cd  /<source_root>/calico/src/github.com/projectcalico/
   git clone https://github.com/coreos/etcd
   cd etcd
   git checkout v3.1.5

Modify Dockerfile-release for s390x

@@ -1,7 +1,8 @@
-FROM alpine:latest
+FROM tmh1999/alpine-s390x

-ADD etcd /usr/local/bin/
-ADD etcdctl /usr/local/bin/
+ADD bin/etcd /usr/local/bin/
+ADD bin/etcdctl /usr/local/bin/
+ENV ETCD_UNSUPPORTED_ARCH=s390x
RUN mkdir -p /var/etcd/
RUN mkdir -p /var/lib/etcd/

Then build etcd and image

   ./build
   docker build -f Dockerfile-release  -t quay.io/coreos/etcd .
   cd bin
   tar cvf etcd-v3.1.5-linux-s390x.tar etcd etcdctl
   gzip etcd-v3.1.5-linux-s390x.tar

4.2 Build image `calico/dind`

Image calico/dind uses docker-in-docker solution. We first need to build docker

   cd  /<source_root>/calico/src/github.com/projectcalico/
   git clone https://github.com/docker/docker
   cd docker
   git checkout v1.13.1
   make binary
   cd bundles/1.13.1/binary-daemon
   cp ../binary-client/docker-1.13.1 .
   ln -s docker-1.13.1 docker
   tar cvf dd-1.13.1.tar docker*
   gzip dd-1.13.1.tar

The built executable files are in bundles/1.13.1/binary-client and bundles/1.13.1/binary-daemon

   cd  /<source_root>/calico/src/github.com/projectcalico/
   git clone https://github.com/docker-library/docker docker-lib
   cd  docker-lib/17.03
   cp  /<source_root>/calico/src/github.com/projectcalico/docker/bundles/1.13.1/binary-daemon/dd-1.13.1.tar.gz .

Modify Dockerfile to install docker system (1.13.1) built before

@@ -1,21 +1,17 @@
-FROM alpine:3.5
+FROM tmh1999/alpine-s390x

RUN apk add --no-cache \
                ca-certificates \
                curl \
                openssl

-ENV DOCKER_BUCKET get.docker.com
-ENV DOCKER_VERSION 17.03.1-ce
-ENV DOCKER_SHA256 820d13b5699b5df63f7032c8517a5f118a44e2be548dd03271a86656a544af55
-
+COPY dd-1.13.1.tar.gz /tmp/
RUN set -x \
-       && curl -fSL "https://${DOCKER_BUCKET}/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz" -o docker.tgz \
-       && echo "${DOCKER_SHA256} *docker.tgz" | sha256sum -c - \
-       && tar -xzvf docker.tgz \
-       && mv docker/* /usr/local/bin/ \
-       && rmdir docker \
-       && rm docker.tgz \
+        && mkdir -p /tmp/docker \
+       && tar -xzvf /tmp/dd-1.13.1.tar.gz -C /tmp/docker \
+        && rm -rf /tmp/docker/*m5 /tmp/docker/*sha256 \
+       && mv /tmp/docker/docker*  /usr/local/bin/ \
+       && rm -rf /tmp/docker \
        && docker -v

Build image docker:1.13.1

    docker build -t docker:1.13.1 .

Then build calico/dind

   cd dind

Modify Dockerfile to use the new built docker:1.13.1 image

@@ -1,4 +1,4 @@
-FROM docker:17.03
+FROM docker:1.13.1

   docker build -t calico/dind .

4.3 Build image `calico/routereflector`

   cd /<source_root>/calico/src/github.com/projectcalico/bird/build_routereflector
   cp /<source_root>/calico/src/github.com/kelseyhightower/confd/confd image
   cp ../dist/bird image/

Modify Dockerfile to use base image s390x/ubuntu

@@ -14,7 +14,7 @@

-FROM ubuntu:14.04
+FROM s390x/ubuntu

And modify image/install.sh

@@ -32,12 +32,11 @@ apt-get update
# - packages required by felix
# - pip (which includes various setuptools package discovery).
$minimal_apt_get_install \
-        bird \
-        bird6
-
+        bird
+cp build/bir* /usr/sbin/
# Install Confd
-curl -L https://www.github.com/kelseyhightower/confd/releases/download/v0.9.0/confd-0.9.0-linux-amd64 -o confd
-chmod +x confd
+cp build/confd /
+chmod +x /confd
+apt-get install -y python3

Modify image/cleanup.sh

@@ -6,7 +6,7 @@ set -x
-cat /tmp/add-apt.txt | xargs xargs dpkg -r --force-depends
+#cat /tmp/add-apt.txt | xargs xargs dpkg -r --force-depends

Build calico/routereflector

   docker build  -t calico/routereflector .

4.4 Build image `workload`

   cd  /<source_root>/calico/src/github.com/projectcalico/calicoctl

Modify workload/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.4
+FROM tmh1999/alpine-s390x
RUN apk add --no-cache \

Then build

   make workload.tar

4.4 Build image `nginx`

   cd  /<source_root>/calico/src/github.com/projectcalico/
   git clone https://github.com/nginxinc/docker-nginx
   cd docker-nginx
   git clone 1.11.10
   cd stable/alpine

Modify Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.4
+FROM tmh1999/alpine-s390x

Then build image

   docker build -t nginx .

4.4 Build image `calico/test`

   cd  /<source_root>/calico/src/github.com/projectcalico/calicoctl

Modify calico_test/Dockerfile.calico_test

@@ -32,7 +32,7 @@
# - run 'nosetests'
-FROM docker:1.13.0
+FROM docker:1.13.1
MAINTAINER Tom Denham <[email protected]>

# Running STs in this container requires that it has all dependencies installed
@@ -45,11 +45,9 @@ RUN apk add --update python python-dev py2-pip py-setuptools openssl-dev libffi-
-# Install etcdctl
-RUN wget https://github.com/coreos/etcd/releases/download/v2.3.3/etcd-v2.3.3-linux-amd64.tar.gz && \
-    tar -xzf etcd-v2.3.3-linux-amd64.tar.gz && \
-    cd etcd-v2.3.3-linux-amd64 && \
-    ln -s etcdctl /usr/local/bin/
+# Install etcdctl0
+COPY pkg /pkg/
+RUN tar -xzf pkg/etcd-v3.1.5-linux-s390x.tar.gz -C /usr/local/bin/

Modify tests/st/utils/workload.py

@@ -32,7 +32,7 @@ class Workload(object):
-    def __init__(self, host, name, image="busybox", network="bridge", ip=None, labels=[]):
+    def __init__(self, host, name, image="s390x/busybox", network="bridge", ip=None, labels=[]):
         """
         Create the workload and detect its IPs.

Modify tests/st/utils/docker_host.py

@@ -412,7 +412,7 @@ class DockerHost(object):

-    def create_workload(self, name, image="busybox", network="bridge", ip=None, labels=[]):
+    def create_workload(self, name, image="s390x/busybox", network="bridge", ip=None, labels=[]):
         """

Then build image calico/test

   mkdir -p calico_test/pkg
   cp /<source_root>/calico/src/github.com/projectcalico/etcd/bin/etcd-v3.1.5-linux-s390x.tar.gz  calico_test/pkg
   make calico_test.created

4.5 Run all tests of `Calicoctl`

Now it is ready to run the all tests

   make st

5. Run `calico with Docker`

After building calictoctl and calico/node, it is ready to run calico with docker, please refer Document

References:

https://github.com/projectcalico

sam calico - levivic/zEcoTest GitHub Wiki

Building Calicoctl

Prerequisites

Building Calicoctl

1. Install the system dependencies

2. Build dependent 3rd part packages

2.1 Build runit

2.2 Build glibc

3. Build calicoctl and other components of calico

3.1 Build go-build

3.2 Build calicoctl

3.3 Build bird

3.4 Build confd

3.4 Build libnetwork-plugin

3.5 Build felix

3.6 Build calico-bgp-daemon

3.7 Build image calico/node

4. Test calicoctl (optional)

4.1 Build image quay.io/coreos/etcd

4.2 Build image calico/dind

4.3 Build image calico/routereflector

4.4 Build image workload

4.4 Build image nginx

4.4 Build image calico/test

4.5 Run all tests of Calicoctl

5. Run calico with Docker

References:

2.1 Build `runit`

2.2 Build `glibc`

3. Build `calicoctl` and other components of `calico`

3.1 Build `go-build`

3.2 Build `calicoctl`

3.3 Build `bird`

3.4 Build `confd`

3.4 Build `libnetwork-plugin`

3.5 Build `felix`

3.6 Build `calico-bgp-daemon`

3.7 Build image `calico/node`

4. Test `calicoctl` (optional)

4.1 Build image `quay.io/coreos/etcd`

4.2 Build image `calico/dind`

4.3 Build image `calico/routereflector`

4.4 Build image `workload`

4.4 Build image `nginx`

4.4 Build image `calico/test`

4.5 Run all tests of `Calicoctl`

5. Run `calico with Docker`