Building Envoy - linux-on-ibm-z/docs GitHub Wiki

Building Envoy

The instructions provided below specify the steps to build Envoy version 1.29.2 on Linux on IBM Z for the following distributions:

  • RHEL (8.8, 8.9, 9.2, 9.3)
  • SLES (12 SP5, 15 SP5)
  • Ubuntu (20.04, 22.04, 24.04)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Envoy

Step 1: Build using script

If you want to build Envoy using manual steps, go to Step 2.

Use the following commands to build Envoy using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Envoy/1.29.2/build_envoy.sh

# Build Envoy
bash build_envoy.sh   [Provide -t option for executing build with tests]

In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

Step 2: Install Dependencies

export SOURCE_ROOT=/<source_root>/
export PREFIX=/usr/local
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Envoy/1.29.2/patch/"

Step 2.1) Install Basic Dependencies

  • RHEL (8.8, 8.9)

    sudo yum install -y autoconf curl libtool patch python3-pip pkg-config vim wget curl file git gzip make python2 python3 java-11-openjdk-devel unzip zip tar openssl-devel xz expat-devel glib2-devel gcc gcc-c++

#set JAVA_HOME location
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk
export PATH=$JAVA_HOME/bin:$PATH

  • RHEL (9.2, 9.3)

    sudo yum install -y wget curl bison ncurses ncurses-devel pkgconfig net-tools gcc-toolset-12-gcc gcc-toolset-12-gcc-c++ gcc-toolset-12-binutils-devel gcc-toolset-12-annobin-plugin-gcc gcc-toolset-12-libstdc++-devel gcc-toolset-12-binutils-gold gcc-toolset-12-libatomic-devel patch unzip openssl-devel java-11-openjdk-devel python3 zlib-devel diffutils libtool libatomic libarchive zip file

#set gcc 12 as default
source /opt/rh/gcc-toolset-12/enable

  • SLES 12 SP5

    sudo zypper install -y autoconf curl libtool patch python3-pip pkg-config vim wget git gzip make cmake python python3 python36 java-11-openjdk-devel unzip zip tar xz which gawk glib2-devel libexpat-devel bind-chrootenv coreutils ed expect file iproute2 iputils less libopenssl-devel python2 python3 python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel unzip zlib-devel python3-PyYAML gcc gcc-c++ gcc12-c++ git-core automake autoconf libtool libelf-devel gawk glibc-devel-static kmod binutils-gold

#set gcc 12 as default
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-12 12
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 12
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-12 12

#set JAVA_HOME location
export JAVA_HOME=/usr/lib64/jvm/java-11-openjdk
export PATH=$JAVA_HOME/bin:$PATH
    • Install Openssl 
      cd $SOURCE_ROOT
wget https://www.openssl.org/source/openssl-1.1.1w.tar.gz --no-check-certificate
tar -xzf openssl-1.1.1w.tar.gz
cd openssl-1.1.1w
./config --prefix=/usr/local --openssldir=/usr/local
make && sudo make install
sudo mkdir -p /usr/local/etc/openssl
sudo wget https://curl.se/ca/cacert.pem --no-check-certificate -P /usr/local/etc/openssl
export LDFLAGS="-L/usr/local/lib/ -L/usr/local/lib64/"
export LD_LIBRARY_PATH=/usr/local/lib/:/usr/local/lib64/${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export LD_RUN_PATH=/usr/local/lib:/usr/local/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
export CPPFLAGS="-I/usr/local/include/ -I/usr/local/include/openssl"
export SSL_CERT_FILE=/usr/local/etc/openssl/cacert.pem
sudo ldconfig /usr/local/lib64
    • Install CMake 
      cd $SOURCE_ROOT
wget https://github.com/Kitware/CMake/releases/download/v3.22.5/cmake-3.22.5.tar.gz
tar -xf cmake-3.22.5.tar.gz
cd cmake-3.22.5
./bootstrap -- -DCMAKE_BUILD_TYPE:STRING=Release
make && sudo make install
sudo ln -sf /usr/local/bin/cmake /usr/bin/cmake

  • SLES 15 SP5

    sudo zypper install -y autoconf curl libtool patch python3-pip pkg-config vim wget git gcc12-c++ gzip make cmake python python3 java-11-openjdk-devel unzip zip tar xz which gawk glib2-devel libexpat-devel meson ninja gobject-introspection-devel python3-bind bind-chrootenv coreutils ed expect file iproute2 iputils lcov less libopenssl-devel netcat python2 python2-devel python3 python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel unzip zlib-devel python3-python-gnupg python3-PyYAML

#set gcc 12 as default
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-12 12
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 12
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-12 12

#set JAVA_HOME location
export JAVA_HOME=/usr/lib64/jvm/java-11-openjdk
export PATH=$JAVA_HOME/bin:$PATH

  • Ubuntu 20.04

    sudo apt-get update 
sudo apt-get install -y autoconf curl git libtool patch python3-pip virtualenv pkg-config gcc g++ locales build-essential openjdk-11-jdk python2 python2-dev python-is-python3 python3 python3-dev zip unzip libssl-dev

#set JAVA_HOME location
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-s390x
export PATH=$JAVA_HOME/bin:$PATH

  • Ubuntu 22.04

    sudo apt-get update
sudo apt-get install -y autoconf curl git libtool patch python3-pip unzip virtualenv pkg-config locales libssl-dev build-essential openjdk-11-jdk python2 python2-dev python3 python3-dev zip

# Install GCC 12 from repo
sudo apt-get install -y gcc-12 g++-12

sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-12 12
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 12
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-12 12

#set JAVA_HOME location
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-s390x
export PATH=$JAVA_HOME/bin:$PATH

  • Ubuntu 24.04

    sudo apt-get update
sudo apt-get install -y autoconf curl git libtool patch python3-pip unzip virtualenv pkg-config locales gcc g++ openssl libssl-dev build-essential openjdk-11-jdk python3 gcc-12 g++-12 zip unzip

#set JAVA_HOME location
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-s390x
export PATH=$JAVA_HOME/bin:$PATH

# use gcc-12 to build bazel due to error compiling with gcc-13:  https://github.com/bazelbuild/bazel/issues/18642
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 20
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 20
sudo update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-12 20
sudo update-alternatives --install /usr/bin/gcov-tool gcov-tool /usr/bin/gcov-tool-12 20
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 13
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 13
sudo update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-13 13
sudo update-alternatives --install /usr/bin/gcov-tool gcov-tool /usr/bin/gcov-tool-13 13

Step 2.2) Install Bazel 6.3.2

cd $SOURCE_ROOT
mkdir bazel && cd bazel
wget https://github.com/bazelbuild/bazel/releases/download/6.3.2/bazel-6.3.2-dist.zip
unzip -q bazel-6.3.2-dist.zip
chmod -R +w .
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.3.2/patch/bazel.patch | patch -p1
bash ./compile.sh

export PATH=$PATH:${SOURCE_ROOT}/bazel/output/

Step 2.3) Install GCC (Only on RHEL 8.x, Ubuntu 20.04 and Ubuntu 24.04)

  • RHEL 8.x

    sudo yum install -y gcc-toolset-12-gcc-c++ gcc-toolset-12-libstdc++-devel gcc-toolset-12-binutils-devel gcc-toolset-12-binutils-gold gcc-toolset-12-libatomic-devel rust-toolset python3.11 pkgconf-pkg-config

#Set gcc 12 as default
source /opt/rh/gcc-toolset-12/enable

  • Ubuntu 20.04

    cd "$SOURCE_ROOT"
URL=https://ftp.gnu.org/gnu/gcc/gcc-12.3.0/gcc-12.3.0.tar.gz
curl -sSL $URL | tar xzf - || error "GCC 12.3.0"

cd gcc-12.3.0
./contrib/download_prerequisites
mkdir objdir && cd objdir

../configure --enable-languages=c,c++ --prefix=${PREFIX} \
  --enable-shared --enable-threads=posix \
  --disable-multilib --disable-libmpx \
  --with-system-zlib --with-long-double-128 --with-arch=zEC12 \
  --disable-libphobos --disable-werror \
  --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu

make -j 8 bootstrap
sudo make install

export PATH=${PREFIX}/bin${PATH:+:${PATH}}

LD_LIBRARY_PATH=${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
LD_LIBRARY_PATH+=:${PREFIX}/lib
LD_LIBRARY_PATH+=:/usr/lib64
export LD_LIBRARY_PATH

LD_RUN_PATH=${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
LD_RUN_PATH+=:${PREFIX}/lib
LD_RUN_PATH+=:/usr/lib64
export LD_RUN_PATH

  • Ubuntu 24.04

    sudo update-alternatives --remove-all gcc
sudo update-alternatives --remove-all g++
sudo update-alternatives --remove-all gcov
sudo update-alternatives --remove-all gcov-tool
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-13 12
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 12
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 12
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-13 12

Step 2.4) Install Clang

  • RHEL (8.x, 9.x)

    sudo yum install -y clang-14.0.6
export CC=clang
export CXX=clang++

  • SLES 12 SP5

    cd $SOURCE_ROOT
URL=https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-14.0.6.tar.gz
curl -sSL $URL | tar xzf - || error "Clang 14.0.6"
cd llvm-project-llvmorg-14.0.6
mkdir build && cd build
cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_C_COMPILER="/usr/bin/gcc" -DCMAKE_CXX_COMPILER="/usr/bin/g++"  -DCMAKE_BUILD_TYPE="Release" -G "Unix Makefiles" ../llvm
make clang -j8
clangbuild=${SOURCE_ROOT}/llvm-project-llvmorg-14.0.6/build
export PATH=$clangbuild/bin:$PATH
export LD_LIBRARY_PATH=$clangbuild/lib:$LD_LIBRARY_PATH
cd $clangbuild/bin
sudo ln -sf clang++ clang++-14
export CC=${SOURCE_ROOT}/llvm-project-llvmorg-14.0.6/build/bin/clang
export CXX=${SOURCE_ROOT}/llvm-project-llvmorg-14.0.6/build/bin/clang++-14

  • SLES 15 SP5

    sudo zypper install -y clang14 llvm14 bpftool libclang-cpp14 libunwind llvm14-gold libLLVM14 binutils-gold
export CC=clang
export CXX=clang++

  • Ubuntu 20.04

    sudo apt-get install -y clang-12

export CC=clang-12
export CXX=clang++-12

sudo ln -sf /usr/bin/clang-12 /usr/bin/clang
sudo ln -sf /usr/bin/clang++-12 /usr/bin/clang++

  • Ubuntu 22.04

    cd $SOURCE_ROOT
sudo apt-get update 
sudo apt install -y lsb-release wget software-properties-common gnupg
wget https://apt.llvm.org/llvm.sh
sed -i 's,add-apt-repository "${REPO_NAME}",add-apt-repository "${REPO_NAME}" -y,g' llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 14
rm ./llvm.sh

export CC=clang-14
export CXX=clang++-14

sudo ln -sf /usr/bin/clang-14 /usr/bin/clang
sudo ln -sf /usr/bin/clang++-14 /usr/bin/clang++

  • Ubuntu 24.04

    sudo apt-get install -y clang-14

export CC=clang-14
export CXX=clang++-14

sudo ln -sf /usr/bin/clang-14 /usr/bin/clang
sudo ln -sf /usr/bin/clang++-14 /usr/bin/clang++

Step 2.5) Install Rust

cd $SOURCE_ROOT
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh /dev/stdin -y
export PATH="$HOME/.cargo/bin:$PATH"

Step 2.6) Set locale (Ubuntu Only)

sudo locale-gen en_US.UTF-8
export LC_ALL=C
unset LANGUAGE

Step 2.7) Install Go 1.21.5

export GOPATH=$SOURCE_ROOT
cd $GOPATH
GO_VERSION=1.21.5
wget -q https://golang.org/dl/go"${GO_VERSION}".linux-s390x.tar.gz
chmod ugo+r go"${GO_VERSION}".linux-s390x.tar.gz
sudo rm -rf /usr/local/go /usr/bin/go
sudo tar -C /usr/local -xzf go"${GO_VERSION}".linux-s390x.tar.gz
sudo ln -sf /usr/local/go/bin/go /usr/bin/ 
sudo ln -sf /usr/local/go/bin/gofmt /usr/bin/
go version  
export PATH=$PATH:$GOPATH/bin

Step 2.8) Install Buildifier and Buildozer

cd $SOURCE_ROOT
git clone -b v6.3.2 https://github.com/bazelbuild/buildtools.git

#Build buildifer
cd $SOURCE_ROOT/buildtools/buildifier
bazel build //buildifier
export BUILDIFIER_BIN=$GOPATH/bin/buildifier

#Build buildozer
cd $SOURCE_ROOT/buildtools/buildozer
bazel build //buildozer 
export BUILDOZER_BIN=$GOPATH/bin/buildozer

Step 2.9) Clone rules_foreign_cc

cd $SOURCE_ROOT
git clone --depth 1 -b 0.10.1 https://github.com/bazelbuild/rules_foreign_cc.git
cd rules_foreign_cc/
curl -sSL $PATCH_URL/rules_foreign_cc.patch |  git apply -
wget -O $SOURCE_ROOT/rules_foreign_cc/toolchains/pkgconfig-valgrind.patch $PATCH_URL/pkgconfig-valgrind.patch

Step 3: Build and Install Envoy

  • Download Envoy and Apply Patches

    cd $SOURCE_ROOT
git clone --depth 1 -b v1.29.2 https://github.com/envoyproxy/envoy.git
cd envoy

    # Apply patch-
curl -sSL $PATCH_URL/envoy_patch.diff |  git apply -

#Apply patch to update certificates-
curl -sSL https://github.com/phlax/envoy/commit/c84d38dbc13982c899b9bedc290525938c92fd16.patch | git apply -

# Patch for failing tests-
curl -sSL $PATCH_URL/envoy-test.patch |  git apply -

# Copy patch files to envoy/bazel which will be applied to external packages while building envoy
wget -O $SOURCE_ROOT/envoy/bazel/boringssl-s390x.patch $PATCH_URL/boringssl-s390x.patch
wget -O $SOURCE_ROOT/envoy/bazel/cel-cpp-memory.patch $PATCH_URL/cel-cpp-memory.patch
wget -O $SOURCE_ROOT/envoy/bazel/grpc-s390x.patch $PATCH_URL/grpc-s390x.patch
wget -O $SOURCE_ROOT/envoy/bazel/foreign_cc/luajit-s390x.patch $PATCH_URL/luajit-s390x.patch
wget -O $SOURCE_ROOT/envoy/bazel/quiche-s390x.patch $PATCH_URL/quiche-s390x.patch
    • Only on SLES 12 SP5
    #Apply distro specific patch
curl -sSL $PATCH_URL/envoy-sl12.patch |  git apply -
wget -O $SOURCE_ROOT/envoy/bazel/io_uring.patch $PATCH_URL/io_uring.patch

  • Build Envoy

    bazel build envoy -c opt --override_repository=rules_foreign_cc=${SOURCE_ROOT}/rules_foreign_cc --config=clang

    The binary will be generated in $SOURCE_ROOT/envoy/bazel-bin/source/exe/envoy-static.

  • Verify the version of Envoy

    $SOURCE_ROOT/envoy/bazel-bin/source/exe/envoy-static --version

    Output should be similar to:

    bazel-bin/source/exe/envoy-static  version: 2092d65bd4d476be8235ea541e5d25c096b513e6/1.29.2/Modified/RELEASE/BoringSSL

Step 4: Test (optional)

cd $SOURCE_ROOT/envoy

# Run tests-
bazel test //test/... -c opt --override_repository=rules_foreign_cc=${SOURCE_ROOT}/rules_foreign_cc --config=clang --keep_going

Notes:

  • Below mentioned test failures are observed on all distros on Intel and s390x:

    • //test/extensions/filters/listener/original_dst:original_dst_integration_test: To fix the test, follow official doc to set up the original_dst filter for a quick manual test. To run this test after applying this fix, the following command can be used.

      bazel test //test/extensions/filters/listener/original_dst:original_dst_integration_test -c opt --override_repository=rules_foreign_cc=${SOURCE_ROOT}/rules_foreign_cc --config=clang

    • //test/extensions/tracers/dynamic_ot:dynamic_opentracing_driver_impl_test: The test can pass by adding --linkopt=-lstdc++ to bazel as discussed here .

  • Below mentioned test failure is observed on all RHEL and SLES distros on Intel and s390x:

    //test/config_test:example_configs_test: This test fails because the file "/etc/ssl/certs/ca-certificates.crt" only exists on debian like systems.

  • Below mentioned test failures are observed on SLES 12 SP5 on Intel and s390x:

    //test/common/common:packed_struct_test

    //test/common/http:header_utility_test

    //test/common/network:utility_test

    //test/extensions/transport_sockets/alts:alts_integration_test

  • Some tests might timeout as they are being run with heap checker enabled in "normal" mode. Heap checker can be disabled using below test command:

    bazel test //test/... -c opt --override_repository=rules_foreign_cc=${SOURCE_ROOT}/rules_foreign_cc --config=clang --keep_going --test_output=all --test_env=HEAPCHECK=

References: