Building Couchbase - linux-on-ibm-z/docs GitHub Wiki

Building Couchbase

The instructions provided below specify the steps to build Couchbase version 7.2.0 on Linux on IBM Z for following distributions:

  • RHEL (8.6, 8.8, 9.0, 9.2)
  • SLES (15 SP4, 15 SP5)
  • Ubuntu (20.04, 22.04)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
  • You can increase the sudo timeout by adding Defaults:USER timestamp_timeout=TIME (replace USER and TIME) at the end of sudo file (sudo visudo).

Step 1: Build using script

If you want to build Couchbase using manual steps, go to STEP 2.

Use the following commands to build Couchbase using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Couchbase/7.2.0/build_couchbase.sh

# Build couchbase
bash build_couchbase.sh #[Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 5. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

Step 2: Install prerequisites

  • defining deps and source root
export SOURCE_ROOT=<source root>
# set distro variable
source "/etc/os-release"
export DISTRO=$ID$VERSION_ID # ubuntu20.04 ubuntu22.04
export DISTRO=rhel9 #on rhel 9.x
export DISTRO=rhel8 #on rhel 8.x
export DISTRO=suse15 #on sles 15
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Couchbase/7.2.0/patch"
export CACHE_DIRECTORY=~/.cbdepscache/
mkdir $SOURCE_ROOT/patch
mkdir -p $CACHE_DIRECTORY

2.1) Install the system dependencies

  • RHEL (8.6, 8.8)
sudo subscription-manager repos --enable codeready-builder-for-rhel-8-s390x-rpms
sudo subscription-manager repos --enable rhel-8-for-s390x-appstream-rpms
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo yum install -y gcc-toolset-10 gcc-toolset-10-gcc gcc-toolset-10-gcc-c++ gcc-toolset-10-libatomic-devel
sudo ln -sf /opt/rh/gcc-toolset-10/root/usr/bin/gcc /usr/local/bin/gcc
sudo ln -sf /opt/rh/gcc-toolset-10/root/usr/bin/g++ /usr/local/bin/g++
sudo yum install -y atk-devel autoconf automake binutils-devel bison bzip2 \
  cmake cups-devel flex git gnome-keyring libcurl-devel langpacks-en glibc-all-langpacks \
  libev-devel libtool make ncurses-devel ninja-build openssl-devel openssl-perl \
  python2 python3 python3-devel python3-httplib2 tar texinfo unzip wget which xz xz-devel \
	glib2-devel clang diffutils libstdc++-devel procps asciidoctor

sudo ln -sf /usr/bin/python3 /usr/bin/python
export CC=/opt/rh/gcc-toolset-10/root/usr/bin/gcc
export CXX=/opt/rh/gcc-toolset-10/root/usr/bin/g++
export LANG=en_US.UTF-8
  • RHEL (9.0, 9.2)
sudo subscription-manager repos --enable codeready-builder-for-rhel-9-s390x-rpms
sudo subscription-manager repos --enable rhel-9-for-s390x-appstream-rpms
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo yum install -y gcc gcc-c++ atk-devel autoconf automake binutils-devel bison bzip2 \
  cmake cups-devel flex git gnome-keyring libcurl-devel langpacks-en glibc-all-langpacks \
  libtool make ncurses-devel ninja-build openssl-devel openssl-perl perl-core \
  python3 python3-devel tar texinfo unzip wget which xz xz-devel glib2-devel clang compat-openssl11 python3-httplib2 asciidoctor
sudo ln -sf /usr/bin/python3 /usr/bin/python
export LANG=en_US.UTF-8
export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
  • SLES 15 (SP4, SP5)
sudo sudo zypper install -y asciidoc autoconf automake cmake curl flex \
    gcc10-c++-10.3.0+git1587 gcc10-10.3.0+git1587 git-core glib2 glib2-devel glibc-locale xz \
    libopenssl-devel libncurses6 \
    libtirpc-devel libtool libxml2-tools libxslt-tools \
    make makedepend ncurses-devel ninja patch pkg-config \
    python python-xml python3-httplib2 re2c ruby sqlite3 tar \
    unixODBC wget which xinetd xmlto python3-pip
sudo ln -sf /usr/bin/python3 /usr/bin/python
pip3 install httplib2 --upgrade
pip3 install six

sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-10 10
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-10 10
export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
export LANG=en_US.UTF-8
  • Ubuntu (20.04)
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive TZ=America/Toronto apt-get install -y \
  autoconf automake autotools-dev binutils-dev bison ccache  cmake curl flex \
  git libssl-dev ncurses-dev ninja-build python python3 locales locales-all \
  python3-httplib2 python3-six pkg-config re2c texinfo tzdata unzip wget \
  g++-10 gcc-10 gcc-10-multilib g++-10-multilib libglib2.0-dev libtool

sudo ln -sf /usr/bin/python3 /usr/bin/python
export LANG=en_US.UTF-8
export CC=gcc
export CXX=g++

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 9
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-9 9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10
  • Ubuntu (22.04)
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive TZ=America/Toronto apt-get install -y \
  autoconf automake autotools-dev binutils-dev bison ccache  cmake curl flex \
  git libssl-dev ncurses-dev ninja-build python3 locales locales-all \
  python3-httplib2 python3-six pkg-config re2c texinfo tzdata unzip wget \
  g++-10 gcc-10 gcc-10-multilib g++-10-multilib libglib2.0-dev libtool

sudo ln -sf /usr/bin/python3 /usr/bin/python
export LANG=en_US.UTF-8
export CC=gcc
export CXX=g++

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10
  • Install Go 1.19.7
cd $SOURCE_ROOT
wget https://golang.org/dl/go1.19.7.linux-s390x.tar.gz
chmod ugo+r go1.19.7.linux-s390x.tar.gz
sudo tar -C /usr/local -xzf go1.19.7.linux-s390x.tar.gz
export PATH=/usr/local/go/bin:$PATH
sudo ln -sf /usr/bin/gcc /usr/bin/s390x-linux-gnu-gcc
go version
  • Install CMake 3.24.2
cd $SOURCE_ROOT
wget https://cmake.org/files/v3.24/cmake-3.24.2.tar.gz
tar -xzf cmake-3.24.2.tar.gz
cd cmake-3.24.2
./bootstrap --prefix=/usr
make -j$(nproc)
sudo make install
hash -r
  • Install gflags 2.2.2
cd $SOURCE_ROOT
version=2.2.2
wget https://github.com/gflags/gflags/archive/refs/tags/v$version.tar.gz
tar xzf v$version.tar.gz
cd gflags-$version
mkdir build && cd build
cmake -D BUILD_SHARED_LIBS=ON ..
make -j$(nproc)
sudo make install

2.2) Build and package couchbase dependencies

  • Packaging benchmark v1.6.0
NAME_BENCHMARK=benchmark
VERSION_BENCHMARK=v1.6.0
BUILD_BENCHMARK=cb1
PACKAGE_BENCHMARK=$NAME_BENCHMARK-$DISTRO-s390x-$VERSION_BENCHMARK-$BUILD_BENCHMARK
cd $SOURCE_ROOT
git clone https://github.com/couchbasedeps/benchmark.git
cd benchmark
git checkout $VERSION_BENCHMARK
cmake  -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_INSTALL_LIBDIR=lib -D CMAKE_CXX_STANDARD=17 -D CMAKE_CXX_STANDARD_REQUIRED=ON -D BUILD_SHARED_LIBS=OFF -D BENCHMARK_ENABLE_TESTING=OFF -D BENCHMARK_ENABLE_GTEST_TESTS=OFF -D BENCHMARK_ENABLE_INSTALL=ON -D BENCHMARK_DOWNLOAD_DEPENDENCIES=OFF 
cmake --build . --target install
cmake -E remove_directory ./_build/lib/pkgconfig
cd _build
# Packaging build files
tar -czvf $PACKAGE_BENCHMARK.tgz *
MD5_BENCHMARK=($(md5sum $PACKAGE_BENCHMARK.tgz))
echo $MD5_BENCHMARK > $PACKAGE_BENCHMARK.md5
# Copy the package to destination
cp $PACKAGE_BENCHMARK.* $CACHE_DIRECTORY
  • Packaging boost 1.74.0
NAME_BOOST=boost
VERSION_BOOST=1.74.0
BUILD_BOOST=cb1
PACKAGE_BOOST=$NAME_BOOST-$DISTRO-s390x-$VERSION_BOOST-$BUILD_BOOST
sudo ldconfig /usr/local/lib64 /usr/local/lib
cd $SOURCE_ROOT
TOOLSET=gcc
URL=https://boostorg.jfrog.io/artifactory/main/release/$VERSION_BOOST/source/boost_1_74_0.tar.gz
curl -sSL $URL | tar xzf -
cd boost_1_74_0
sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/matrix.hpp
sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/storage.hpp

./bootstrap.sh --with-libraries=context,chrono,date_time,filesystem,program_options,regex,system,thread  --prefix=`pwd`/_build --libdir=`pwd`/_build/lib
options=( toolset=$TOOLSET variant=release link=static runtime-link=shared threading=multi --without-python address-model=64 cflags=-fno-omit-frame-pointer cxxflags=-fno-omit-frame-pointer cxxflags=-fPIC cxxflags=-std=c++17)
./b2 -j 4 ${options[@]} install
cd _build
# Packaging build files
tar -czvf $PACKAGE_BOOST.tgz *
MD5_BOOST=($(md5sum $PACKAGE_BOOST.tgz))
echo $MD5_BOOST > $PACKAGE_BOOST.md5
# Copy the package to destination
cp $PACKAGE_BOOST.* $CACHE_DIRECTORY
  • Packaging cbpy 7.5.0
NAME_CBPY=cbpy
VERSION_CBPY=7.5.0
BUILD_CBPY=cb1
PACKAGE_CBPY=$NAME_CBPY-linux-s390x-$VERSION_CBPY-$BUILD_CBPY
cd $SOURCE_ROOT
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-s390x.sh
chmod a+x Miniconda3-py39_4.12.0-Linux-s390x.sh
./Miniconda3-py39_4.12.0-Linux-s390x.sh -b -p ./cbpy
cd cbpy
./bin/pip3 install msgpack-python
./bin/pip3 install natsort
./bin/pip3 install pem
./bin/pip3 install pycryptodome
./bin/pip3 install python-snappy
./bin/pip3 install requests

# Packaging build files
tar -czvf $PACKAGE_CBPY.tgz bin lib share ssl
MD5_CBPY=($(md5sum $PACKAGE_CBPY.tgz))
echo $MD5_CBPY > $PACKAGE_CBPY.md5
# Copy the package to destination
cp $PACKAGE_CBPY.* $CACHE_DIRECTORY
  • Packaging curl 7.84.0
NAME_CURL=curl
VERSION_CURL=7.84.0
BUILD_CURL=5
PACKAGE_CURL=$NAME_CURL-linux-s390x-$VERSION_CURL-$BUILD_CURL

cd $SOURCE_ROOT
sudo ldconfig /usr/local/lib64 /usr/local/lib
git clone https://github.com/curl/curl.git
cd curl
git checkout curl-7_78_0
cmake -G "Unix Makefiles" .
autoreconf -fi
./configure --prefix=`pwd`/_build --libdir=`pwd`/_build/lib --without-ssl
make -j$(nproc)
make install
hash -r
cd _build 
cat <<EOF > CMakeLists.txt
FILE (COPY bin lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
SET_PROPERTY (GLOBAL APPEND PROPERTY CBDEPS_PREFIX_PATH "\${CMAKE_CURRENT_SOURCE_DIR}")
EOF
echo "$VERSION_CURL-7" > VERSION.txt
# Packaging build files
tar -czvf $PACKAGE_CURL.tgz *
MD5_CURL=($(md5sum $PACKAGE_CURL.tgz))
echo $MD5_CURL > $PACKAGE_CURL.md5
# Copy the package to destination
cp $PACKAGE_CURL.* $CACHE_DIRECTORY
  • Packaging double-conversion 3.0.0
NAME_DCONVERTION=double-conversion
VERSION_DCONVERTION=3.0.0
BUILD_DCONVERTION=cb4
PACKAGE_DCONVERTION=$NAME_DCONVERTION-$DISTRO-s390x-$VERSION_DCONVERTION-$BUILD_DCONVERTION
cd $SOURCE_ROOT
git clone https://github.com/google/double-conversion.git
cd double-conversion
git checkout v3.0.0
cmake -D BUILD_SHARED_LIBS=OFF -D BUILD_TESTING=OFF -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D CMAKE_INSTALL_LIBDIR=lib .
make -j$(nproc)
make install
cmake -E remove_directory `pwd`/_build/lib/cmake
cd _build
# Packaging build files
tar -czvf $PACKAGE_DCONVERTION.tgz *
MD5_DCONVERTION=($(md5sum $PACKAGE_DCONVERTION.tgz))
echo $MD5_DCONVERTION > $PACKAGE_DCONVERTION.md5
# Copy the package to destination
cp $PACKAGE_DCONVERTION.* $CACHE_DIRECTORY
  • Packaging erlang neo
NAME_ERLANG=erlang
VERSION_ERLANG=24.3.4.4
BUILD_ERLANG=4
PACKAGE_ERLANG=$NAME_ERLANG-linux-s390x-$VERSION_ERLANG-$BUILD_ERLANG

cd $SOURCE_ROOT
git clone https://github.com/couchbasedeps/erlang.git
cd erlang && git checkout couchbase-$VERSION_ERLANG
./configure --prefix=`pwd`/_build --libdir=`pwd`/_build/lib --enable-smp-support --disable-hipe --disable-fp-exceptions --without-javac --enable-m64-build CFLAGS="-fno-strict-aliasing -O3 -ggdb3"
make -j$(nproc)
make install
hash -r
cd _build
echo $VERSION_ERLANG > VERSION.txt
cat <<EOF > CMakeLists.txt
FILE (COPY bin lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EXECUTE_PROCESS (
    COMMAND "\${CMAKE_INSTALL_PREFIX}/lib/erlang/Install"
        -minimal "\${CMAKE_INSTALL_PREFIX}/lib/erlang"
)
EOF
# Packaging build files
tar -czvf $PACKAGE_ERLANG.tgz *
MD5_ERLANG=($(md5sum $PACKAGE_ERLANG.tgz))
echo $MD5_ERLANG > $PACKAGE_ERLANG.md5
# Copy the package to destination
cp $PACKAGE_ERLANG.* $CACHE_DIRECTORY
  • Packaging flatbuffers 1.10.0
NAME_FBUF=flatbuffers
VERSION_FBUF=1.10.0
BUILD_FBUF=cb5
PACKAGE_FBUF=$NAME_FBUF-$DISTRO-s390x-$VERSION_FBUF-$BUILD_FBUF

cd $SOURCE_ROOT
git clone https://github.com/google/flatbuffers
cd flatbuffers && git checkout v1.10.0
wget $PATCH_URL/flatbuffers.diff -P $SOURCE_ROOT/patch
git apply $SOURCE_ROOT/patch/flatbuffers.diff
curl https://github.com/google/flatbuffers/commit/2e865f4d4e67a9b628c137aab7da8140dd9339a4.patch | git apply -
cmake -G "Unix Makefiles" -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D CMAKE_BUILD_TYPE=Release -D FLATBUFFERS_BUILD_TESTS=OFF -D CMAKE_INSTALL_LIBDIR=lib
make -j $(nproc)
make install
hash -r
cd _build
touch CMakeLists.txt 
# Packaging build files
tar -czvf $PACKAGE_FBUF.tgz *
MD5_FBUF=($(md5sum $PACKAGE_FBUF.tgz))
echo $MD5_FBUF > $PACKAGE_FBUF.md5
# Copy the package to destination
cp $PACKAGE_FBUF.* $CACHE_DIRECTORY
  • Packaging fmt 7.1.3
NAME_FMT=fmt
VERSION_FMT=7.1.3
BUILD_FMT=cb2
PACKAGE_FMT=$NAME_FMT-$DISTRO-s390x-$VERSION_FMT-$BUILD_FMT
cd $SOURCE_ROOT
git clone https://github.com/fmtlib/fmt.git
cd fmt
git checkout $VERSION_FMT
cmake -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_CXX_VISIBILITY_PRESET=hidden -D CMAKE_POSITION_INDEPENDENT_CODE=ON -D CMAKE_INSTALL_LIBDIR=lib .
make -j$(nproc)
make install
cd _build
# Packaging build files
tar -czvf $PACKAGE_FMT.tgz *
MD5_FMT=($(md5sum $PACKAGE_FMT.tgz))
echo $MD5_FMT > $PACKAGE_FMT.md5
# Copy the package to destination
cp $PACKAGE_FMT.* $CACHE_DIRECTORY
  • Packaging snappy 1.1.8
NAME_SNAPPY=snappy
VERSION_SNAPPY=1.1.8
BUILD_SNAPPY=cb4
PACKAGE_SNAPPY=$NAME_SNAPPY-linux-s390x-$VERSION_SNAPPY-$BUILD_SNAPPY
cd $SOURCE_ROOT
wget https://github.com/google/snappy/archive/refs/tags/$VERSION_SNAPPY.tar.gz
tar -xzf $VERSION_SNAPPY.tar.gz
cd snappy-$VERSION_SNAPPY
cmake -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D SNAPPY_BUILD_TESTS=OFF -D BUILD_SHARED_LIBS=ON  -D CMAKE_INSTALL_LIBDIR=lib  .
make -j$(nproc)
make install
hash -r
cd _build
cat <<EOF > CMakeLists.txt
file(GLOB snappy_libs lib/*snappy*)
file(COPY \${snappy_libs} DESTINATION "\${CMAKE_INSTALL_PREFIX}/lib")
EOF
# Packaging build files
tar -czvf $PACKAGE_SNAPPY.tgz *
MD5_SNAPPY=($(md5sum $PACKAGE_SNAPPY.tgz))
echo $MD5_SNAPPY > $PACKAGE_SNAPPY.md5
# Copy the package to destination
cp $PACKAGE_SNAPPY.* $CACHE_DIRECTORY
  • Packaging OpenSSL 1.1.1o
NAME_OSSL=openssl
VERSION_OSSL=1.1.1t
BUILD_OSSL=1
PACKAGE_OSSL=$NAME_OSSL-linux-s390x-$VERSION_OSSL-$BUILD_OSSL

cd $SOURCE_ROOT
wget https://www.openssl.org/source/openssl-$VERSION_OSSL.tar.gz
tar -xzf openssl-$VERSION_OSSL.tar.gz
cd openssl-$VERSION_OSSL
./config --prefix=`pwd`/_build --openssldir=`pwd`/_build
make -j$(nproc)
make install
hash -r
cd _build
echo '1.1.1t-1' > VERSION.txt
cat <<EOF > CMakeLists.txt
FILE (COPY bin DESTINATION "\${CMAKE_INSTALL_PREFIX}")
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
SET_PROPERTY (GLOBAL APPEND PROPERTY CBDEPS_PREFIX_PATH "\${CMAKE_CURRENT_SOURCE_DIR}")
EOF
# Packaging build files
tar -czvf $PACKAGE_OSSL.tgz *
MD5_OSSL=($(md5sum $PACKAGE_OSSL.tgz))
echo $MD5_OSSL > $PACKAGE_OSSL.md5
# Copy the package to destination
cp $PACKAGE_OSSL.* $CACHE_DIRECTORY
  • Packaging libevent 2.1.11
NAME_LIBEVENT=libevent
VERSION_LIBEVENT=2.1.11
BUILD_LIBEVENT=cb9
PACKAGE_LIBEVENT=$NAME_LIBEVENT-$DISTRO-s390x-$VERSION_LIBEVENT-$BUILD_LIBEVENT

OPENSSL_BUILD=$SOURCE_ROOT/openssl-1.1.1t/_build

cd $SOURCE_ROOT
git clone https://github.com/libevent/libevent.git
cd libevent
git checkout release-2.1.11-stable

cmake -D CMAKE_INSTALL_PREFIX=`pwd`/_build \
      -D CMAKE_INSTALL_LIBDIR=lib \
      -D CMAKE_BUILD_TYPE=RelWithDebInfo \
      -D EVENT__DISABLE_BENCHMARK=ON \
      -D EVENT__DISABLE_REGRESS=ON \
      -D EVENT__DISABLE_SAMPLES=ON \
      -D OPENSSL_ROOT_DIR=$OPENSSL_BUILD .

make -j$(nproc)
make install
cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION \${CMAKE_INSTALL_PREFIX})
EOF
# Packaging build files
tar -czvf $PACKAGE_LIBEVENT.tgz *
MD5_LIBEVENT=($(md5sum $PACKAGE_LIBEVENT.tgz))
echo $MD5_LIBEVENT > $PACKAGE_LIBEVENT.md5
# Copy the package to destination
cp $PACKAGE_LIBEVENT.* $CACHE_DIRECTORY
  • Packaging glog 0.4.0
NAME_GLOG=glog
VERSION_GLOG=v0.4.0
BUILD_GLOG=cb1
PACKAGE_GLOG=$NAME_GLOG-$DISTRO-s390x-$VERSION_GLOG-$BUILD_GLOG

cd $SOURCE_ROOT
git clone https://github.com/google/glog.git
cd glog
git checkout $VERSION_GLOG
cmake -S . -B build -G "Unix Makefiles" -D BUILD_SHARED_LIBS=OFF -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_INSTALL_LIBDIR=lib
cmake --build build --target install
cd _build
# Packaging build files
tar -czvf $PACKAGE_GLOG.tgz *
MD5_GLOG=($(md5sum $PACKAGE_GLOG.tgz))
echo $MD5_GLOG > $PACKAGE_GLOG.md5
# Copy the package to destination
cp $PACKAGE_GLOG.* $CACHE_DIRECTORY
  • Packaging zstd 1.5.0
NAME_ZSTD=zstd-cpp
VERSION_ZSTD=1.5.0
BUILD_ZSTD=2
PACKAGE_ZSTD=$NAME_ZSTD-linux-s390x-$VERSION_ZSTD-$BUILD_ZSTD

cd $SOURCE_ROOT
wget https://github.com/facebook/zstd/archive/refs/tags/v1.5.0.tar.gz
tar -xzvf v1.5.0.tar.gz
cd zstd-1.5.0
make DESTDIR=`pwd`/_build install
cd _build
cp -r ./usr/local/* . && rm -drf usr bin share
echo '1.5.0-2' > VERSION.txt
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EOF
# Packaging build files
tar -czvf $PACKAGE_ZSTD.tgz *
MD5_ZSTD=($(md5sum $PACKAGE_ZSTD.tgz))
echo $MD5_ZSTD > $PACKAGE_ZSTD.md5
# Copy the package to destination
cp $PACKAGE_ZSTD.* $CACHE_DIRECTORY
  • Packaging folly 2020.09.07.00
NAME_FOLLY=folly
VERSION_FOLLY=v2020.09.07.00
BUILD_FOLLY=couchbase-cb2
PACKAGE_FOLLY=$NAME_FOLLY-$DISTRO-s390x-$VERSION_FOLLY-$BUILD_FOLLY

mkdir $SOURCE_ROOT/jemalloc-5.3.0 && cd $SOURCE_ROOT/jemalloc-5.3.0
git clone https://github.com/couchbasedeps/jemalloc.git .
git checkout 5.3.0
configure_args="--prefix=`pwd`/_build \
   --with-jemalloc-prefix=je_ \
   --disable-cache-oblivious \
   --disable-zone-allocator \
   --disable-initial-exec-tls \
   --disable-cxx \
   --enable-prof \
   --libdir=`pwd`/_build/lib"

./autogen.sh ${configure_args}
make -j$(nproc)
make install
JEMALLOC_BUILD=$SOURCE_ROOT/jemalloc-5.3.0/_build

FMT_BUILD=$SOURCE_ROOT/fmt/_build
GLOG_BUILD=$SOURCE_ROOT/glog/_build
BOOST_BUILD=$SOURCE_ROOT/boost_1_74_0/_build
DOUBLE_CONVERSION_BUILD=$SOURCE_ROOT/double-conversion/_build
LIBEVENT_BUILD=$SOURCE_ROOT/libevent/_build
OPENSSL_BUILD=$SOURCE_ROOT/openssl-1.1.1t/_build
SNAPPY_BUILD=$SOURCE_ROOT/snappy-1.1.8/_build
ZSTD_BUILD=$SOURCE_ROOT/zstd-1.5.0/_build

LIBRARIES="$JEMALLOC_BUILD/lib;$FMT_BUILD/lib;$GLOG_BUILD/lib;$BOOST_BUILD/lib;$DOUBLE_CONVERSION_BUILD/lib;$LIBEVENT_BUILD/lib;$OPENSSL_BUILD/lib;$SNAPPY_BUILD/lib;$ZSTD_BUILD/lib;"
INCLUDES="$JEMALLOC_BUILD/include;$FMT_BUILD/include;$GLOG_BUILD/include;$BOOST_BUILD/include;$DOUBLE_CONVERSION_BUILD/include;$LIBEVENT_BUILD/include;$OPENSSL_BUILD/include;$SNAPPY_BUILD/include;$ZSTD_BUILD/include"

cd $SOURCE_ROOT
git clone https://github.com/facebook/folly
cd folly
git checkout $VERSION_FOLLY
if [ $ID$VERSION_ID == "ubuntu22.04" ] || [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
    git config --global user.email "[email protected]"
    git config --global user.name  "your_name"
    git cherry-pick 7df2d7e5098119c1562422ac9571e70f032adb50
fi
if [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
  sed -i '20i #include <limits>' ./folly/TimeoutQueue.cpp
fi
wget $PATCH_URL/folly.diff -P $SOURCE_ROOT/patch
git apply $SOURCE_ROOT/patch/folly.diff
sudo ldconfig /usr/local/lib64 /usr/local/lib

cmake \
    -D CMAKE_INCLUDE_PATH=$INCLUDES \
    -D CMAKE_LIBRARY_PATH=$LIBRARIES \
    -D CMAKE_CXX_FLAGS="-fPIC -fvisibility=hidden" \
    -D CMAKE_BUILD_TYPE=RelWithDebInfo \
    -D CMAKE_INSTALL_PREFIX=`pwd`/_build \
    -D BUILD_SHARED_LIBS:STRING=OFF \
    -D Boost_INCLUDE_DIR=$BOOST_BUILD/include \
    -D Boost_ADDITIONAL_VERSIONS=1.74 \
    -D Boost_USE_STATIC_LIBS=ON \
    -D Boost_NO_SYSTEM_PATHS=ON \
    -D Boost_NO_BOOST_CMAKE=ON \
    -D BOOST_ROOT=$BOOST_BUILD \
    -D CMAKE_PREFIX_PATH=$FMT_BUILD \
    -D CMAKE_DISABLE_FIND_PACKAGE_ZLIB=TRUE .

make -j$(nproc)
make install
cd _build
# Packaging build files
tar -czvf $PACKAGE_FOLLY.tgz *
MD5_FOLLY=($(md5sum $PACKAGE_FOLLY.tgz))
echo $MD5_FOLLY > $PACKAGE_FOLLY.md5
# Copy the package to destination
cp $PACKAGE_FOLLY.* $CACHE_DIRECTORY
  • Packaging googletest 1.11.0
NAME_GTEST=googletest
VERSION_GTEST=1.11.0
BUILD_GTEST=cb4
PACKAGE_GTEST=$NAME_GTEST-linux-s390x-$VERSION_GTEST-$BUILD_GTEST

cd $SOURCE_ROOT
git clone https://github.com/google/googletest.git
cd googletest
git checkout release-$VERSION_GTEST
cmake -D CMAKE_INSTALL_PREFIX=`pwd`/_build \
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D CMAKE_INSTALL_LIBDIR=lib \
-D CMAKE_CXX_STANDARD=17 \
-D CMAKE_CXX_STANDARD_REQUIRED=ON \
-D BUILD_SHARED_LIBS=OFF \
-D gtest_force_shared_crt=ON .

make -j$(nproc)
make install
cd _build
rm -drf ./lib/pkgconfig
# Packaging build files
tar -czvf $PACKAGE_GTEST.tgz *
MD5_GTEST=($(md5sum $PACKAGE_GTEST.tgz))
echo $MD5_GTEST > $PACKAGE_GTEST.md5
# Copy the package to destination
cp $PACKAGE_GTEST.* $CACHE_DIRECTORY
  • Packaging zlib 1.2.13
NAME_ZLIB=zlib
VERSION_ZLIB=1.2.13
BUILD_ZLIB=1
PACKAGE_ZLIB=$NAME_ZLIB-linux-s390x-$VERSION_ZLIB-$BUILD_ZLIB

cd $SOURCE_ROOT
git clone https://github.com/madler/zlib.git
cd zlib
git checkout v1.2.13 # v1.2.12 doesn't work on some distros (rhel8)

./configure --prefix=`pwd`/_build --64
make -j$(nproc)
make test # tests fail on and rhel 8.6 . I used zlib 1.2.11 on those distros!
make install
cd _build
echo '1.2.13-1' > VERSION.txt
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
SET_PROPERTY (GLOBAL APPEND PROPERTY CBDEPS_PREFIX_PATH "\${CMAKE_CURRENT_SOURCE_DIR}")
EOF
# Packaging build files
tar -czvf $PACKAGE_ZLIB.tgz *
MD5_ZLIB=($(md5sum $PACKAGE_ZLIB.tgz))
echo $MD5_ZLIB > $PACKAGE_ZLIB.md5
# Copy the package to destination
cp $PACKAGE_ZLIB.* $CACHE_DIRECTORY
  • Packaging GRPC 1.28.1
NAME_GRPC=grpc
VERSION_GRPC=1.28.1
BUILD_GRPC=cb3
PACKAGE_GRPC=$NAME_GRPC-$DISTRO-s390x-$VERSION_GRPC-$BUILD_GRPC

cd $SOURCE_ROOT
git clone -b v1.28.1 https://github.com/grpc/grpc
cd grpc/
git submodule update --init
if [ $ID$VERSION_ID == "ubuntu22.04" ] || [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
  sed -i '59i #undef SIGSTKSZ' third_party/abseil-cpp/absl/debugging/failure_signal_handler.cc
  sed -i '60i #define SIGSTKSZ 16384' third_party/abseil-cpp/absl/debugging/failure_signal_handler.cc
fi
if [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
  sed -i '44i #include <limits>' third_party/abseil-cpp/absl/synchronization/internal/graphcycles.cc
  sed -i '20i #include <limits>' third_party/abseil-cpp/absl/flags/usage_config.cc
fi
BUILD_DIR=`pwd`/_build
# Point to zlib build file.
ZLIB_BUILD_DIR=$SOURCE_ROOT/zlib/_build
# Point to OpenSSL build file.
OPENSSL_BUILD_DIR=$SOURCE_ROOT/openssl-1.1.1o/_build
(
cd third_party/protobuf/cmake
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_INSTALL_PREFIX="$BUILD_DIR" \
  -Dprotobuf_BUILD_TESTS=OFF \
  -D CMAKE_PREFIX_PATH="$ZLIB_BUILD_DIR" .
make -j$(nproc) install
)
(
cd third_party/cares/cares
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_INSTALL_PREFIX="$BUILD_DIR" \
  -DCARES_STATIC=ON -DCARES_STATIC_PIC=ON -DCARES_SHARED=OFF .
make -j$(nproc) install
)
cmake -D CMAKE_BUILD_TYPE=RelWithDebInfo \
  -D CMAKE_INSTALL_PREFIX=$BUILD_DIR \
  -D CMAKE_PREFIX_PATH="$ZLIB_BUILD_DIR;$OPENSSL_BUILD_DIR;$BUILD_DIR" \
  -DgRPC_INSTALL=ON \
  -DgRPC_BUILD_TESTS=OFF \
  -DgRPC_PROTOBUF_PROVIDER=package \
  -DgRPC_ZLIB_PROVIDER=package \
  -DgRPC_CARES_PROVIDER=package \
  -DgRPC_SSL_PROVIDER=package \
  -D CMAKE_INSTALL_LIBDIR=lib .
make -j$(nproc) install
cd _build
# Packaging build files
tar -czvf $PACKAGE_GRPC.tgz *
MD5_GRPC=($(md5sum $PACKAGE_GRPC.tgz))
echo $MD5_GRPC > $PACKAGE_GRPC.md5
# Copy the package to destination
cp $PACKAGE_GRPC.* $CACHE_DIRECTORY
  • Packaging Jemalloc 5.2.1
NAME_JEMALLOC=jemalloc
VERSION_JEMALLOC=5.2.1
BUILD_JEMALLOC=cb9
PACKAGE_JEMALLOC=$NAME_JEMALLOC-$DISTRO-s390x-$VERSION_JEMALLOC-$BUILD_JEMALLOC

cd $SOURCE_ROOT
git clone https://github.com/couchbasedeps/jemalloc.git
cd jemalloc && git checkout $VERSION_JEMALLOC
configure_args="--prefix=`pwd`/_build \
   --with-jemalloc-prefix=je_ \
   --disable-cache-oblivious \
   --disable-zone-allocator \
   --disable-initial-exec-tls \
   --disable-cxx \
   --enable-prof \
   --libdir=`pwd`/_build/lib"

./autogen.sh ${configure_args}
make -j8 build_lib_shared
make -j8 check
make install_lib_shared install_include install_bin

# just if you need to debug.
CFLAGS=-Og ./autogen.sh ${configure_args} --enable-debug --with-install-suffix=d
make -j8 build_lib_shared
make -j8 check
make install_lib_shared

cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY bin/jeprof DESTINATION "\${CMAKE_INSTALL_PREFIX}/bin")
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
SET_PROPERTY (GLOBAL APPEND PROPERTY CBDEPS_PREFIX_PATH "\${CMAKE_CURRENT_SOURCE_DIR}")
EOF
# Packaging build files
tar -czvf $PACKAGE_JEMALLOC.tgz *
MD5_JEMALLOC=($(md5sum $PACKAGE_JEMALLOC.tgz))
echo $MD5_JEMALLOC > $PACKAGE_JEMALLOC.md5
# Copy the package to destination
cp $PACKAGE_JEMALLOC.* $CACHE_DIRECTORY
  • Packaging JSON hpp 3.9.0
NAME_JSON=json
VERSION_JSON=3.9.0
BUILD_JSON=cb1
PACKAGE_JSON=$NAME_JSON-$DISTRO-s390x-$VERSION_JSON-$BUILD_JSON

cd $SOURCE_ROOT
git clone https://github.com/nlohmann/json
cd $SOURCE_ROOT/json
git checkout v3.9.0
cmake -D CMAKE_INSTALL_PREFIX=`pwd`/_build -D CMAKE_BUILD_TYPE=RelWithDebInfo -D JSON_BuildTests=OFF -D JSON_Install=ON -D JSON_MultipleHeaders=ON -D JSON_SystemInclude=ON -D CMAKE_INSTALL_LIBDIR=lib
cmake --build . --target install
cd _build
# Packaging build files
tar -czvf $PACKAGE_JSON.tgz *
MD5_JSON=($(md5sum $PACKAGE_JSON.tgz))
echo $MD5_JSON > $PACKAGE_JSON.md5
# Copy the package to destination
cp $PACKAGE_JSON.* $CACHE_DIRECTORY
  • Packaging liburing 0.6
NAME_LIBURING=liburing
VERSION_LIBURING=0.6
BUILD_LIBURING=2
PACKAGE_LIBURING=$NAME_LIBURING-$DISTRO-s390x-$VERSION_LIBURING-$BUILD_LIBURING

cd $SOURCE_ROOT
git clone https://github.com/axboe/liburing.git
cd liburing
git checkout liburing-$VERSION_LIBURING
if [ $ID$VERSION_ID == "ubuntu22.04" ] || [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
    git cherry-pick 8aac320ae8445c5434ab3be1761414a5247e5d42
fi

sed -i 's/|\sMAP_HUGE_2MB\s//g' ./test/io_uring_register.c

./configure --prefix=`pwd`/_build --libdir=`pwd`/_build/lib
make -j$(nproc)
make install
cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EOF
# Packaging build files
tar -czvf $PACKAGE_LIBURING.tgz *
MD5_LIBURING=($(md5sum $PACKAGE_LIBURING.tgz))
echo $MD5_LIBURING > $PACKAGE_LIBURING.md5
# Copy the package to destination
cp $PACKAGE_LIBURING.* $CACHE_DIRECTORY
  • Packaging libuv 1.20.3
NAME_LIBUV=libuv
VERSION_LIBUV=1.20.3
BUILD_LIBUV=22
PACKAGE_LIBUV=$NAME_LIBUV-$DISTRO-s390x-$VERSION_LIBUV-$BUILD_LIBUV

cd $SOURCE_ROOT
git clone https://github.com/couchbasedeps/libuv
cd libuv
git checkout v1.20.3
./autogen.sh
./configure  --disable-silent-rules --prefix=`pwd`/_build --libdir=`pwd`/_build/lib
make -j$(nproc)
make install
cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EOF
# Packaging build files
tar -czvf $PACKAGE_LIBUV.tgz *
MD5_LIBUV=($(md5sum $PACKAGE_LIBUV.tgz))
echo $MD5_LIBUV > $PACKAGE_LIBUV.md5
# Copy the package to destination
cp $PACKAGE_LIBUV.* $CACHE_DIRECTORY
  • Packaging lz4 1.9.2
NAME_LZ4=lz4
VERSION_LZ4=1.9.2
BUILD_LZ4=cb2
PACKAGE_LZ4=$NAME_LZ4-linux-s390x-$VERSION_LZ4-$BUILD_LZ4

cd $SOURCE_ROOT
wget https://github.com/lz4/lz4/archive/refs/tags/v1.9.2.tar.gz
tar -xzf v1.9.2.tar.gz
cd lz4-1.9.2
make -j$(nproc)
make DESTDIR=`pwd`/_build install
hash -r
cd _build
cp -r ./usr/local/* . && rm -drf usr
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EOF
# Packaging build files
tar -czvf $PACKAGE_LZ4.tgz *
MD5_LZ4=($(md5sum $PACKAGE_LZ4.tgz))
echo $MD5_LZ4 > $PACKAGE_LZ4.md5
# Copy the package to destination
cp $PACKAGE_LZ4.* $CACHE_DIRECTORY
  • Packaging numactl 2.0.14
NAME_NUMCTL=numactl
VERSION_NUMCTL=2.0.11
BUILD_NUMCTL=cb3
PACKAGE_NUMCTL=$NAME_NUMCTL-$DISTRO-s390x-$VERSION_NUMCTL-$BUILD_NUMCTL
cd $SOURCE_ROOT
git clone https://github.com/numactl/numactl.git
cd numactl

# v2.0.11 doesn't work on some distroes
git checkout v2.0.14
./autogen.sh
./configure --prefix=`pwd`/_build --libdir=`pwd`/_build/lib
make -j$(nproc)
make install
cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
SET_PROPERTY (GLOBAL APPEND PROPERTY CBDEPS_PREFIX_PATH "\${CMAKE_CURRENT_SOURCE_DIR}")
EOF
rm -dfr share bin 
# Packaging build files
tar -czvf $PACKAGE_NUMCTL.tgz *
MD5_NUMCTL=($(md5sum $PACKAGE_NUMCTL.tgz))
echo $MD5_NUMCTL > $PACKAGE_NUMCTL.md5
# Copy the package to destination
cp $PACKAGE_NUMCTL.* $CACHE_DIRECTORY
  • Packaging pcre 8.44
NAME_PCRE=pcre
VERSION_PCRE=8.44
BUILD_PCRE=cb1
PACKAGE_PCRE=$NAME_PCRE-linux-s390x-$VERSION_PCRE-$BUILD_PCRE

cd $SOURCE_ROOT
wget https://sourceforge.net/projects/pcre/files/pcre/$VERSION_PCRE/pcre-$VERSION_PCRE.tar.gz
tar -xzf pcre-$VERSION_PCRE.tar.gz
cd pcre-$VERSION_PCRE
./configure --prefix=`pwd`/_build --libdir=`pwd`/_build/lib
make -j$(nproc)
make install
cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY bin lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
SET_PROPERTY (GLOBAL APPEND PROPERTY CBDEPS_PREFIX_PATH "\${CMAKE_CURRENT_SOURCE_DIR}")
EOF
# Packaging build files
tar -czvf $PACKAGE_PCRE.tgz *
MD5_PCRE=($(md5sum $PACKAGE_PCRE.tgz))
echo $MD5_PCRE > $PACKAGE_PCRE.md5
# Copy the package to destination
cp $PACKAGE_PCRE.* $CACHE_DIRECTORY
  • Packaging prometheus 2.33.3
cd $SOURCE_ROOT
NAME_PROM=prometheus
VERSION_PROM=2.33.3
BUILD_PROM=3
PACKAGE_PROM=$NAME_PROM-linux-s390x-$VERSION_PROM-$BUILD_PROM
cd $SOURCE_ROOT
export GOPATH=$(go env GOPATH)
wget https://nodejs.org/dist/v16.20.1/node-v16.20.1-linux-s390x.tar.xz
tar xf node-v16.20.1-linux-s390x.tar.xz
export PATH=$SOURCE_ROOT/node-v16.20.1-linux-s390x/bin:$PATH
npm install -g yarn
mkdir -p $GOPATH/src/github.com
cd $GOPATH/src/github.com
git clone https://github.com/couchbasedeps/prometheus.git
cd prometheus
git checkout couchbase-v2.33.3
make build
mkdir -p _build/bin && cp prometheus _build/bin && cd _build
echo '2.33.3-3' > VERSION.txt
cat <<EOF > CMakeLists.txt
FILE (COPY bin DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EOF
# Packaging build files
tar -czvf $PACKAGE_PROM.tgz *
MD5_PROM=($(md5sum $PACKAGE_PROM.tgz))
echo $MD5_PROM > $PACKAGE_PROM.md5
# Copy the package to destination
cp $PACKAGE_PROM.* $CACHE_DIRECTORY
  • Packaging prometheus-cpp 0.10.0
NAME_PROMCPP=prometheus-cpp
VERSION_PROMCPP=v0.10.0
BUILD_PROMCPP=couchbase-cb2
PACKAGE_PROMCPP=$NAME_PROMCPP-$DISTRO-s390x-$VERSION_PROMCPP-$BUILD_PROMCPP

# Point to zlib build file.
ZLIB_BUILD_DIR=$SOURCE_ROOT/zlib/_build

cd $SOURCE_ROOT
git clone https://github.com/jupp0r/prometheus-cpp.git
cd prometheus-cpp
git checkout $VERSION_PROMCPP
git submodule init
git submodule update
if [ $ID$VERSION_ID == "ubuntu22.04" ] || [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
    sed -i '8i #include <limits>' core/src/histogram.cc
fi

mkdir _build
cd _build

cmake -DBUILD_SHARED_LIBS=OFF \
-D ENABLE_PUSH=OFF \
-D CMAKE_POSITION_INDEPENDENT_CODE=ON \
-D CMAKE_INSTALL_PREFIX=`pwd`/_build \
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D ZLIB_ROOT=$ZLIB_BUILD_DIR \
-D CMAKE_INSTALL_LIBDIR=lib ..

make -j$(nproc)
make install
cd _build
# Packaging build files
tar -czvf $PACKAGE_PROMCPP.tgz *
MD5_PROMCPP=($(md5sum $PACKAGE_PROMCPP.tgz))
echo $MD5_PROMCPP > $PACKAGE_PROMCPP.md5
# Copy the package to destination
cp $PACKAGE_PROMCPP.* $CACHE_DIRECTORY
  • Packaging protoc-gen-go 1.2.5
NAME_PROTOC=protoc-gen-go
VERSION_PROTOC=1.2.5
BUILD_PROTOC=4
PACKAGE_PROTOC=$NAME_PROTOC-$DISTRO-s390x-$VERSION_PROTOC-$BUILD_PROTOC

mkdir -p $SOURCE_ROOT/protoc-gen-go/bin && cd $SOURCE_ROOT/protoc-gen-go
go install github.com/golang/protobuf/[email protected]
cp $GOPATH/bin/protoc-gen-go ./bin
# Packaging build files
tar -czvf $PACKAGE_PROTOC.tgz *
MD5_PROTOC=($(md5sum $PACKAGE_PROTOC.tgz))
echo $MD5_PROTOC > $PACKAGE_PROTOC.md5
# Copy the package to destination
cp $PACKAGE_PROTOC.* $CACHE_DIRECTORY
  • Packaging RocksDB 5.18.3
NAME_ROCK=rocksdb
VERSION_ROCK=5.18.3
BUILD_ROCK=cb6
PACKAGE_ROCK=$NAME_ROCK-$DISTRO-s390x-$VERSION_ROCK-$BUILD_ROCK

cd $SOURCE_ROOT
git clone https://github.com/facebook/rocksdb.git
cd rocksdb/
git checkout v5.18.3
if [ $ID$VERSION_ID == "rhel9.0" ] || [ $ID$VERSION_ID == "rhel9.2" ]; then
  CXXFLAGS='-Wno-error=deprecated-copy -Wno-error=pessimizing-move -Wno-error=redundant-move -Wno-error=range-loop-construct'  make -j$(nproc) shared_lib
else
  CXXFLAGS='-Wno-error=deprecated-copy -Wno-error=pessimizing-move -Wno-error=redundant-move'  make -j$(nproc) shared_lib
fi
make install-shared INSTALL_PATH=`pwd`/_build
cd _build
cat <<EOF > CMakeLists.txt
FILE (COPY lib DESTINATION "\${CMAKE_INSTALL_PREFIX}")
EOF
# Packaging build files
tar -czvf $PACKAGE_ROCK.tgz *
MD5_ROCK=($(md5sum $PACKAGE_ROCK.tgz))
echo $MD5_ROCK > $PACKAGE_ROCK.md5
# Copy the package to destination
cp $PACKAGE_ROCK.* $CACHE_DIRECTORY
  • Packaging spdlog v1.8.5
NAME_SPDLOG=spdlog
VERSION_SPDLOG=v1.8.5
BUILD_SPDLOG=cb3
PACKAGE_SPDLOG=$NAME_SPDLOG-$DISTRO-s390x-$VERSION_SPDLOG-$BUILD_SPDLOG

cd $SOURCE_ROOT
# Point to fmt build file.
FMT_BUILD_DIR=$SOURCE_ROOT/fmt/_build
git clone https://github.com/gabime/spdlog.git
cd spdlog
git checkout $VERSION_SPDLOG
wget https://raw.githubusercontent.com/couchbase/tlm/v7.2.0/deps/packages/spdlog/custom_level_names.patch
git apply custom_level_names.patch
wget https://raw.githubusercontent.com/couchbase/tlm/v7.2.0/deps/packages/spdlog/relocatable_export_package.patch
git apply relocatable_export_package.patch
mkdir build && cd build
cmake  \
    -D CMAKE_CXX_VISIBILITY_PRESET=hidden \
    -D CMAKE_POSITION_INDEPENDENT_CODE=ON \
    -D SPDLOG_BUILD_EXAMPLE=OFF \
    -D SPDLOG_FMT_EXTERNAL=ON \
    -D CMAKE_PREFIX_PATH=$FMT_BUILD_DIR \
    -D CMAKE_BUILD_TYPE=RelWithDebInfo \
    -D CMAKE_INSTALL_PREFIX=`pwd`/_build \
    -D CMAKE_INSTALL_LIBDIR=lib ..

make -j$(nproc)
make install
cd _build
# Packaging build files
tar -czvf $PACKAGE_SPDLOG.tgz *
MD5_SPDLOG=($(md5sum $PACKAGE_SPDLOG.tgz))
echo $MD5_SPDLOG > $PACKAGE_SPDLOG.md5
# Copy the package to destination
cp $PACKAGE_SPDLOG.* $CACHE_DIRECTORY
  • Packaging v8 10.7.21
NAME_V8=v8
VERSION_V8=10.7.21
BUILD_V8=2
PACKAGE_V8=$NAME_V8-linux-s390x-$VERSION_V8-$BUILD_V8

cd $SOURCE_ROOT
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
cd $SOURCE_ROOT/depot_tools
git checkout a0382d39be0d7bf0f0766633185f20dcdd32a459
export PATH=$PATH:$SOURCE_ROOT/depot_tools/
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"
export DEPOT_TOOLS_UPDATE=0

cd $SOURCE_ROOT
git clone https://gn.googlesource.com/gn
cd gn
git checkout fae280e
sed -i -e 's/-Wl,--icf=all//g' ./build/gen.py
sed -i -e 's/-lpthread/-pthread/g' ./build/gen.py
python build/gen.py
ninja -C out
export PATH=$SOURCE_ROOT/gn/out:$PATH

cd $SOURCE_ROOT
sudo ldconfig /usr/local/lib64 /usr/local/lib

cat > .gclient <<EOF
solutions = [
  {
    "url": "https://chromium.googlesource.com/v8/[email protected]",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
  },
];
EOF
gclient sync

cd v8
wget $PATCH_URL/v8.diff -P $SOURCE_ROOT/patch
git apply $SOURCE_ROOT/patch/v8.diff

V8_ARGS='is_component_build=true target_cpu="s390x" v8_target_cpu="s390x" use_goma=false goma_dir="None" v8_enable_backtrace=true treat_warnings_as_errors=false is_clang=false use_custom_libcxx_for_host=false use_custom_libcxx=false v8_use_external_startup_data=false  use_sysroot=false use_gold=false linux_use_bundled_binutils=false    v8_enable_pointer_compression=false'
# build release
gn gen out/s390x.release --args="$V8_ARGS is_debug=false"
LD_LIBRARY_PATH=$SOURCE_ROOT/v8/out/s390x.release ninja -C $SOURCE_ROOT/v8/out/s390x.release -j$(nproc)
# build debug (only if you need to debug!)
gn gen out/s390x.debug --args="$V8_ARGS is_debug=true"
LD_LIBRARY_PATH=$SOURCE_ROOT/v8/out/s390x.debug ninja -C $SOURCE_ROOT/v8/out/s390x.debug -j$(nproc)

INSTALL_DIR=`pwd`/_build
mkdir -p \
    $INSTALL_DIR/lib/Release \
    $INSTALL_DIR/lib/Debug \
    $INSTALL_DIR/include/libplatform \
    $INSTALL_DIR/include/cppgc \
    $INSTALL_DIR/include/unicode
(
    cd out/s390x.release
    cp -avi libv8*.* $INSTALL_DIR/lib/Release
    cp -avi libchrome*.* $INSTALL_DIR/lib/Release
    cp -avi libcppgc*.* $INSTALL_DIR/lib/Release
    cp -avi libicu*.* $INSTALL_DIR/lib/Release
    cp -avi icu*.* $INSTALL_DIR/lib/Release
    rm -f $INSTALL_DIR/lib/Release/*.TOC
    rm -f $INSTALL_DIR/lib/Release/*for_testing*
    rm -f $INSTALL_DIR/lib/Release/*debug_helper*
)
(
    cd include
    cp -avi v8*.h $INSTALL_DIR/include
    cp -avi libplatform/[a-z]*.h $INSTALL_DIR/include/libplatform
    cp -avi cppgc/* $INSTALL_DIR/include/cppgc
)
(
    cd third_party/icu/source/common/unicode
    cp -avi *.h $INSTALL_DIR/include/unicode
)
(
    cd third_party/icu/source/io/unicode
    cp -avi *.h $INSTALL_DIR/include/unicode
)
(
    cd third_party/icu/source/i18n/unicode
    cp -avi *.h $INSTALL_DIR/include/unicode
)
(
    cd third_party/icu/source/extra/uconv/unicode
    cp -avi *.h $INSTALL_DIR/include/unicode
)
# do this only if you need to debug
(
    cd out/s390x.debug
    cp -avi libv8*.* $INSTALL_DIR/lib/Debug
    cp -avi libchrome*.* $INSTALL_DIR/lib/Debug
    cp -avi libcppgc*.* $INSTALL_DIR/lib/Debug
    cp -avi libicu*.* $INSTALL_DIR/lib/Debug
    cp -avi icu*.* $INSTALL_DIR/lib/Debug
    rm -f $INSTALL_DIR/lib/Debug/*.TOC
    rm -f $INSTALL_DIR/lib/Debug/*for_testing*
    rm -f $INSTALL_DIR/lib/Debug/*debug_helper*
)


cd _build
cat <<EOF > CMakeLists.txt
FILE(MAKE_DIRECTORY \${CMAKE_INSTALL_PREFIX}/bin)

# Determine which directory to copy libs from
IF (CMAKE_BUILD_TYPE STREQUAL "Debug" AND
    IS_DIRECTORY "\${CMAKE_CURRENT_SOURCE_DIR}/lib/Debug")
    SET (LIB_DIR Debug)
ELSE()
    SET (LIB_DIR Release)
ENDIF ()

FILE (COPY lib/\${LIB_DIR}/
      DESTINATION "\${CMAKE_INSTALL_PREFIX}/lib"
      PATTERN .dat EXCLUDE)
FILE (COPY lib/\${LIB_DIR}/icudtb.dat DESTINATION "\${CMAKE_INSTALL_PREFIX}/bin")
FILE (COPY lib/\${LIB_DIR}/icudtl_extra.dat DESTINATION "\${CMAKE_INSTALL_PREFIX}/bin")
EOF
# Packaging build files
tar -czvf $PACKAGE_V8.tgz *
MD5_V8=($(md5sum $PACKAGE_V8.tgz))
echo $MD5_V8 > $PACKAGE_V8.md5
# Copy the package to destination
cp $PACKAGE_V8.* $CACHE_DIRECTORY
  • Packaging icu
NAME_ICU=icu
printf -- "Start building %s .\n" "$NAME_ICU"
if [ -d  $SOURCE_ROOT/$NAME_ICU ]; then
    printf -- "The file already exists. Nothing to do. \n"
    return 0
fi

cd $SOURCE_ROOT
git clone --recursive -b release-71-1 https://github.com/unicode-org/icu.git
cd icu/icu4c/source
./runConfigureICU Linux
make -j8

printf -- "Finished building icu .\n"

Step 3: Build Couchbase

3.1) Download the Repo tool

Download the Repo tool and ensure that it has execute permissions

cd $SOURCE_ROOT
curl https://storage.googleapis.com/git-repo-downloads/repo > repo
chmod a+x repo
sudo mv repo /usr/bin

3.2) Setup Couchbase source

  • Clone Couchbase

    Set up Git user

    git config --global user.email "[email protected]"
git config --global user.name  "your_name"

    Clone the Couchbase 7.2.0 release using the the manifest file via Repo tool with the init and sync commands

    mkdir -p $SOURCE_ROOT/couchbase
cd $SOURCE_ROOT/couchbase
repo init -u https://github.com/couchbase/manifest -m released/couchbase-server/7.2.0.xml
repo sync

  • Patch Files

    cd $SOURCE_ROOT/couchbase/tlm
wget "${PATCH_URL}"/tlm.diff
git apply tlm.diff

cd $SOURCE_ROOT/couchbase/couchstore
wget "${PATCH_URL}"/couchstore.diff
git apply couchstore.diff

cd $SOURCE_ROOT/couchbase/kv_engine
wget "${PATCH_URL}"/kv_engine.diff
git apply kv_engine.diff

cd $SOURCE_ROOT/couchbase/platform
wget "${PATCH_URL}"/platform.diff
git apply platform.diff

# replace icudtl with icudtb in couchdb with this command 
sed -i 's/icudtl/icudtb/g' $SOURCE_ROOT/couchbase/couchdb/src/mapreduce/CMakeLists.txt

# add s390x to supported arches by replacing aarch64
sed -i '272s/aarch64/s390x/g' $SOURCE_ROOT/couchbase/forestdb/src/arch.h
# Only on ubuntu22.04 run this after applying tlm diff
sed -i 's/ubuntu18.04/ubuntu22.04/g' $SOURCE_ROOT/couchbase/tlm/deps/manifest.cmake
# Only on rhel8 run this after applying tlm diff
sed -i 's/centos7/rhel8/g' $SOURCE_ROOT/couchbase/tlm/deps/manifest.cmake
# Only on rhel9 run this after applying tlm diff
sed -i 's/centos7/rhel9/g' $SOURCE_ROOT/couchbase/tlm/deps/manifest.cmake

3.3) Build Couchbase

cd $SOURCE_ROOT/couchbase/
sudo ldconfig /usr/local/lib64 /usr/local/lib
make -j$(nproc) build/Makefile
sudo LD_LIBRARY_PATH=$SOURCE_ROOT/couchbase/install/lib CC=$CC CXX=$CXX make -j$(nproc) everything

Step 4: Run test cases (Optional)

sudo mkdir $SOURCE_ROOT/couchbase/install/lib/icu_orig
sudo mv $SOURCE_ROOT/couchbase/install/lib/libicu* $SOURCE_ROOT/couchbase/install/lib/icu_orig/
sudo cp $SOURCE_ROOT/icu/icu4c/source/lib/libicu* $SOURCE_ROOT/couchbase/install/lib/
# Only for distos other than rhel8 where linkig of icu gets issue
sudo ldconfig $SOURCE_ROOT/couchbase/install/lib
cd $SOURCE_ROOT/couchbase/build/
sudo ctest --timeout 1000

Note:

  • The following tests failed on both s390x and x86_64:
ep-engine_ep_unit_tests.WarmupDiskTest
ep-engine_ep_unit_tests.MutationLogTest
ep_testsuite.value_eviction.comp_passive
ep_testsuite.value_eviction.comp_active
ep_testsuite.full_eviction.comp_passive
ep_testsuite.full_eviction.comp_active
ep_testsuite_basic.value_eviction.comp_passive
ep_testsuite_basic.value_eviction.comp_active
ep_testsuite_basic.full_eviction.comp_passive
ep_testsuite_basic.full_eviction.comp_active
memcached_testapp.default.TransportProtocols/AuditTest
memcached_testapp.ep.TransportProtocols/AuditTest

  • ep-engine_ep_unit_tests.BloomFilterTest failed as there were less false positives on s390x than the expected number on x86_64.

  • The following test cases are flaky and you may need to run each a few times to pass:

memcached_testapp.ep.TransportProtocols/XattrTest
couchdb-couch_dcp-05-couch-dcp-client
  • If you install couchbase in a non default path, ns_test would fail.

Step 5: Start the server

  • Use the below command to start Couchbase Server
sudo $SOURCE_ROOT/couchbase/install/bin/couchbase-server --start

You can view the Couchbase UI at http://<hostname>:8091. Follow this tutorial to get started with couchbase. Please note that you can not load sample buckets because the cbimport tool is not supported on s390x. Check this link for more details.

References:

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

⚠️ **GitHub.com Fallback** ⚠️