Building Elasticsearch - linux-on-ibm-z/docs GitHub Wiki
Building Elasticsearch
Below versions of Elasticsearch are available in respective distributions at the time of creation of these build instructions.
The instructions provided below specify the steps to build Elasticsearch 8.13.3 on Linux on IBM Z for following distributions:
- RHEL (7.8, 7.9, 8.8, 8.9, 9.2, 9.3)
- SLES (12 SP5, 15 SP5)
- Ubuntu (20.04, 22.04)
Prerequisites:
-
Docker packages are provided for RHEL, SLES and Ubuntu in their respective repositories. More information about Docker CE can be found here
-
Ensure the current user belongs to group
docker
:Use the below command to add group
docker
if it does not exist:sudo groupadd docker
Use the below command to add current user to group
docker
if it has not been done:sudo usermod -aG docker $USER && newgrp docker
-
Docker Compose is required for some elasticsearch integration tests. Docker Compose should be installed on all platforms where it is available if the integration tests are going to be run.
General Notes
- When following the steps below please use a standard permission user unless otherwise specified.
- A directory
/<source_root>/
will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
Build and Install Elasticsearch
1) Build using script
If you want to build Elasticsearch using manual steps, go to STEP 2.
Use the following commands to build Elasticsearch using the build script. Please make sure you have wget
installed.
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.13.3/build_elasticsearch.sh
# Build Elasticsearch
bash build_elasticsearch.sh [Provide -t option for executing build with tests]
If the build completes successfully, go to STEP 9. In case of error, check logs
for more details or go to STEP 2 to follow manual build steps.
2) Install build dependencies
export SOURCE_ROOT=/<source_root>/
export GRADLE_USER_HOME=$SOURCE_ROOT/.gradle
-
RHEL (7.8, 7.9)
sudo yum install -y curl git gzip tar wget patch make gcc gcc-c++ rh-git227 fontconfig urw-fonts source /opt/rh/rh-git227/enable
-
RHEL (8.8, 8.9, 9.2, 9.3)
sudo yum install -y curl git gzip tar wget patch make gcc gcc-c++
-
SLES 12 SP5
sudo zypper install -y curl libnghttp2-devel git gzip tar wget patch make gcc gcc-c++
-
SLES 15 SP5
sudo zypper install -y curl git gzip tar wget patch make gcc gcc-c++
-
Ubuntu (20.04, 22.04)
sudo apt-get update sudo apt-get install -y curl git gzip tar wget patch locales make gcc g++ sudo locale-gen en_US.UTF-8
3) Install Java
-
With Eclipse Adoptium Temurin Runtime
-
With OpenJDK 17
- RHEL (8.8, 8.9, 9.2, 9.3)
sudo yum install -y java-17-openjdk-devel
- SLES 15 SP5
sudo zypper install -y java-17-openjdk java-17-openjdk-devel
- Ubuntu (20.04, 22.04)
sudo apt-get install -y openjdk-17-jdk
- RHEL (8.8, 8.9, 9.2, 9.3)
-
With OpenJDK 21
- RHEL (8.8, 8.9, 9.2, 9.3)
sudo yum install -y java-21-openjdk-devel
- Ubuntu (20.04, 22.04)
sudo apt-get install -y openjdk-21-jdk
- RHEL (8.8, 8.9, 9.2, 9.3)
If running tests with OpenJDK, create a symbolic link to the java home directory to work around a gradle problem where gradle does not recognize the distro provided OpenJDK
sudo mkdir -p /opt/java/
sudo ln -s <Path to JDK> /opt/java/jdk
Note: At the time of creation of these build instructions, Elasticsearch was verified with Eclipse Adoptium Temurin Runtime (Java 17 build 17.0.11+9, Java 21 build 21.0.3+9), OpenJDK 17 and 21.
4) Set the environment variables
export LANG="en_US.UTF-8"
export JAVA_HOME=<Path to JDK>
export ES_JAVA_HOME=/<Path to JDK>/
export PATH=$ES_JAVA_HOME/bin:$PATH
Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.
5) Build Prerequite Projects
5.1) JANSI v2.4.0
cd $SOURCE_ROOT
git clone -b jansi-2.4.0 https://github.com/fusesource/jansi.git
cd jansi
make clean-native native OS_NAME=Linux OS_ARCH=s390x
mkdir -p $SOURCE_ROOT/jansi-jar
cd $SOURCE_ROOT/jansi-jar
wget https://repo1.maven.org/maven2/org/fusesource/jansi/jansi/2.4.0/jansi-2.4.0.jar
jar xvf jansi-2.4.0.jar
cd org/fusesource/jansi/internal/native/Linux
mkdir s390x
cp $SOURCE_ROOT/jansi/target/native-Linux-s390x/libjansi.so s390x/
cd $SOURCE_ROOT/jansi-jar
jar cvf jansi-2.4.0.jar .
mkdir -p $SOURCE_ROOT/.gradle/caches/modules-2/files-2.1/org.fusesource.jansi/jansi/2.4.0/321c614f85f1dea6bb08c1817c60d53b7f3552fd/
cp jansi-2.4.0.jar $SOURCE_ROOT/.gradle/caches/modules-2/files-2.1/org.fusesource.jansi/jansi/2.4.0/321c614f85f1dea6bb08c1817c60d53b7f3552fd/
export sha256=$(sha256sum jansi-2.4.0.jar | awk '{print $1}')
5.2) Docker Images for osixia/openldap
cd $SOURCE_ROOT
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.13.3/patch"
git clone -b v1.2.0 https://github.com/osixia/docker-light-baseimage.git
cd docker-light-baseimage/
curl -sSL "${PATCH_URL}/docker-light-baseimage.patch" | git apply -
make build
cd $SOURCE_ROOT
git clone -b v1.4.0 https://github.com/osixia/docker-openldap.git
cd docker-openldap/
curl -sSL "${PATCH_URL}/docker-openldap.patch" | git apply -
make build
6) Download Elasticsearch and apply patches
cd $SOURCE_ROOT
git clone -b v8.13.3 https://github.com/elastic/elasticsearch
cd elasticsearch
- Apply gradle patches to create s390x distribution
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.13.3/patch/elasticsearch.patch"
curl -o elasticsearch.patch $PATCH_URL
git apply elasticsearch.patch
sed -i 's|6cd91991323dd7b2fb28ca93d7ac12af5a86a2f53279e2b35827b30313fd0b9f|'"${sha256}"'|g' $SOURCE_ROOT/elasticsearch/gradle/verification-metadata.xml
git fetch --tags
latest_tag=$(git tag | sort -V | tail -n1)
latest_tag="${latest_tag:1}-SNAPSHOT"
sed -i 's|${project.version}|'"${latest_tag}"'|g' $SOURCE_ROOT/elasticsearch/x-pack/plugin/ml/build.gradle
Since the latest Elasticsearch version has added docker-export files in their repo, we also need to do the same for s390x. So create the following directories -
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-rpm/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-deb/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/archives/linux-s390x-tar/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/ubi-docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/cloud-docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/cloud-ess-docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/ironbank-docker-s390x-export/
build.gradle
file with the following content -
Inside each directory create a // This file is intentionally blank. All configuration of the
// export is done in the parent project.
7) Build
cd $SOURCE_ROOT/elasticsearch
CPU_NUM="$(grep -c ^processor /proc/cpuinfo)"
export GRADLE_USER_HOME=$SOURCE_ROOT/.gradle
./gradlew :distribution:archives:linux-s390x-tar:assemble --max-workers="$CPU_NUM" --parallel
8) Testing (Optional)
- Build Docker image for OpenLDAP test
cd $SOURCE_ROOT/elasticsearch/x-pack/test/idp-fixture/src/main/resources/openldap/
docker build -f Dockerfile -t docker.elastic.co/elasticsearch-dev/openldap-fixture:1.0 .
- Run test
cd $SOURCE_ROOT/elasticsearch
export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
export JAVA_HOME=/<Path to JDK>/
export RUNTIME_JAVA_HOME=/<Path to JDK>/ # Only with Temurin 17, 21
export RUNTIME_JAVA_HOME=/opt/java/jdk # Only with OpenJDK 17, 21
./gradlew --continue test internalClusterTest -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"
Notes:
- Environment Variables needed for rerunning tests have been added to $HOME/setenv.sh. To set the Environment Variables needed to rerun tests, please run: source $HOME/setenv.sh
- On RHEL 8 and SLES with OpenJDK, you may need to change the system crypto policy with a command like: 'sudo update-crypto-policies --set LEGACY' in order for all security tests to pass.
- On RHEL 9 with OpenJDK, some security tests will fail even with the LEGACY crypto policy.
- If there is an stack overflow error, increase the
-Xss
arg value in the above command. - Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
- The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
- For more information regarding Elasticsearch testing, please refer to their testing documentation.
- User can also create distributions as deb, rpm and docker using below commands.
./gradlew :distribution:packages:s390x-deb:assemble
./gradlew :distribution:packages:s390x-rpm:assemble
./gradlew :distribution:docker:docker-s390x-export:assemble
9) Install Elasticsearch
cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/linux-s390x-tar/build/distributions/elasticsearch-8.13.3-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/
sudo /usr/sbin/groupadd elastic
sudo chown [username]:elastic -R /usr/share/elasticsearch/
- Update configurations to disable unsupported xpack.ml
sudo echo 'xpack.ml.enabled: false' >> /usr/share/elasticsearch/config/elasticsearch.yml
10) Verify Elasticsearch Server
> elasticsearch --version
Version: 8.13.3-SNAPSHOT, Build: tar/617f7b76c4ebcb5a7f1e70d409a99c437c896aea/2024-05-15T16:45:43.022331142Z, JVM: 21.0.2
11) Start Elasticsearch Server
elasticsearch &
export ELASTIC_PASSWORD=[PASSWORD]
Use curl --cacert /usr/share/elasticsearch/config/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200
when running Elasticsearch. The output should be similar to this:
{
"name" : "es01.example.com",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "em6dpwSVT2eXIZSI_Xm9dQ",
"version" : {
"number" : "8.13.3-SNAPSHOT",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "617f7b76c4ebcb5a7f1e70d409a99c437c896aea",
"build_date" : "2024-05-15T16:45:43.022331142Z",
"build_snapshot" : true,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
Notes:
- The password for the elastic user can be found in the output of the
elasticsearch &
command above. - On RHEL 9 with OpenJDK 17, it might be necessary to add the command line options
--tls-max 1.1 --ciphers DEFAULT@SECLEVEL=0
to the curl command above.