Building TensorFlow - linux-on-ibm-z/docs GitHub Wiki

Building TensorFlow

The instructions provided below specify the steps to build TensorFlow version 2.20.0 on Linux on IBM Z for the following distributions:

  • Ubuntu (22.04, 24.04)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

1. Build using script

If you want to build TensorFlow using manual steps, go to STEP 2.

Use the following commands to build TensorFlow using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Tensorflow/2.20.0/build_tensorflow.sh

# Build Tensorflow
bash build_tensorflow.sh [Provide -t option for executing build with tests, -p option for choosing the Python version from {3.9, 3.10, 3.11, 3.12, 3.13}, if not specified, the script will use the system python version]

If the build completes successfully, go to STEP 4. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2. Install dependencies

export SOURCE_ROOT=/<source_root>/
  • Only when choosing Python version 3.9 
    PYTHON_V=3.9
PY_VERSION=3.9.7
  • Only when choosing Python version 3.10 
    PYTHON_V=3.10
PY_VERSION=3.10.6
  • Only when choosing Python version 3.11 
    PYTHON_V=3.11
PY_VERSION=3.11.4
  • Only when choosing Python version 3.12 
    PYTHON_V=3.12
PY_VERSION=3.12.0
  • Only when choosing Python version 3.13 
    PYTHON_V=3.13
PY_VERSION=3.13.9

2.1. Install Basic Dependencies

 sudo apt-get update
 sudo DEBIAN_FRONTEND=noninteractive apt-get install python3-dev python3-pip python3-venv libjpeg-dev libhdf5-dev gfortran libopenblas-dev -y
 export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=True

2.2. Install Clang

cd $SOURCE_ROOT
sudo apt-get install -y lsb-release software-properties-common gnupg
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 21

sudo ln -sf /usr/bin/clang-21 /usr/bin/clang
sudo ln -sf /usr/bin/clang++-21 /usr/bin/clang++
clang --version

2.3. Install Bazel

cd $SOURCE_ROOT
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/7.4.1/build_bazel.sh
bash build_bazel.sh -y
sudo cp $SOURCE_ROOT/bazel/output/bazel /usr/local/bin/bazel
bazel --version

2.4. Install Python (Only for installing Python version 3.9 and 3.10 during build)

cd $SOURCE_ROOT
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Python3/$PY_VERSION/build_python3.sh
sed -i 's/apt-get install/DEBIAN_FRONTEND=noninteractive apt-get install/g' build_python3.sh
sed -i 's/ubuntu-20.04/ubuntu-22.04/g' build_python3.sh    # For Ubuntu 22.04 with Python version 3.9
sed -i 's/ubuntu-20.04/ubuntu-24.04/g' build_python3.sh    # For Ubuntu 24.04 only
bash build_python3.sh -y
sudo update-alternatives --install /usr/local/bin/python python /usr/local/bin/python3 40  
sudo update-alternatives --install /usr/local/bin/pip3 pip3 /usr/local/bin/pip${PYTHON_V} 50
python3 --version
pip3 --version

2.5. Install Numpy wheel (Only for installing Python version 3.9 and 3.10 during build)

cd $SOURCE_ROOT
pip3 wheel numpy==2.0.2 # For Python version 3.9
pip3 wheel numpy==2.2.6 # For Python version 3.10

NUMPY_WHEEL_FILE=$(realpath "$(ls numpy-*.whl | head -n 1)")
NUMPY_WHEEL_HASH=$(sha256sum "$NUMPY_WHEEL_FILE" | awk '{print $1}')

2.6. Build ICU data in big-endian format

cd $SOURCE_ROOT
export ICU_MAJOR_VERSION="69"
export ICU_RELEASE="release-${ICU_MAJOR_VERSION}-1"
git clone --depth 1 --single-branch --branch "$ICU_RELEASE" https://github.com/unicode-org/icu.git
cd icu/icu4c/source/
# create ./filters.json
cat << 'EOF' > filters.json
{
  "localeFilter": {
    "filterType": "language",
    "includelist": [
      "en"
    ]
  }
}
EOF
ICU_DATA_FILTER_FILE=filters.json ./runConfigureICU Linux
make clean && make
# Workaround makefile issue where not all of the resource files may have been processed
find data/out/build/ -name '*pool.res' -print0 | xargs -0 touch
make
cd data/out/tmp
LD_LIBRARY_PATH=../../../lib ../../../bin/genccode "icudt${ICU_MAJOR_VERSION}b.dat"
echo "U_CAPI const void * U_EXPORT2 uprv_getICUData_conversion() { return icudt${ICU_MAJOR_VERSION}b_dat.bytes; }" >> "icudt${ICU_MAJOR_VERSION}b_dat.c"
cp icudt${ICU_MAJOR_VERSION}b_dat.c icu_conversion_data_big_endian.c
gzip icu_conversion_data_big_endian.c
split -a 3 -b 100000 icu_conversion_data_big_endian.c.gz icu_conversion_data_big_endian.c.gz.

3. Build TensorFlow

3.1. Download source code

cd $SOURCE_ROOT
git clone -b v2.20.0 --depth 1 https://github.com/tensorflow/tensorflow
cd tensorflow

3.2. Apply patches

curl -o tf_v2.20.0.patch https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Tensorflow/2.20.0/patch/tf_v2.20.0.patch
patch -p1 < tf_v2.20.0.patch
rm -f tf_v2.20.0.patch
# Only while using Python version 3.9 or 3.10
sed -i \
  -e "s|NUMPY_WHEEL_FILE|$NUMPY_WHEEL_FILE|g" \
  -e "s|NUMPY_WHEEL_HASH|$NUMPY_WHEEL_HASH|g" \
  requirements_lock_3_*.txt

3.3. Copy the big-endian version of ICU data

cp ${SOURCE_ROOT}/icu/icu4c/source/data/out/tmp/icu_conversion_data_big_endian.c.gz.* third_party/icu/data/

3.4. Configure

yes "" | ./configure || true

3.5. Build TensorFlow wheel

 bazel build //tensorflow/tools/pip_package:wheel --repo_env=USE_PYWRAP_RULES=1 --repo_env=WHEEL_NAME=tensorflow_cpu --repo_env=HERMETIC_PYTHON_VERSION=$PYTHON_V

Notes:

  • TensorFlow build is resource intensive operation. If build continues to fail try increasing the swap space and reduce the number of concurrent jobs by specifying --jobs=n in the build command above, where n is the number of concurrent jobs.

  • Building TensorFlow from source can use a lot of RAM. If your system is memory-constrained, limit Bazel's RAM usage with: --local_ram_resources=2048.

4. Install TensorFlow wheel

  • If a Python version was explicitly provided during the build (using --repo_env=HERMETIC_PYTHON_VERSION or using -p option), the SAME Python version must be used to install the generated wheel. If the system Python version differs from the explicitly specified version, install the required Python version as described in Step 2.4 before proceeding.

  • If no Python version was explicitly specified during the build, it uses the system Python by default. In this case, the same system Python version must be used to install the generated wheel.

4.1. Python environment

python3 -m venv tf
source tf/bin/activate
pip3 install --upgrade pip

4.2. Installing Tensorflow wheel

cd $SOURCE_ROOT/tensorflow
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=True pip3 install bazel-bin/tensorflow/tools/pip_package/wheel_house/tensorflow_cpu-2.20.0*-cp*-cp*-linux_s390x.whl

5. Verify TensorFlow (Optional)

Run TensorFlow from command Line and check the installed version

$ cd $SOURCE_ROOT
$ python -c "import tensorflow as tf; print(tf.__version__)"
2.20.0-dev0+selfbuilt

$ python
>>> import tensorflow as tf
>>> tf.add(1, 2).numpy()
3
>>> hello = tf.constant('Hello, TensorFlow!')
>>> hello.numpy()
b'Hello, TensorFlow!'
>>>

6. Execute Test Suite (Optional)

  • Run complete testsuite

    cd $SOURCE_ROOT/tensorflow
bazel test --repo_env=USE_PYWRAP_RULES=1 --repo_env=WHEEL_NAME=tensorflow_cpu --repo_env=HERMETIC_PYTHON_VERSION=$PYTHON_V --build_tests_only --keep_going --test_output=errors --verbose_failures=true --test_tag_filters=-no_oss,-oss_excluded,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_oss_py313 --build_tag_filters=-no_oss,-oss_excluded,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_oss_py313 --test_lang_filters=cc,py -k --test_timeout 300,450,1200,3600 -- //tensorflow/... -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/core/tpu/... -//tensorflow/lite/... -//tensorflow/tools/toolchains/...

  • Run individual test

    bazel test --repo_env=USE_PYWRAP_RULES=1 --repo_env=WHEEL_NAME=tensorflow_cpu --repo_env=HERMETIC_PYTHON_VERSION=$PYTHON_V //tensorflow/<module_name>:<testcase_name>

    For example,

    bazel test --repo_env=USE_PYWRAP_RULES=1 --repo_env=WHEEL_NAME=tensorflow_cpu --repo_env=HERMETIC_PYTHON_VERSION=$PYTHON_V //tensorflow/core/kernels:shape_ops_test

Note:

  • Below test cases will fail as it uses test data generated in LE format. 
    //tensorflow/cc/saved_model:fingerprinting_chunked_test 
//tensorflow/cc/saved_model:fingerprinting_utils_test 
//tensorflow/tools/proto_splitter:merge_test   
//tensorflow/tools/proto_splitter/cc:util_test

7. Model Training Verification (Optional)

Follow the instructions provided here to verify TensorFlow v2.20.0.

References:

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

⚠️ **GitHub.com Fallback** ⚠️