The instructions provided below specify the steps to build TensorFlow Transform version 1.21.0 on Linux on IBM Z for the following distributions:

• Ubuntu (22.04, 24.04)

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

If you want to build TensorFlow Transform using manual steps, go to STEP 1.2.

Use the following commands to build TensorFlow Transform using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowTransform/1.21.0/build_tensorflow_transform.sh

# Build TensorFlow Transform
bash build_tensorflow_transform.sh    [Provide -t option for executing build with tests, -p option for choosing the Python version from {3.10, 3.11, 3.12, 3.13}, if not specified, uses whichever python3 is already the system default.]

If the build completes successfully, go to STEP 2. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

export SOURCE_ROOT=/<source_root>/
export PYTHON_VERSION=<python_version>  #Choose Python version from {3.10, 3.11, 3.12, 3.13}
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

• Ubuntu 22.04

  sudo apt-get update
  sudo apt-get install -y build-essential cargo curl git cmake libopenblas-dev libgeos-dev \
      wget pkg-config libhdf5-dev libffi-dev libjpeg-dev libpng-dev libtiff-dev \
      libgrpc-dev libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc

• Ubuntu 24.04

  sudo apt-get update
  sudo apt-get install -y build-essential cargo curl git cmake libopenblas-dev libgeos-dev \
      wget pkg-config libhdf5-dev libffi-dev libjpeg-dev libpng-dev libtiff-dev \
      libgrpc-dev libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc

• TensorFlow is a pre-requisite and is not built by this script. To install a TensorFlow refer - Building-TensorFlow.md.

cd $SOURCE_ROOT
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/7.4.1/build_bazel.sh
bash build_bazel.sh -y
sudo cp $SOURCE_ROOT/bazel/output/bazel /usr/local/bin/bazel
bazel --version

• Download source code

  cd $SOURCE_ROOT
  git clone -b apache-arrow-17.0.0 --depth 1 https://github.com/apache/arrow.git

• Build and install Arrow C++ library

  cd $SOURCE_ROOT/arrow/cpp
  mkdir -p build
  cd build
  cmake -DCMAKE_INSTALL_PREFIX=/usr/local \
        -DCMAKE_INSTALL_LIBDIR=lib \
        -DARROW_PARQUET=ON \
        -DARROW_PYTHON=ON \
        -DARROW_ORC=OFF \
        -DARROW_DATASET=ON \
        -DCMAKE_BUILD_TYPE=Release \
        ..
  make -j$(nproc)
  sudo make install
  sudo ldconfig
  export LD_LIBRARY_PATH=/usr/local/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

• Build and install pyarrow library

  cd $SOURCE_ROOT/arrow/python
  sed -i 's|setuptools>=64|setuptools>=64,<72|' requirements-build.txt
  sed -i 's|cython>=0.29.31|cython>=0.29.31,<3.3|' requirements-build.txt
  pip3 install -r requirements-build.txt
  export PYARROW_WITH_PARQUET=1
  export PYARROW_PARALLEL=$(nproc)
  ARROW_HOME=/usr/local python${PYTHON_VERSION} setup.py build_ext --build-type=release --bundle-arrow-cpp bdist_wheel
  pip3 install dist/*.whl

• Install grpcio 1.73.0

  GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip3 install --no-build-isolation 'grpcio==1.73.0'

• Install grpcio-status 1.73.0

  GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip3 install --no-build-isolation 'grpcio-status==1.73.0'

• Install tensorflow-serving-api 2.19.0

  pip3 install tensorflow-serving-api==2.19.0

• Build and install Apache Beam

  cd $SOURCE_ROOT
  PROTOBUF_VER=$(pip3 show protobuf | awk '/^Version:/{print $2}')
  printf "grpcio==1.73.0\ngrpcio-status==1.73.0\nprotobuf==${PROTOBUF_VER}\ncryptography<47\norjson==3.11.9\n" > constraints.txt
  printf "setuptools==79.0.0\n" > build_constraints.txt
  CRC32C_PURE_PYTHON=1 \
  GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 \
  GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 \
      pip3 install \
          --constraint constraints.txt \
          --build-constraint build_constraints.txt \
          'apache-beam[gcp]==2.75.0'

• Download source code

  cd $SOURCE_ROOT
  git clone -b v1.21.0 --depth 1 https://github.com/tensorflow/tfx-bsl.git
  cd tfx-bsl

• Build and install tfx-bsl

  sed -i 's/versions.check("7.7.0")/versions.check("7.4.1")/' WORKSPACE
  python${PYTHON_VERSION} setup.py bdist_wheel
  pip3 install --no-deps dist/*.whl
  pip3 install \
      --constraint "$SOURCE_ROOT/constraints.txt" \
      'absl-py<2.0.0,>=0.9' \
      dill \
      'google-api-python-client>=1.7.11' \
      'pandas>=2.0'
  pip3 install --no-deps 'tensorflow-metadata<1.22.0,>=1.21.0'

• Download source code

  cd $SOURCE_ROOT
  git clone -b v1.21.0 --depth 1 https://github.com/tensorflow/transform.git

• Build and install

  cd $SOURCE_ROOT/transform
  python${PYTHON_VERSION} setup.py bdist_wheel
  pip3 install --no-deps dist/*.whl
  pip3 install --constraint "$SOURCE_ROOT/constraints.txt" 'pydot<2,>=1.2' 'tf-keras>=2'

Note: If any other particular version of a python package is required during installation, please run sudo pip3 install '<package-name>==<version>' to install it:

• Run TensorFlow Transform from command Line

export LD_LIBRARY_PATH=/usr/local/lib\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}}
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
cat << 'EOF' > verify_tft.py
import tempfile
import tensorflow as tf
import tensorflow_transform as tft
import tensorflow_transform.beam as tft_beam
from tensorflow_transform.tf_metadata import dataset_metadata, schema_utils

raw_data = [
    {'x': 1, 'y': 1, 's': 'hello'},
    {'x': 2, 'y': 2, 's': 'world'},
    {'x': 3, 'y': 3, 's': 'hello'},
]

raw_data_metadata = dataset_metadata.DatasetMetadata(
    schema_utils.schema_from_feature_spec({
        'y': tf.io.FixedLenFeature([], tf.float32),
        'x': tf.io.FixedLenFeature([], tf.float32),
        's': tf.io.FixedLenFeature([], tf.string),
    }))

def preprocessing_fn(inputs):
    return {
        'x_centered': inputs['x'] - tft.mean(inputs['x']),
        'y_normalized': tft.scale_to_0_1(inputs['y']),
        's_integerized': tft.compute_and_apply_vocabulary(inputs['s']),
    }

with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
    transformed_dataset, transform_fn = (
        (raw_data, raw_data_metadata)
        | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn))

transformed_data, _ = transformed_dataset
print(transformed_data)
EOF
python${PYTHON_VERSION} verify_tft.py

• Expected Output

[{'s_integerized': 0, 'x_centered': -1.0, 'y_normalized': 0.0}, {'s_integerized': 1, 'x_centered': 0.0, 'y_normalized': 0.5}, {'s_integerized': 0, 'x_centered': 1.0, 'y_normalized': 1.0}]

• Follow instructions in this tutorial to use TensorFlow Transform to preprocess data.

• Run the complete testsuite

  cd $SOURCE_ROOT/transform
  python${PYTHON_VERSION} -m pytest -v tensorflow_transform/

• https://www.tensorflow.org/tfx/transform/api_docs/python/tft
• https://github.com/tensorflow/transform