Building QUDA with HIP/ROCm

Introduction

QUDA can now be built with HIP/ROCm using the hipcc compiler as both the C++ and the HIP compiler. The procedure utilizes Native HIP language support in CMake and this requires minimally CMake 3.21.5.

Which branch shall I use?

The HIP support has been merged into develop and should be available with that branch.

Current ROCm and CMake

The build insturctions below have been tested with ROCm v4.5.2 and CMake v3.22.1

PLEASE NOTE Some issues have been reported building with ROCm-5.0.x

CMake Settings needed to build with HIP

• `QUDA_TARGET_TYPE="HIP" -- use this to enable a HIP/ROCm build
• QUDA_GPU_ARCH -- use this to set the GPU architecture (e.g. gfx908). You can also set a list ( e.g. gfx906;gfx908;gfx90a )
• CMAKE_CXX_COMPILER=hipcc -- use HIPCC as the C++ compiler
• CMAKE_C_COMPILER=hipcc -- use HIPCC as the C compiler
• ROCM_PATH=<path to toplevel rocm e.g. ${ROCM_PATH} > -- point to the ROCm installation to use

The table below lists the AMD architecture names to the various GPU models and ISAs.

Architecture Name	ISA	GPUs
gfx906	Vega20(GCN5)	MI50, MI60
gfx908	Vega20+CDNA	MI100
gfx90a	Vega20+CDNA2	MI210, MI250, MI250X

The remaining options should work as for normal QUDA. In the case of QDP-JIT, it helps also to supply some QDP_JIT related cache variables

• LLVN_DIR=${ROCM_PATH}/llvm/lib/cmake/llvm -- the location of the LLVMConfig.cmake files
• LLD_DIR=${ROCM_PATH}/llvm/lib/cmake/lld -- the location of the LLDConfig.cmake files

These latter can also be set by simply adding the toplevel ${ROCM_PATH}/llvm to the CMAKE_PREFIX_PATH environment variable.

Notes for HPE/Cray AMD Systems.

Currently we build the codes using hipcc rather than the Cray Programming Environment CC and cc wrappers. As such to link against MPI, the include paths must be added onto CMAKE_CXX_FLAGS and the link paths for MPI and for the GPU-Transport-Layer (GTL) libraries must be manually added to the link-paths. In addition for GPU aware MPI the craype-accel-amd-XXXXXX module (with XXXXXX being the placeholder for the GPU architecture, e.g. gfx908) needs to be loaded. Further to enable GPU aware MPI at runtime the environment variable MPICH_GPU_SUPPORT_ENABLED must be enabled (e.g. set to 1).

A simple way is to use a script to set up modules and the environment as well as these flags, let us call this file ./setup_env.sh. An example such file, appropriate for the Spock system at OLCF at the time of writing (using Cray MPICH 8.1.12) would look like so:

### Load the modules 
module load craype-accel-amd-gfx908
module unload PrgEnv-cray
module load cmake/3.22.1
module load rocm/4.5.0
module load libxml2

### These variables will save my fingers 
export MPICH_ROOT=/opt/cray/pe/mpich/8.1.12/
export MPICH_DIR=${MPICH_ROOT}/ofi/crayclang/10.0

### This env var must be set before running
export MPICH_GPU_SUPPORT_ENABLED=1

### CFLAGS and LDFLAGS for MPI relative to $MPICH_DIR and $MPICH_ROOT
MPI_CFLAGS="-I${MPICH_DIR}/include"
MPI_LDFLAGS="-L${MPICH_DIR}/lib -lmpi -L${MPICH_ROOT}/gtl/lib -lmpi_gtl_hsa"

### Shared library paths that are useful to set at runtime 
### Your mileage may vary
export PATH=${ROCM_PATH}/bin:${PATH}
export LD_LIBRARY_PATH=${ROCM_PATH}/llvm/lib64:${ROCM_PATH}/llvm/lib:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH=${MPICH_ROOT}/gtl/lib:${MPICH_DIR}/lib:${LD_LIBRARY_PATH}

Example CMake Script

Here is an example CMake Script for building on Spock using the setup_env.sh script above. This builds assumes that

• the env var SRCROOT points to a directory that contains QUDA in a subdirectory ${SRCDIR}/quda
• the env var BUILDROOT points to an existing directory where a build_quda subdirectory will be (re)created (if it already exists)
• the env var INSTALLROOT points to an existing directory for packages to be installed under
• the QMP library is installed in $INSTALLROOT/qmp
• I already have Eigen-3.3.9 downloaded in $SRCROOT/eigen-3.3.9

source ./setup_env.sh

export BUILDROOT=...    ### Set BUILDROOT to the directory in which you want to build
export SRCROOT=...      ### Set SRCROOT  to the directory where the source for QUDA lives
export INSTALLROOT=...  ### Set INSTALLROOT to the directory where you want to install packages

pushd ${BUILDROOT}
if [ -d ./build_quda ];
then
  rm -rf ./build_quda
fi

mkdir  ./build_quda
cd ./build_quda

cmake ${SRCROOT}/quda \
        -G "Unix Makefiles" \
        -DQUDA_TARGET_TYPE="HIP" \
        -DQUDA_GPU_ARCH="gfx908" \
        -DROCM_PATH=${ROCM_PATH} \
        -DQUDA_DIRAC_CLOVER=ON \
        -DQUDA_DIRAC_CLOVER_HASENBUSCH=ON \
        -DQUDA_DIRAC_DOMAIN_WALL=ON \
        -DQUDA_DIRAC_NDEG_TWISTED_MASS=OFF \
        -DQUDA_DIRAC_STAGGERED=ON \
        -DQUDA_DIRAC_TWISTED_MASS=OFF \
        -DQUDA_DIRAC_TWISTED_CLOVER=OFF \
        -DQUDA_DIRAC_WILSON=ON \
        -DQUDA_DYNAMIC_CLOVER=OFF \
        -DQUDA_FORCE_GAUGE=ON \
        -DQUDA_FORCE_HISQ=ON \
        -DQUDA_GAUGE_ALG=ON \
        -DQUDA_GAUGE_TOOLS=OFF \
        -DQUDA_INTERFACE_MILC=ON \
        -DQUDA_INTERFACE_CPS=OFF \
        -DQUDA_INTERFACE_QDP=ON \
        -DQUDA_INTERFACE_TIFR=OFF \
        -DQUDA_QMP=ON \
        -DQMP_DIR=${INSTALLROOT}/qmp/lib/cmake/QMP \
        -DQUDA_QIO=OFF \
        -DQUDA_OPENMP=OFF \
        -DQUDA_MULTIGRID=ON \
        -DQUDA_MAX_MULTI_BLAS_N=9 \
        -DQUDA_DOWNLOAD_EIGEN=OFF \
        -DEIGEN_INCLUDE_DIR=${SRCROOT}/eigen-3.3.9 \
        -DCMAKE_INSTALL_PREFIX=${INSTALLROOT}/quda \
        -DCMAKE_BUILD_TYPE="DEVEL" \
        -DCMAKE_CXX_COMPILER="hipcc"\
        -DCMAKE_C_COMPILER="hipcc" \
        -DBUILD_SHARED_LIBS=ON \
        -DQUDA_BUILD_SHAREDLIB=ON \
        -DQUDA_BUILD_ALL_TESTS=OFF \
        -DQUDA_CTEST_DISABLE_BENCHMARKS=ON \
        -DCMAKE_CXX_FLAGS="${MPI_CFLAGS}" \
        -DCMAKE_C_FLAGS="${MPI_CFLAGS}" \
        -DCMAKE_EXE_LINKER_FLAGS="${MPI_LDFLAGS}" \
        -DCMAKE_SHARED_LINKER_FLAGS="${MPI_LDFLAGS}" \
        -DCMAKE_C_STANDARD=99


cmake --build . -j 32  -v
cmake --install .

NB: The QUDA_DOWNLOAD_USQCD CMake option should work as before.

FIXME/Todo Items

• Allow building using GCC/G++ and using HIPCC only for compiling specific HIP codes