OpenMPI and OpenSHMEM installation with UCX - openucx/ucx GitHub Wiki

UCX installation

Requirements: Autoconf 2.63 and above.

  1. Get latest version of the UCX code
$ git clone https://github.com/openucx/ucx.git ucx
$ cd ucx
  1. Run autogen:
$ ./autogen.sh
  1. (This step is only required for OpenPOWER platforms) Power 8 On Ubuntu platform the config.guess file is a bit outdated and does not have support for power. In order to resolve the issue you have to download an updated config.guess. From the root of the project:
$ wget https://github.com/shamisp/ucx/raw/topic/power8-config/config.guess
  1. Configure:
$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_path

NOTE: For best performance configuration, use ../contrib/configure-release.
This will strip all debugging and profiling code.

NOTE: For debug/development build configuration, use ../contrib/configure-devel.
This will enable all debugging and profiling code.

  1. Build and install:
$ make
$ make install

OpenMPI and OpenSHMEM installation

  1. Get latest-and-greatest OpenMPI version:
$ git clone https://github.com/open-mpi/ompi.git
  1. Autogen:
$ cd ompi
$ ./autogen.pl
  1. Configure with UCX:
$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_path/ --with-ucx=/path_to_ucx_installation

NOTE With OpenMPI 4.0 and above, there could be compilation errors from "btl_uct" component. This component is not critical for using UCX; so it could be disabled this way:

$ ./configure ... --enable-mca-no-build=btl-uct ...
  1. Build:
$ make
$ make install

Running Open MPI with UCX

Example of the command line (with optional flag to select IB device mlx5_0 port 1):

$ mpirun -np 2 -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 ./app

IMPORTANT NOTE: Recent OpenMPI versions contain a BTL component called 'uct', which can cause data corruption when enabled, due to conflict on malloc hooks between OPAL and UCM. In order to work-around this, use one of the following alternatives:

Alternative 1: Disable btl/uct in OpenMPI build configuration:

$ ./configure ... --enable-mca-no-build=btl-uct ...

Alternative 2: Disable btl/uct at runtime

$ mpirun -np 2 -mca pml ucx -mca btl ^uct -x UCX_NET_DEVICES=mlx5_0:1 ./app

Open MPI runtime optimizations for UCX

  • By default OpenMPI enables build-in transports (BTLs), which may result in additional software overheads in the OpenMPI progress function. In order to workaround this issue you may try to disable certain BTLs.
$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 ./app

MPI/SHMEM versions tested with UCX master

  1. UCX current tarball: https://github.com/openucx/ucx/archive/master.zip

  2. The table of MPI and OpenSHMEM distributions that are tested with the HEAD of UCX master

MPI/OpenSHMEM project
OpenMPI/OSHMEM 4.0 and above
MPICH Latest

Running on Cray systems with Aries and Gemini interconnect

In order to run UCX on Cray systems a user has explicitly enabled UCX unified mode: UCX_UNIFIED_MODE=1. For example:

$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_UNIFIED_MODE=1 ./app