IO Benchmarks with Spack - GEOS-ESM/MAPL GitHub Wiki

Spack for MAPL IO Benchmarks

This document outlines how to install the MAPL IO Benchmarks using Spack.

Note: in this example, we will be using the [email protected] compiler. This is being chosen mainly due to odd behavior currently seen in testing with the Intel oneAPI compilers and Spack.

Note 2: In recent versions of spack, [email protected] was marked as "deprecated". In a section below, we talk about this. If installing [email protected] has issues, you can use [email protected] or install [email protected] as a deprecated package.

Cloning Spack

To clone Spack:

git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git

Shell configuration

Next, once you have spack cloned, you should enable the shell integration by sourcing the correct setup-env file. For example, for bash:

export SPACK_ROOT=$HOME/spack
. $SPACK_ROOT/share/spack/setup-env.sh

Of course, update the SPACK_ROOT variable to point to the location of your spack clone.

This is also recommended to be put in ~/.bashrc or similar for ease of use, or can be run each time you open a new shell.

Add TMPDIR to bashrc (optional)

On some systems, the default TMPDIR might be limited or in a problematic location. If so, update it to a location with more space and add to ~/.bashrc:

mkdir -p $HOME/tmpdir
echo 'export TMPDIR=$HOME/tmpdir' >> ~/.bashrc

Install spack needed packages

From https://spack.readthedocs.io/en/latest/getting_started.html#system-prerequisites there are pre-requisites for spack. For example, on Ubuntu:

sudo apt update
sudo apt install bzip2 ca-certificates g++ gcc gfortran git gzip lsb-release patch python3 tar unzip xz-utils zstd

If you are using a different operating system, you might need to translate these into the packaging system for your OS. The above link has a similar list for RHEL.

Update spack configuration files

Spack configuration files are (by default) in ~/.spack. We will be updating the packages.yaml and repos.yaml files.

Make spack configuration directory

mkdir -p ~/.spack

`packages.yaml`

The complete packages.yaml file we will use is:

packages:
  all:
    providers:
      mpi: [openmpi]
      blas: [openblas]
      lapack: [openblas]
  hdf5:
    variants: +fortran +szip +hl +threadsafe +mpi
  netcdf-c:
    variants: +dap
  esmf:
    variants: ~pnetcdf ~xerces
  cdo:
    variants: ~proj ~fftw3
  pflogger:
    variants: +mpi
  pfunit:
    variants: +mpi +fhamcrest
  fms:
    variants: ~gfs_phys +pic ~yaml constants=GEOS precision=32,64 +deprecated_io
  mapl:
    variants: +extdata2g +fargparse +pflogger +pfunit ~pnetcdf

Copy this into ~/.spack/packages.yaml.

This not only hardcodes the compiler and MPI stack (see below), but also the default variants for packages Spack will build. This eases the spack commands that will be used later (for convenience).

Compiler and MPI

As seen above, the packages.yaml file is used to configure the packages that are available to spack.

In this example, we are telling it we want:

GNU Fortran as our Fortran compiler
Open MPI as our MPI stack
OpenBLAS as our BLAS and LAPACK stack

Supported compilers and MPI stacks

MAPL is not required to use these specific compilers (and barely has a dependency, if any, on BLAS/LAPACK). We use GCC 14 here as a reliable default. GEOSfvdycore supports for Fortran compilers:

GCC 13+
Intel Fortran Classic (ifort) 2021.6+
Intel OneAPI Fortran (ifx) 2025.0+

And MPI stacks that have been tested are:

Open MPI
Intel MPI
MPICH
MPT

It's possible MVAPICH2 might also work, but issues have been seen when using it on Discover, but time has not been invested to determine the actual issue.

NOTE: MAPL does not have a strong dependence on the C and C++ compiler, we mainly focus on Fortran compilers in our testing.

Possible changes for Open MPI and SLURM

Some testing with Open MPI as found that the following variants might be useful or even need for packages.yaml on systems using SLURM. For example using Open MPI 4.1:

  openmpi:
    require:
    - "@4.1.7"
    - "%[email protected]"
    buildable: True
    variants: +legacylaunchers +internal-hwloc +internal-libevent +internal-pmix schedulers=slurm fabrics=ucx
  slurm:
    buildable: False
    externals:
    - spec: [email protected]
      prefix: /usr
  ucx:
    buildable: False
    externals:
    - spec: [email protected]
      prefix: /usr

NOTE 1: You will probably want to change the SLURM and UCX external specs to match the version of SLURM on your system.

NOTE 2: Testing has found that the system UCX is about the only "good" way we have found for Open MPI to see things like Infiniband interfaces (e.g., mlx5_0).

Install GCC 14.2.0

With Spack

To install GCC 14.2.0 with Spack run:

spack install [email protected]

You then also need to tell Spack where the compiler is with:

spack compiler find $(spack location -i [email protected])

When you do that, you'll see it in the ~/.spack/packages.yaml file.

Issues installing GCC 14.2.0 with spack

At some point, [email protected] was marked as Deprecated by spack. As such, when you try to install you might see something like:

==> Error: failed to concretize `[email protected]` for the following reasons:
     1. Cannot satisfy '[email protected]'
        required because [email protected] requested explicitly
     2. Cannot satisfy '[email protected]'
     3. Cannot satisfy 'gcc@=11.5.0' and '[email protected]
        required because [email protected] requested explicitly
        required because gcc available as external when satisfying gcc@=11.5.0 languages:='c,c++,fortran'
          required because [email protected] requested explicitly
     4. Cannot satisfy '[email protected]:' and '[email protected]
        required because [email protected] requested explicitly
        required because @11.3: is a requirement for package gcc
          required because gcc available as external when satisfying gcc@=11.5.0 languages:='c,c++,fortran'
            required because [email protected] requested explicitly
          required because [email protected] requested explicitly

There are three "solutions" for this:

You can install [email protected] as a deprecated package with

spack install --deprecated [email protected]

You can tell spack to allow deprecated packages by editing/creating a file called ~/.spack/config.yaml and adding:

config:
  deprecated: true

You can move to [email protected] as that is now the "preferred" GCC 14 version:

spack install [email protected]

NOTE: If you choose the [email protected] path, then everywhere in this document you see a reference to [email protected] use [email protected]

With system package manager

If your system has GCC 14.2.0, you can instead just install it via your system package manager. Then you can run:

spack compiler find

might find it. You can check if it found it by looking at ~/.spack/packages.yaml and there should be entries for gcc

Install MAPL

You can now install MAPL with:

spack install mapl %[email protected]

On a test install, this built about 90 packages.

Running the MAPL IO Benchmarks

MAPL in general has three benchmarks. Two are write and read counterparts and the third is a pure bandwidth benchmark. In each case, the executables can be found in the install/bin directory found by:

cd $(spack location -i mapl %[email protected])/bin

Also, before running them, you'll want to run:

spack load mapl

so that it MPI, etc. is in the environment.

`checkpoint_simulator.x`

checkpoint_simulator.x is a code that simulates the checkpoint writing of MAPL. By default, it will write out netCDF4 files as that is how GEOS writes out the checkpoints of the state. The code has many options the full list of which can be found in its README.md.

The main options a user will change are:

  --nx                       The number of cells in the x direction (default: 4)
  --ny                       The number of cells in the y direction (default: 4)
  --im_world                 The resolution of the cubed sphere (default: 90)
  --lm                       The number of levels in each 3D variable (default: 137)
  --num_writers              The number of processes that will write (default: 1)
  --num_arrays               The number of 3D arrays to write (default: 5)
  --ntrials                  The number of trials to run (default: 3)

In each case, a user will need to call checkpoint_simulator.x using mpirun (or similar MPI launcher). When a user does this they need to call it with nx * ny * 6 processes. So if a user wants to run this set of options:

--nx 6 --ny 6 --im_world 180 --lm 181

then it must be called as:

mpirun -np 216 checkpoint_simulator.x --nx 6 --ny 6 --im_world 180 --lm 181

The README.md also details a way for users to use a configuration file to control checkpoint_simulator.x.

`restart_simulator.x`

restart_simulator.x is the read counterpart to checkpoint_simulator.x. It has all the same caveats as checkpoint_simulator.x such as needing to use nx * ny * 6 MPI processes. However, the one main difference is it is a read benchmark so it needs a file to read! Thus, you MUST run checkpoint_simulator.x first to generate the file used in restart_simulator.x.

Like checkpoint_simulator.x, the restart_simulator.x README.md has the full list of options, but they mainly echo checkpoint_simulator.x except if an option for checkpoint_simulator.x has "writer" in the name, it'll be "reader" for restart_simulator.x

The best advice is, if you run checkpoint_simulator.x as:

mpirun -np 216 checkpoint_simulator.x --nx 6 --ny 6 --im_world 180 --lm 181

you'd run restart_simulator.x as:

mpirun -np 216 restart_simulator.x --nx 6 --ny 6 --im_world 180 --lm 181

`raw_bw.x`

raw_bw.x is, as it says, a raw bandwidth simulator. It's options are much simpler compared to the above benchmark codes:

 usage: ./raw_bw.x [-h][--nx NX][--n_levs N_LEVS][--n_writers N_WRITERS][--n_tries N_TRIES][--file_type FILE_TYPE]

 optional arguments:
  -h, --help                 This message.
  --nx                       dimension of cube face
  --n_levs                   number of vertical levels
  --n_writers                number of simultaneous, independent writes to disk
  --n_tries                  number of times to call benchmark to get statistics
  --file_type                type of file, options netcdf or binary

The "best" way to run raw_bw.x is often in a bash script in a loop like:

mpirun -np 6 $INSTALLDIR/bin/raw_bw.x --nx 4 --n_levs 1 --n_writers 1 --n_tries 1 | head -2
for n_writers in 24 30 36 42 48 54 60 66 72 78 84 90 96 102 108 114 120
do
   mpirun -np 120 $INSTALLDIR/bin/raw_bw.x --nx 1440 --n_levs 181 --n_writers $n_writers --n_tries 3 | tail -1
done

The script above is running a c1440 cube-sphere resolution and varying the number of writers from 24 to 120. The "first" mpirun you see is actually just there to make a header for the output:

 Tested with file type: binary
        NX,    # levs, # writers,     write (GB),    packet (GB),       Time (s), Eff. BW (GB/s), Avg. BW (GB/s), Rel. Std. Dev.
      1440,       181,        24,         9.0077,         0.3753,         0.7240,        12.4412,         0.5184,         0.3103
      1440,       181,        30,         9.0077,         0.3003,         0.6610,        13.6281,         0.4543,         0.1406
      1440,       181,        36,         9.0077,         0.2502,         0.6936,        12.9878,         0.3608,         0.1865