Amanzi on NERSC Cori - amanzi/amanzi GitHub Wiki
There are a lots of quirks with NERSC systems.
To login to one of the front-end machines,
ssh <username>@cori.nersc.gov
You probably have two-factor authentication setup so you'll enter your password+OTP (see NERSC - MFA help).
You should be a member of the ASCEM project on NERSC, and so you should also be in the unix group m1012. You can check with the groups command
moulton@cori06:~/amanzi-tpls/cori> groups moulton
moulton : moulton m2421 m2511 m3421 m3780 m3940 m1012 m2398
So our project has space for installations of TPLs, Amanzi, and tools in
/project/projectdirs/m1012
I suggest setting an environment variable (e.g., ASCEM_HOME) so you don't have to keep typing the full path in scripts or on the command line.
You will build the code on the front-end, and we recommend building in your home directory. It's a good idea to make sure you have enough space in your home directory before you start using myquota,
moulton@cori06:~/amanzi-tpls/cori> myquota
FILESYSTEM SPACE_USED SPACE_QUOTA SPACE_PCT INODE_USED INODE_QUOTA INODE_PCT
home 15.88GiB 40.00GiB 39.7% 525.11K 1.00M 52.5%
cscratch1 36.40GiB 20.00TiB 0.2% 2.97M 10.00M 29.7%
The build is is pretty large ~20GB so here I probably do have enough space. Note that once the install is finished you can delete the build directories.
We have some history here, as we tried to make things very uniform and easy for the team if they wanted to build their own TPLs and Amanzi. Although it's worth noting that many developers just rely on TPLs we've installed an only build Amanzi. To provide a uniform environment we settled on using the GNU compilers, the corresponding Cray MPICH, and a recent cmake. This is managed by a script that loads a module (yes, that's a bit ugly and probably not necessary anymore but historically there were problems swapping the programming environment in a module).
I load the modules using
. ${ASCEM_HOME}/tools/init/ascem.bashrc gnu
and end up with the following environment
Currently Loaded Modulefiles:
1) modules/3.2.11.4 14) dmapp/7.1.1-7.0.1.1_4.72__g38cf134.ari
2) altd/2.0 15) gni-headers/5.0.12.0-7.0.1.1_6.46__g3b1768f.ari
3) darshan/3.2.1 16) xpmem/2.2.20-7.0.1.1_4.28__g0475745.ari
4) craype-network-aries 17) job/2.2.4-7.0.1.1_3.55__g36b56f4.ari
5) gcc/8.3.0 18) dvs/2.12_2.2.167-7.0.1.1_17.11__ge473d3a2
6) craype/2.6.2 19) alps/6.6.58-7.0.1.1_6.30__g437d88db.ari
7) cray-mpich/7.7.10 20) rca/2.2.20-7.0.1.1_4.74__g8e3fb5b.ari
8) craype-haswell 21) atp/2.1.3
9) craype-hugepages2M 22) PrgEnv-gnu/6.0.5
10) cray-libsci/19.06.1 23) cmake/3.21.3
11) udreg/2.3.2-7.0.1.1_3.61__g8175d3d.ari 24) git/2.21.0
12) ugni/6.0.14.0-7.0.1.1_7.63__ge78e5b0.ari 25) AmanziTools/2.3
13) pmi/5.0.14
Note that the ``AmanziTools/2.3` module also sets the cray build environment to use dynamic linking (shared libraries)
CRAYPE_LINK_TYPE=dynamic
Amanzi (and probably any sizable application) cannot be linked statically on Cori (long story around Cray Hugepages libraries).
Create a directory for the TPLs build and use bootstrap.sh to build the TPLs. Since I have a several builds I'm working on, I have some scripts that help create verbose directory names and hierarchies for my installation. This is not necessary for most developers, and I suggest you do whatever works (and is easiest) for you. Here's the list of options I pass to bootstrap.sh
opt: --parallel=6
opt: --opt
opt: --with-mpi=/opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2
opt: --disable-build_amanzi
opt: --disable-build_user_guide
opt: --enable-shared
opt: --arch=NERSC
opt: --with-c-compiler=/opt/cray/pe/craype/2.6.2/bin/cc
opt: --with-cxx-compiler=/opt/cray/pe/craype/2.6.2/bin/CC
opt: --with-fort-compiler=/opt/cray/pe/craype/2.6.2/bin/ftn
opt: --enable-silo
opt: --enable-alquimia
opt: --enable-pflotran
opt: --enable-crunchtope
opt: --tpl-install-prefix=/project/projectdirs/m1012/amanzi/tpls/install/cori/mpich2-7.7.10-gnu-8.3.0/Release-0.98.2-shared
opt: --tpl-build-dir=/global/homes/m/moulton/amanzi-tpls/cori/gcc-TPLs-0.98.2-shared-2022
opt: --tpl-download-dir=/project/projectdirs/m1012/amanzi/tpls/Downloads
Note what's critical, are the following options,
--with-mpi=/opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2
--with-c-compiler=/opt/cray/pe/craype/2.6.2/bin/cc
--with-cxx-compiler=/opt/cray/pe/craype/2.6.2/bin/CC
--with-fort-compiler=/opt/cray/pe/craype/2.6.2/bin/ftn
--enable-shared
--arch=NERSC
Where the compiler full paths/names were obtained with which cc, which CC, and which ftn, respectively. Also, note that the --arch=NERSC sets the following additional flags for our cmake
Setting ARCH: NERSC
ARCH TPL OPTIONS = -DAMANZI_ARCH:STRING=NERSC -DMPI_EXEC:STRING=srun -DMPI_EXEC_NUMPROCS_FLAG:STRING=-n -DPREFER_STATIC_LIBRARIES:BOOL=0 -DBUILD_STATIC_EXECUTABLES:BOOL=0
ARCH AMANZI OPTIONS = -DTESTS_REQUIRE_MPIEXEC:BOOL=1 -DTESTS_REQUIRE_FULLPATH:BOOL=1
Finally, note that I'm using common project space for the TPL tarball downloads,
--tpl-download-dir=/project/projectdirs/m1012/amanzi/tpls/Downloads
This is optional, but saves time and space for you.
We have customized FindBLAS.cmake and FindLAPACK.cmake in a few small but important ways. In this case they are able to find the BLAS and LAPACK libraries that are provided by the Cray Scientific Library:
-- Found BLAS: /opt/cray/pe/libsci/19.06.1/GNU/8.1/haswell/lib/libsci_gnu.so
-- >>> Amanzi: BLAS_LIBRARIES = /opt/cray/pe/libsci/19.06.1/GNU/8.1/haswell/lib/libsci_gnu.so
-- Found LAPACK: /opt/cray/pe/libsci/19.06.1/GNU/8.1/haswell/lib/libsci_gnu.so;/opt/cray/pe/libsci/19.06.1/GNU/8.1/haswell/lib/libsci_gnu.so
-- >>> Amanzi: LAPACK_LIBRARIES = /opt/cray/pe/libsci/19.06.1/GNU/8.1/haswell/lib/libsci_gnu.so;/opt/cray/pe/libsci/19.06.1/GNU/8.1/haswell/lib/libsci_gnu.so
And we have confirmed that these libraries are passed to, and used by, all of the TPLs.
With all the above setup in place, just use bootstrap.sh and provide the necessary options. Using the options listed above, my build completed and provided the following message:
[Mon Feb 21 11:04:15 PST 2022] TPL build complete
[Mon Feb 21 11:04:15 PST 2022] For future Amanzi builds use /project/projectdirs/m1012/amanzi/tpls/install/cori/mpich2-7.7.10-gnu-8.3.0/Release-0.98.2-shared/share/cmake/amanzi-tpl-config.cmake
Note, your Amanzi build will need that configuration file to ensure that it can find all the TPLs we just built.