Bulding Spack‐Stack - myroms/roms GitHub Wiki

Installing the Spack-Stack libraries on computers with different operating systems is not trivial and requires expertise and system privileges. You can find much information online, which is more verbose than the simple guidelines below. Please check the following Spack-Stack tutorial:

https://spack-tutorial.readthedocs.io/en/latest/tutorial_stacks.html

Instructions

Here are the steps we use at Rutgers to install Spack-Stack on our Linux computers and clusters. We use modules to load the required libraries to compile and link ROMS-JEDI and UFS applications. We build modules for Intel and Gfortran. For example, to load the Intel module that we named stack-intel, we get:

> module load stack-intel; module list

Currently Loaded Modules:
  1) MATLAB/R2024a              40) g2tmpl/1.13.0                79) py-attrs/23.1.0
  2) totalview/2024.2.11        41) ip/5.1.0                     80) py-pycparser/2.21
  3) intel/2021.8.0             42) cmake/3.27.9                 81) py-cffi/1.17.1
  4) glibc/2.34                 43) git/2.43.5                   82) py-findlibs/0.0.2
  5) numactl/2.0.18             44) nccmp/1.9.0.1                83) py-eccodes/1.5.0
  6) zlib-ng/2.2.1              45) py-setuptools/69.2.0         84) py-f90nml/1.4.3
  7) pmix/5.0.3                 46) py-packaging/24.1            85) py-h5py/3.12.1
  8) openmpi/4.1.7              47) py-tomli/2.0.1               86) py-certifi/2023.7.22
  9) stack-openmpi/4.1.7        48) py-typing-extensions/4.12.2  87) py-cftime/1.0.3.4
 10) bacio/2.4.1                49) py-setuptools-scm/8.0.4      88) py-netcdf4/1.7.1.post2
 11) snappy/1.2.1               50) py-wheel/0.41.2              89) py-six/1.16.0
 12) zstd/1.5.6                 51) wget/1.21.1                  90) py-python-dateutil/2.8.2
 13) c-blosc/1.21.5             52) base-env/1.0.0               91) py-pytz/2023.3
 14) nghttp2/1.63.0             53) boost/1.84.0                 92) py-tzdata/2023.3
 15) curl/8.10.1                54) bufr/12.1.0                  93) py-pandas/2.2.3
 16) hdf5/1.14.3                55) eigen/3.4.0                  94) py-pybind11/2.13.5
 17) netcdf-c/4.9.2             56) eckit/1.28.3                 95) py-pycodestyle/2.11.0
 18) netcdf-fortran/4.6.1       57) gsl-lite/0.37.0              96) py-pyhdf/0.11.4
 19) parallel-netcdf/1.12.3     58) netcdf-cxx4/4.3.1            97) py-scipy/1.13.1
 20) parallelio/2.6.2           59) bufr-query/0.0.4             98) py-xarray/2024.7.0
 21) gettext/0.21               60) ecbuild/3.7.2                99) udunits/2.2.28
 22) sqlite/3.34.1              61) openjpeg/2.3.1              100) jedi-base-env/1.0.0
 23) util-linux-uuid/2.40.2     62) eccodes/2.33.0              101) gftl/1.14.0
 24) python/3.11.7              63) fckit/0.13.2                102) gftl-shared/1.9.0
 25) python-venv/1.0            64) fiat/1.4.1                  103) fargparse/1.8.0
 26) py-mpi4py/4.0.1            65) ectrans/1.5.0               104) yafyaml/1.4.0
 27) intel-tbb/2021.12.0        66) qhull/2020.2                105) pflogger/1.15.0
 28) intel-oneapi-mkl/2024.2.2  67) atlas/0.40.0                106) mapl/2.53.0-esmf-8.6.1
 29) py-numpy/1.26.4            68) git-lfs/3.4.1               107) antlr/2.7.7
 30) py-pip/23.1.2              69) sp/2.5.0                    108) gsl/2.8
 31) libyaml/0.2.5              70) gsibec/1.2.1                109) nco/5.2.4
 32) py-pyyaml/6.0.2            71) krb5/1.21.3                 110) nemsio/2.5.4
 33) esmf/8.6.1                 72) libtirpc/1.3.3              111) sfcio/1.4.2
 34) fms/2024.02                73) hdf/4.2.15                  112) sigio/2.3.3
 35) libjpeg/2.1.0              74) jedi-cmake/1.4.0            113) w3nco/2.4.1
 36) jasper/2.0.32              75) ncview/2.1.8                114) jedi-ufs-env/1.0.0
 37) libpng/1.6.37              76) json/3.11.3                 115) stack-intel/2021.8.0
 38) w3emc/2.10.0               77) json-schema-validator/2.3.0
 39) g2/3.5.1                   78) odc/1.5.2

To build the Spack-Stack needed on our computers at Rutgers, we execute the following steps:

If you have the permissions necessary to install software with your operating system's package manager, we recommend installing the following packages to reduce the number of packages Spack-Stack needs to build:
```
qt5-devel
git-lfs
bash-completion
bzip2-devel
patch
xorg-x11-xauth
xterm
texlive
hwloc-devel
sqlite-devel
expat-devel
ncview
perl-open
perl-FindBin
perl-File-Compare
curl-devel
```
Most Linux distributions also have a group of software development packages that can be installed with a single command. For Red Hat-style systems, the command below can be used, but other distributions have similar commands:
```
dnf group install "Development Tools"
```
Load your compiler environment. We have modules set up, but they will be different from other systems:
```
module load intel
```

Download and extract the latest spack-stack from https://github.com/JCSDA/spack-stack/releases:

wget https://github.com/JCSDA/spack-stack/archive/refs/tags/spack-stack-1.9.1.tar.gz
tar zxf spack-stack-1.9.1.tar.gz
cd spack-stack

We use prefx, stamp, and comp so we can copy and paste the commands no matter the version, but for this wiki, maybe we should pick a specific case, like the second block below:

source setup.sh
export prefx='<devel|tags>'
export stamp=' <date of git pull or tag version>'
export comp=' <compiler (i.e., intel or gcc or oneapi)'
export SPACK_DISABLE_LOCAL_CONFIG=true

source setup.sh
export prefx='tag'
export stamp='1.9.1'
export comp='intel'
export SPACK_DISABLE_LOCAL_CONFIG=true

Edit configs/sites/tier2/linux.default/modules.yaml and replace the two tcl occurrences with lmod:
```
 vim configs/sites/tier2/linux.default/modules.yaml
```

Create the environment, make a directory, and activate it:

spack stack create env --compiler ${comp} --site linux.default --name ${prefx}${stamp}_${comp} --prefix /opt/spack-stack

sudo mkdir /opt/spack-stack && sudo chown robertson:admin /opt/spack-stack

spack env activate -p envs/${prefx}${stamp}_${comp}

Edit envs/${prefx}${stamp}_${comp}/site/modules.yaml adding the block below:

vim envs/${prefx}${stamp}_${comp}/site/modules.yaml

    openmpi:
      environment:
        set:
          MPIRUN: '{prefix}/bin/mpirun'
    netcdf-c:
      environment:
        set:
          NETCDF_C: '{prefix}'
          NC_CONFIG: '{prefix}/bin/nc-config'
    netcdf-fortran:
      environment:
        set:
          NETCDF: '{prefix}'
          NF_CONFIG: '{prefix}/bin/nf-config'

Tell Spack which environment to set up:

unset SPACK_DISABLE_LOCAL_CONFIG
export SPACK_SYSTEM_CONFIG_PATH="/opt/src/spack-stack/envs/${prefx}${stamp}_${comp}/site"

Search for installed system packages and compilers. Note that curl is found with the first command; the curl find is unnecessary:

spack external find --scope system --exclude cmake --exclude curl --exclude openssh --exclude openssl --exclude perl --exclude python
spack external find --scope system MySQL
spack external find --scope system bzip2
spack external find --scope system xz
spack external find --scope system sqlite
spack external find --scope system wget

Again, this might be an unnecessary step:

PATH=/usr/lib64/qt5/bin:${PATH} spack external find --scope system qt

Let Spack find your compiler:
```
spack compiler find --scope system
```
The documentation insists you not forget to unset the SPACK_SYSTEM_CONFIG_PATH environment variable:
```
unset SPACK_SYSTEM_CONFIG_PATH
export SPACK_DISABLE_LOCAL_CONFIG=true
```
Intel: Only if your system uses modules to load the compiler:
```
vim envs/${prefx}${stamp}_${comp}/site/compilers.yaml
```
Specify the module to load the Intel compiler:
```
modules: [intel/2021.8.0]
```

In the [email protected] section change flags: {} and environment: {} to the blocks below:

  flags:
    cflags: -diag-disable=10441
    cxxflags: -diag-disable=10441
    fflags: -diag-disable=10441

  environment:
    prepend_path:
      LD_LIBRARY_PATH: /opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin

Set default compiler and MPI library (require line: https://github.com/spack/spack/issues/30730):

spack config add "packages:all:providers:mpi:[[email protected]]"
spack config add "packages:all:compiler:[[email protected]]"
spack config add "packages:all:require:'%intel'"

or:

spack config add "packages:all:providers:mpi:[[email protected]]"
spack config add "packages:all:compiler:[[email protected]]"
spack config add "packages:all:require:'%oneapi'"

or:

spack config add "packages:all:providers:mpi:[[email protected]]"
spack config add "packages:all:compiler:[[email protected]]"
spack config add "packages:all:require:'%oneapi'"

or:

spack config add "packages:all:providers:mpi:[[email protected]]"
spack config add "packages:all:compiler:[[email protected]]"
spack config add "packages:all:require:'%gcc'"

If using the Intel OneAPI MPI, add the mpi details to spack.yaml

vim envs/${prefx}${stamp}_${comp}/spac.yaml

  mpi:
    buildable: False
  intel-oneapi-mpi:
    externals:
    - spec: intel-oneapi-mpi@2021.14%oneapi@2025.0.4
      prefix: /opt/sw/apps/intel/oneapi
      modules:
      - intel/2025.0
      - impi/2021.14
  intel-oneapi-mkl:
    externals:
    - spec: intel-oneapi-mkl@2025.0%oneapi@2025.0.4
      prefix: /opt/sw/apps/intel/oneapi
      modules:
      - intel/2025.0

Tell Spack to use 'Open MPI 4.1.x` so we can debug with TotalView** (We might remove it and have users build with the latest Open MPI).

Edit packages.yaml adding ncview and openmpi:

vim envs/${prefx}${stamp}_${comp}/site/packages.yaml

openmpi:
  version: ['4.1.6']
ncview:
  externals:
  - spec: ncview@2.1.8
    prefix: /usr

Make a directory to store logs:

mkdir envs/${prefx}${stamp}_${comp}/logs

Add the jedi-ufs-env and nco specs. If you add nco to the jedi-ufs-env, skip this.

spack add jedi-ufs-env%intel nco%intel

or:

spack add jedi-ufs-env%gcc nco%gcc

The concretize -f is only needed if concretize has been run before. The --fresh is used so Spack doesn't try to re-use dependencies built by other compilers. It doesn't appear smart enough to know that all dependencies, including Fortran modules, must be rebuilt every time.
```
spack concretize --fresh -f 2>&1 | tee envs/${prefx}${stamp}_${comp}/logs/concretize.log01
```

Build and install:

spack install --verbose --fail-fast 2>&1 | tee envs/${prefx}${stamp}_${comp}/logs/install.log01

Create the LMOD modulefiles:

spack module lmod refresh
spack stack setup-meta-modules

Edit /opt/spack-stack/modulefiles/Core/stack-intel/2021.8.0.lua to auto-load the Stack.

Bulding Spack‐Stack - myroms/roms GitHub Wiki

Instructions

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️