Maintaining Spack Buildcache - GEOS-ESM/MAPL GitHub Wiki

This page details the efforts to create a buildcache for use in Spack CI tests in MAPL.

Useful Links

• Spack Buildcache Documentation
• setup-spack GitHub Action
• Possible Example

Building Packages on AWS

Preliminaries

Padded Length

Testing showed we apparently need to increase the length of the padding in the install tree. This is what I ran:

spack config add config:install_tree:padded_length:256

Installing GCC 15

I need to move the build cache to gcc 15. So I need to install it on ubuntu:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt update
sudo apt install gcc-15 g++-15 gfortran-15 -y

toolchains.yaml

I have a toolchains.yaml that looks like this:

toolchains:
  gcc-gfortran-14:
  - spec: "%[email protected]"
    when: "%c"
  - spec: "%[email protected]"
    when: "%cxx"
  - spec: "%[email protected]"
    when: "%fortran"
  gcc-gfortran-15:
  - spec: "%[email protected]"
    when: "%c"
  - spec: "%[email protected]"
    when: "%cxx"
  - spec: "%[email protected]"
    when: "%fortran"

packages.yaml

My current packages.yaml is set as:

packages:

  gcc:
    externals:
    - spec: [email protected] languages='c,c++,fortran'
      prefix: /usr
      extra_attributes:
        compilers:
          c: /usr/bin/gcc
          cxx: /usr/bin/g++
          fortran: /usr/bin/gfortran
        flags: {}
        environment: {}
        extra_rpaths: []
    - spec: [email protected] languages='c,c++,fortran'
      prefix: /usr
      extra_attributes:
        compilers:
          c: /usr/bin/gcc-14
          cxx: /usr/bin/g++-14
          fortran: /usr/bin/gfortran-14
        flags: {}
        environment: {}
        extra_rpaths: []
    - spec: [email protected] languages:='c,c++,fortran'
      prefix: /usr
      extra_attributes:
        compilers:
          c: /usr/bin/gcc-15
          cxx: /usr/bin/g++-15
          fortran: /usr/bin/gfortran-15
    buildable: false

  all:
    require: target=x86_64_v3
    providers:
      mpi: [openmpi, intel-oneapi-mpi]
      blas: [openblas, intel-oneapi-mkl]
      lapack: [openblas, intel-oneapi-mkl]

  gcc:
    buildable: false
  openmpi:
    require:
    - any_of: ['%gcc']
      message: Only use Open MPI with GCC

  intel-oneapi-mpi:
    require:
    - any_of: ['%oneapi']
      message: Only use Intel MPI with Intel

  hdf5:
    variants: +fortran +szip +hl +threadsafe +mpi
    # Note that cdo requires threadsafe, but hdf5 doesn't
    # seem to want that with parallel. Hmm.
  netcdf-c:
    variants: +dap
  esmf:
    variants: ~pnetcdf ~xerces
  cdo:
    variants: ~proj ~fftw3
    # cdo wanted a lot of extra stuff for proj and fftw3. Turn off for now
  pflogger:
    variants: +mpi
  pfunit:
    variants: +mpi +fhamcrest
  fms:
    require: '@2024.03 ~gfs_phys +pic constants=GEOS precision=32,64 +deprecated_io ~yaml target=x86_64_v3'
    # NOTE: I have to re-add the target here otherwise spack seemed to not build for a generic processor
  mapl:
    variants: +extdata2g +fargparse +pflogger +pfunit ~pnetcdf

repos.yaml

Spack is now split with packages in a separate repo. To be consistent, we manually clone and put it in a place easy to maintain. As such, our repos.yaml is:

repos:
  builtin:
    git: [email protected]:spack/spack-packages.git
    destination: /home/ubuntu/spack-packages
  geosesm: /home/ubuntu/geosesm-spack

Building the Buildcache (ESMF 9, GCC 15)

ESMF 9

There is now a need to have ESMF 9 in a buildcache. But, testing shows that you can't really have two versions of ESMF in the same Spack environment. You should, but the fact that ESMF v9.0.0b11 is not a "real" release but a git commit makes it tricky.

So, let's make a new Spack environment and buildcache just for ESMF 9.

First, I did spack edit esmf and added the 9.0.0b11 version to the package.py file (see below).

diff --git a/repos/spack_repo/builtin/packages/esmf/package.py b/repos/spack_repo/builtin/packages/esmf/package.py
index 295f2c42de..1b20392bdc 100644
--- a/repos/spack_repo/builtin/packages/esmf/package.py
+++ b/repos/spack_repo/builtin/packages/esmf/package.py
@@ -33,6 +33,7 @@ class Esmf(MakefilePackage, PythonExtension):
     # Develop is a special name for spack and is always considered the newest version
     version("develop", branch="develop")
     # generate chksum with 'spack checksum esmf x.y.z'
+    version("9.0.0b11", tag="v9.0.0b11")
     version("8.9.1", sha256="e3fafd0c057bf1c3b4c41c997b392016d621b1f1a7c601355c325a7f58425d78")
     version("8.9.0", sha256="586e0101d76ff9842d9ad43567fae50317ee794d80293430d9f1847dec0eefa5")
     version("8.8.1", sha256="b0acb59d4f000bfbdfddc121a24819bd2a50997c7b257b0db2ceb96f3111b173")

Create Spack Environment

It turns out, you can't build or push a buildcache unless you are in a Spack environment. So we need to create one:

spack env create geos-buildcache-esmf9-gcc15
spack env activate -p geos-buildcache-esmf9-gcc15

Add Packages to Environment

spack spec geosgcm-deps %gcc-gfortran-15 ^[email protected]

Check to make sure that has the right ESMF version. Then:

spack add geosgcm-deps %gcc-gfortran-15 ^[email protected]
spack install geosgcm-deps %gcc-gfortran-15 ^[email protected]

Add mirror for buildcache

spack mirror add --oci-username mathomp4 --oci-password-variable GEOS_BUILDCACHE_TOKEN --unsigned geos-buildcache oci://ghcr.io/GEOS-ESM/geos-buildcache-esmf9-gcc15

Pushing the Buildcache

spack buildcache push --update-index oci://ghcr.io/GEOS-ESM/geos-buildcache-esmf9-gcc15