Introduction

NERSC consultants will have access to a pseudo-user by the name of swowner. The purpose of swowner is to install packages under /global/common/sw/ or other areas where software is installed under admin permissions. This guide will highlight some of the features available in Spack and also run you through an example workflow. The goal of this guide is to help consultants develop some kind of workflow as well as become a reference guide to users.

Installing with Spack

Spack Packages

Hopefully your package is popular enough to already have a package.py already written. If you're not sure whether your package is already in the Spack repo then do:

$ spack list

Spack has about > 2000 packages available (as of 9/10/2018). Let's assume that you are lucky and your package is available, we'll go through the basics of package writing in a different section but just be aware that package file writing can get pretty complicated if your package's install is complicated. Like all things software, you're better off avoiding package file development and let someone else do it but sometimes you'll be that someone else.

Specifying compilers on command line

You won't have to detect compilers present on either Cori or Edison since a compilers.yaml is already generated on swowner. In order to use a specific compiler AND programming environment, it is enough just to:

$ spack install [package]%[compiler]

This will swap PrgEnv modules to the correct environment as specified in your spec.

Configuration files

We have a couple of site-wide defaults available in the swowner instance of Spack, they are packages.yaml and config.yaml. The packages.yaml configuration file is there to help link with some of the Cray packages that are present on the systems. Here is how packages.yaml and config.yaml would look like:

packages:
    all:
        compiler: [[email protected], [email protected], [email protected]]
        variants: ~shared
        craype_link_type: "static"
        providers:
            mpi: [mpich, openmpi]
            mkl: [intel-mkl]
            blas: [intel-mkl, cray-libsci]
            scalapack: [intel-mkl, cray-libsci]
            pkgconfig: [pkg-config]
    pkg-config:
        buildable: false
        paths:
            [email protected]: /usr
    cmake:
        buildable: false
        paths:
            [email protected]: /usr
    mpich:
        buildable: false
        modules:
            [email protected]%gcc: cray-mpich
            [email protected]%intel: cray-mpich
            [email protected]%cce: cray-mpich
    intel-mkl:
        buildable: false
        paths:
            [email protected]%intel: /opt/intel
            [email protected]%gcc: /opt/intel
            [email protected]%cce: /opt/intel
    fftw:
        buildable: false
        modules:
            [email protected]%gcc+openmp: cray-fftw
            [email protected]%intel+openmp: cray-fftw
            [email protected]%cce+openmp: cray-fftw

config:
  # This is the path to the root of the Spack install tree.
  # You can use $spack here to refer to the root of the spack instance.
  # install_tree: /global/common/sw   # Change to the desired installation

  # Locations where templates should be found
  template_dirs:
    - $spack/templates

  # This is the scheme that will be the default for NERSC.
  install_path_scheme: "${PLATFORM}/${OS}/${TARGET}/${PACKAGE}/${VERSION}/${COMPILERNAME}/${COMPILERVERSION}/${HASH:7}"  # NoQA: E501

  # Locations where different types of modules should be installed.
  module_roots:
    tcl:    $spack/share/spack/modules
    lmod:   $spack/share/spack/lmod
    dotkit: $spack/share/spack/dotkit

Note:

There are slight differences between our fork and the upstream fork. If you noticed, we have global variants enabled and we also have an additional entry called craype_link_type. Because we (NERSC), statically link our packages we need to make sure that we are not linking in dynamic libraries which can cause errors. We also make sure to make the cray compiler wrappers are using static linking.

Installing Your Package (A workflow example)

If you are interested in installing a package, the first thing you need to do is to usgrsu as the swowner pseudo-user.

$: usgrsu swowner

If you want to use spack's shell commands you will want to activate the setup-env.sh script. This will activate some of spack's shell functionality (e.g spack cd, spack load etc).

WARNING: There is a bug that will cause your session to hang and enter an infinite loop if you put the following line in your ~/.bashrc.ext. So it is best that you avoid having that line in your bashrc.

$swowner: . spack_root/share/spack/setup-env.sh

To install a package you want to invoke the following command:

$swowner: spack install [package_name]

For a more concrete example, let's try to install gromacs:

$swowner: spack install gromacs

Spack has a spec syntax that allows you to have full control of the combinatorial space of a package. You can specify the compiler, package version, variants, and architecture. For more information check the Spack Docs

Once the command is executed, Spack will go on and install the entire software stack for that package. This includes dependencies of the package as well as dependencies of those dependencies.

The installation will then complete, and you will have your package installed under /global/common/sw. This is the root installation path. The full installation path will be deeper than this. You may now start to worry that the path will be difficult to remember but there a commands to make it easier for you to find the location of a particular package.

To find where all packages are installed, you want to use the following command:

$swowner: spack find -p

which will produce this output:

==> 8 installed packages.                                                                                                               │
-- cray-cnl6-ivybridge / [email protected] --------------------------------                                                                       │
    [email protected]        /global/common/sw/cray/cnl6/ivybridge/cmake/3.8.1/gcc/6.3.0/tft5myv                         │
    [email protected]       /opt/cray/pe/fftw/3.3.6.1/ivybridge                                                                                │
    [email protected]     /global/common/sw/cray/cnl6/ivybridge/gromacs/2016.3/gcc/6.3.0/nygucva                      │
    [email protected]        /opt/cray/pe/mpt/7.4.4/gni/mpich-gnu/5.1                                                                         │
    [email protected]        /global/common/sw/cray/cnl6/ivybridge/ncurses/6.0/gcc/6.3.0/g5ryw7v                         │
    [email protected]     /global/common/sw/cray/cnl6/ivybridge/openssl/1.0.2k/gcc/6.3.0/yvdij4x                      │
    [email protected]  /global/common/sw/cray/cnl6/ivybridge/pkg-config/0.29.2/gcc/6.3.0/4mgeofl                   │
    [email protected]        /global/common/sw/cray/cnl6/ivybridge/zlib/1.2.11/gcc/6.3.0/hmujgbu

The command with the -p flag allows us to see the path for the installation of all these packages. You can locate where a single package is installed by providing a spec, like so:

$swowner: spack find -p openssl

If you are interested in moving to the install location and have enabled shell-support you can 'cd' into the directory quite easily.

$swowner: spack cd -i gromacs

This will then cd you into the installed directory.

Specifying The Target

Since we are going to be building for Edison and Cray systems, it is very important to build against the right architecture. Cray has compiler wrappers that take care of all the architecture optimizations. To use the right compiler wrapper you will want to specify the target spec. Here is an example:

$: spack install gromacs target=ivybridge

Spack defaults to the backend (compute node) architecture. So the defaults for Edison and Cori are ivybridge and haswell, respectively. Since they are defaults, you don't have to explicitly declare them on the CLI.

To compile for the mic-knls you will want to explicitly tell Spack to compile for it:

$: spack install gromacs target=mic_knl

Building for the frontend (login nodes) is a bit different. Since Spack defaults to compute nodes, you will want explicitly tell Spack to build for the frontend. At the moment, this is a bit clunky and requires that the user enter in both the operating system AND the target. An example is given below:

$: spack install vim os=fe target=fe

Spack has reserved keywords for the frontend and backend (fe and be, respectively), so the user doesn't need to know specifics of the architecture. So the previous examples could also be written as:

$: spack install gromacs target=be

Creating the Modulefiles

Once you have installed a package, you will want to create a modulefile. Modulefiles will be placed in /global/homes/s/swowner/software/cori_modulefiles. Here you can create a modulefile under version control without doing modifications to the production environment. To create a module you will run a script called make_modulefile.sh. An interactive prompt will run you through the modulefile creation.

*TODO: Run through an example of modulefile creation.

Once that modulefile is completed, you will want to test it out. To test out your newly created modulefile run module use on /global/homes/s/swowner/software/cori_modulefiles/package. This will add the package to your modules environment. You can test this by loading the module and attempting to run commands to see if there are any issues with the package or with permissions.

After testing the modulefile, you will want to add and commit the modulefile. Modules are version controlled using subversion. You will want to svn add package/ and then svn commit -m "Commit message". This will push your changes to production and you will see your modulefile present on the system.