installed software build scripts - raeker/ARC-Wiki-Test GitHub Wiki

ARC Installed software build scripts

Great Lakes cluster

Scripts for building software in /sw/arcts/centos7 on the Great Lakes cluster are maintained in the git repository, arc_glb_build_scripts, on Bitbucket.  See https://bitbucket.org/um_flux_software/arc_glb_build_scripts/src/master/.  Installers in the UM_Flux_software group on Bitbucket should clone the git repository to /var/software/${USER} where ${USER} is the installers username for their *sw account:

    git clone [email protected]:um_flux_software/arc_glb_build_scripts.git

The *sw accounts must be used in order to have write permission in the /sw/arcts/centos7 directory, where software managed by Lmod reside.  

Build scripts on Great Lakes have a specific format that requires sourcing a common configure script, build.cfg, providing a number of functions that automatically determine installation paths, creation of log directories, etc.  

The installer should set the variable SCRIPT_ROOT in their .bashrc file to point to the cloned build script repository as follows:

    export SCRIPT_ROOT=/var/software/${USER}/arc_glb_build_scripts

With SCRIPT_ROOT set in a users .bashrc file, each build script begins by confirming that the variable is set.  Then, the build.cfg file is sourced to make predefined build functions available to the package build script:

   source $SCRIPT_ROOT/build.cfg

Within each package build script, modules are purged, correct ones loaded, and then loaded modules are listed.  The STACKS variable must be set to "yes" if the module will be part of a module hierarchy (described below) or "no" if not.  Depending on the given package, an extension for the source file (TAR_TYPE) may be set, as well as any special CONFIG_OPTIONS.  A basic installation, without any unusual steps or naming conventions, would then call the build_setup function from build.cfg which will set up the build directory, set up the install directory, and unpack the tar ball before continuing with the remaining package specific build steps.   Following the standard build script approach, settings are confirmed before proceeding with installation and log files are saved when installation completes. 

Graphical depiction of the software installation hierarchy

Installation paths for software packages that depend on a specific compiler and MPI versions

We have chosen to organize our package installation tree based on package dependencies such as compiler type and version and MPI type where relevant, similar to the way our modulefiles hierarchy has been implemented.  All examples assume you start at the top of the stacks directory.  This is what the stacks directory tree looks like with only a few applications shown for example.

/sw/arcts/centos7

stacks
├── gcc
│   ├─── 4.8.5
│   │    └── hdf5
│   │    └── openmpi
│   └─── 8.2.0
│        ├── hdf5
│        ├── openmpi
│        └── mpi-pkgs
│             ├── impi
│             │   └── 2018.4.274
│             │       └── hdf5
│             └── openmpi
│                 └── 3.1.4
│                     └── hdf5
├── intel
│   ├─── 14.0.2
│   │    ├── hdf5
│   │    ├── openmpi
│   │    └── mpi-pkgs
│   │        └── openmpi
│   │            └── 3.1.4
│   │                └── fftw
│   ├─── 18.0.5
│   │    ├── hdf5
│   │    ├── openmpi
│   │    └─── mpi-pkgs
│   │         ├── impi
│   │         │   └── 2018.4.274
│   │         │       └── hdf5
│   │         └── openmpi
│   │             └── 3.1.4
│   │                 └── hdf5
│   └─── 19.1
│        └── hdf5
│        └── openmpi
└── pgi
    └─── 19.1
         └── openmpi
⚠️ **GitHub.com Fallback** ⚠️