Modules: Advanced - nthu-ioa/cluster GitHub Wiki

Adding new modules and modulefiles

MODULEPATH

Commands like module load are made available by functions that get added to the user's shell environment on login [admin note: this is done by (symlinks to) scripts in /etc/profile.d]. As well as shell functions, these scripts set up some environment variables used to manage Modules itself. The most important of these is MODULEPATH, which stores a list of directories that Modules will search for modulefiles.

The system-wide modulefiles for Fomalhaut are stored in /cluster/software/modulefiles. This is the only entry in MODULEPATH by default.

Users are encouraged to write their own modulefiles for software they have built in their homespace or elsewhere. To do this, you first need to create a directory (e.g. ~/modulefiles) and add it to your MODULEPATH (e.g. in ~/.profile). Any modules in user-defined paths will then show up in the output of module avail. They are listed separately to the system-wide modules, but otherwise work in exactly the same way.

Tip: in your personal modulefile directories, you are strongly recommended to follow the same modulename/version layout as we use the system directory.

Adding new modules

The easiest way to write modulefiles is to copy and modify an existing example for a similar package (e.g. those in /cluster/software/modulefiles). Usually only two or three lines need to be edited, unless the library or package is very complicated.

Modulefile location

Modulefiles are collections of functions in the Tcl scripting language, that are run when the module is loaded. The Modules package defines many common functions (see man modulefiles). Extra commands, written in Tcl, can be defined and imported. All directories under $MODULEPATH are searched for modulefiles.

Modules takes advantage of a conventional layout for modulefile directories to give sensible names to modules, so it's important to follow those conventions. On Fomalhaut, the layout we use is:

/cluster/software/modulefiles/{module}/{version}

  • module is a directory (e.g. gcc/)
  • version is the modulefile (e.g. 8.3.0)

Note: by default, module load some_software, with no version number, will load the 'highest' version number modulefile available for some_software. A different default choice can also be specified (see the docs).

Modulefile structure

As an example, we'll dissect the gcc/8.3.0 module:

#%Module#####################################################################
##
## gnu_comp modulefile
##
# for Tcl script use only
set name    gcc
set version 8.3.0 
  • Lines starting with # are comments in Tcl.
  • set name and set version declare constants (for the name and version of the module) that will be substituted into strings later on. This minimizes the number of changes required when copying modulefiles to make new modules.
# Import some extra TCL scripts for Fomalhaut
set base  /cluster/software
source    $base/modulefiles/.common
source    $base/modulefiles/.compiler_tools
  • This imports two sets of Tcl functions that are used later on. The functions in .common are provided by the Modules system to work with module flavours (see below). Those in .compiler_tools are just macro-like functions to save typing when setting things like LD_RUN_PATH for libraries. The set base is because we have to specify the full path to these extra source files. (The filenames start with a dot because they live alongside modulefiles, but we don't want Modules to see them as modules.)
conflict $name
conflict intel
conflict pgi

These tell Modules which other modules can't be loaded at the same time as the current one. conflict $name means different versions of this module can't be loaded together. The other modules here are other brands of compiler.

# compute installation path based on selected toolchain and flavors
set prefix $base/[getInstallPrefix]

This is important. The function [getInstallPrefix] (from .common) figures out that this package is installed under $name/$version, to which we prepend $base to get the absolute path (The getInstallPrefix function is necessary because more complicated situations can occur with module flavours; we don't hardcode $base in getInstallPrefix for a little extra flexibility).

# set software environment
prepend-path PATH $prefix/bin
prepend-path MANPATH $prefix/share/man

This is the core business of modules, modifying some basic environment variables with the builtin function prepend-path.

# Suggested flags for our cluster
setenv          CLUSTER_C_OPT "-O2 -march=native"

# Set compilers to use
setenv          GNUROOT          $prefix
setenv          CC               $prefix/bin/gcc
setenv          CXX              $prefix/bin/g++
setenv          FC               $prefix/bin/gfortran
setenv          F77              $prefix/bin/gfortran
setenv          F90              $prefix/bin/gfortran
setenv          F9X              $prefix/bin/gfortran

In the case of gcc there are a lot of other useful environment variables we can define, like $CC. These will all be unset when we unload the module (but note, not restored to their original state -- doing that is not trivial, because the state has to be saved somehow).

# Add library search paths
AddLibraryDir $prefix/lib64/

# Add header file search paths
AddHeaderDir $prefix/include

# Add Fortran modules path
AddF90ModuleDir $prefix/include

These call functions from .compiler_tools that provide shortcuts for writing things like:

prepend-path    --delim { } LDFLAGS -Wl,-rpath=/cluster/software/gcc/8.3.0/lib64/

Try module show gcc to see the full list.

proc ModulesHelp { } {
        global version prefix
        puts stderr "\t\Loads the GNU compiler environment"
        puts stderr "\n\tUse this module to compile code to run using the"
        puts stderr "\tGNU compiler version ${version}.\n"
        puts stderr "\tThe compiler commands are:\n"
        puts stderr "\tC compiler       : gcc"
        puts stderr "\tC++ compiler     : g++"
        puts stderr "\tFortran compiler : gfortran"
        puts stderr "\n"
}

module-whatis   "loads the GNU compiler environment"

The last commands show how to add inline documentation and a simple description string. This information is optional.

Building and installing code for a new module

To make use of Modules, each software package/version should be installed to an isolated directory tree, rather than putting everything into a common set of directories under /usr/local, /opt or similar.

For an easy life, it's recommended to install to directories that follow the same conventional layout as the corresponding modulefiles.

For example, the system-wide modules are installed to /cluster/software/{module}/{version}. In this case version is also a directory (substituting for /usr/local or similar), corresponding to the modulefile in /cluster/software/modulefiles/{module}/{version}.

For code that builds with configure, installing into a custom directory is usually easy to do with a --prefix argument:

./configure --prefix=/cluster/software/foo/1.2.0/
make
make install

will give you /cluster/software/foo/1.2.0/bin, /cluster/software/foo/1.2.0/lib, /cluster/software/foo/1.2.0/man etc.. A different version can then be installed under /cluster/software/foo/2.1.0.

Users can do the same thing under paths they can write to (e.g. /cluster/home/evergiven/modules).

Toolchains

Most libraries will only have one element in their toolchain, the compiler. Some are more complex. For example, the hdf-parallel module has different builds for different compilers and different versions of MPI.

For example, one such build is installed in: /cluster/software/hdf5-parallel/1.8.21/gcc--8.3.0/openmpi--3.1.4/

In this case the module toolchain has two elements: gcc--8.3.0 and openmpi--3.1.4. Modules automatically maps these to modules with the names gcc/8.3.0 and openmpi/3.1.4.

The modulefile specifies the elements of the toolchain like this:

# builds available for following compiler
set buildcompiler [availForBuildModule compiler gcc/8.3.0]
# builds available for following mpi
set buildmpi [availForBuildModule mpi openmpi/3.1.4]

Most modules will only need the set buildcompiler line. If the module was built for two different compilers, it would look like:

set buildcompiler [availForBuildModule compiler gcc/8.3.0 intel/14.7]

If a module depends on some other toolchain element foo provided by the modular bar/1.0, just add another similar line as follows: set buildfoo [availForBuildModule foo bar/1.0]

Flavors

Flavor is a new feature of the latest version of Modules that add an extra layer of complexity. Currently we don't use the flavor system.