access_tildeAccess - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

~access is the shared directory for all users of the ACCESS system. Files that get used by multiple different groups should get installed here. It is for permanent, long term storage.

PageOutline

Directories

~access/apps

apps is where ACCESS-specific progams & utility libraries are stored. It uses the same structure as the supercomputer's root /apps directory - programs are installed to directories named after the program name & version. For instance version 1.2.3 of the foo library would be installed to the path ~access/apps/foo/1.2.3. Researchers shouldn't access these programs directly - instead they should load the app into their environment using the module system.

~access/modules

In order to load a program a user should load it into their environment using the module command. The module system will look for a corresponding script in ~access/modules which defines the environment variables to set. This means that we can avoid hard-coding paths - to change the version of a library getting used by a simulation you just need to change the module that gets loaded.

A module is configured using a plain-text file, stored in a directory named after the program or library. For instance to find the module file for foo/1.2.3 you would look in the directory ~access/modules/foo for the file named 1.2.3.

The most basic module file looks like

#%Module

set help            "Library for foo-ing"
set install-contact "Scott Wales <[email protected]>"
set install-date     2013-07-23
set url              foo.example.com
set prefix           /projects/access/apps/foo/1.2.3

source               /projects/access/modules/common

The module file includes a small amount of meta-data, including who installed it and where it came from. It also sets a variable prefix, which defines where the library was installed. The final thing any module should do is to source the file ~access/modules/common. This file uses the prefix variable to set up the user environment, setting variables including:

  • PATH: applications ($prefix/bin)
  • CPATH: headers ($prefix/include)
  • LD_RUN_PATH: libraries ($prefix/lib)
  • MANPATH: documentation ($prefix/man)

To add additional PATHs you can use the prepend-path command in the module file. For instance a Python library may wish to add to the module file:

prepend-path PYTHONPATH $prefix/lib/python2.7/site-packages

When the module gets loaded the directory is added to the PATH variable, when it is unloaded it gets removed.

~access/data

Larger model input files, such as ancillary data for UM runs, gets placed in the data directory. Currently this is a mirror of vayu:/data/projects/access, a hand edit exists on accesscollab to make a UM job use this new format (~access/raijin/data-paths.sh).

~access/umdir

Files & scripts specifically for the UM. This includes the small execs and default run scripts for each model version.

Installed Modules

Top level ACCESS modules include:

  • cdat-lite: Python library for climate data processing
  • use version 6.0rc2-fixed. See https://github.com/MartinDix/cdat_lite_test for the fixes in this version.
  • cylc: Suite scheduler
  • fcm: Met Office build system
  • gcom: MPI abstraction library
  • iris: Python library for climate data processing
  • rose: Model configuration tool
  • um: UM environment settings
  • This sets the $DATAOUTPUT and $UMDIR environment variables
  • oasis3: Coupling library
  • oasis3-mct: Coupling library
  • mars: mars client for accessing OpenDAP archive on raijin
  • first export GRB_ROOT=/path/to/data/archive, default is /short/$PROJECT/$USER/opendap
  • ar_mars.sh to archive, where file must be pxnnn in grib format, with filename as per output from the model, and x = a..g but not pi
  • mars.sh to retrieve. Set GRB_ROOT to access other users' archive, eg. /share/dp9/yix548/opendap
  • ODB:
  • odb/32.0: based on ECMWF version 32 for use in APS0/1
  • odb/29.1: updated version to match lastest versions from UKMO and ECMWF for use in OPS29.1+
  • cawcr-utils: various cawcr-utils such as um/grib/bufr utils and metview4

To use these you must first enable the access modules, then load them normally:

module use ~access/modules
module load rose

Additionally a number of python libraries have been installed, generally as pre-requisites for other models.

  • Cython
  • PIL
  • ScientificPython
  • Shapely
  • cartopy
  • mock
  • netCDF4
  • nose
  • pep8
  • pyke
  • pyshp

These are in the pythonlib subdirectory, load them either like

module load pythonlib/netCDF4

or do a 'module use' on the pythonlib directory

module use ~access/modules/pythonlib
module load netCDF4

Requesting modules

To request a program or library be installed please email [email protected]

Installing modules

Members of the access.admin group are able to set up modules in the ~access space, following the directory structure above.

Python libraries can be installed using a special script

~access/admin/install-python-lib.sh LIBNAME

This script will download and install a library from PyPI - if there are no prerequisites this happens automatically. If the library uses C code that needs to be compiled this should use gcc, rather than the intel compilers (the intel compilers cause library conflicts)