ELM on ORNL CADES-baseline: environments, building, and running

UPDATED: 2025-07-02 for the most recent E3SM master with modified machine settings

NOTE: this is for CCSI users, with an account associated with Project CLI185 only.

System Docs: https://docs.cades.olcf.ornl.gov/baseline_user_guide/baseline_user_guide.html

I. Environments

I-1. project-wide disk spaces

ONE project home space, with VERY LIMITED disk space, and NOT regularly purged. SO PLEASE DON'T PUT anything large there.

     /ccsopen/proj/cli185

This is for group accessible tools, e.g.

(1) PETSc, ALQUIMIA, etc. for PFLOTRAN users;

    /ccsopen/proj/cli185/petsc-3.x-noopt
    /ccsopen/proj/cli185/alquimia-pflotran/v2021/alquimia
    /ccsopen/proj/cli185/alquimia-pflotran/v2021/pflotran

(2) ATS tools and executables (excluding sources and builds) modulized.

/ccsopen/proj/cli185/ats-dev/

TWO project user accessible spaces, BUT not writable by other than creator. THIS implies that creator must do house-cleaning from time to time.

     /gpfs/wolf2/cades/cli185/proj-shared
     /gpfs/wolf2/cades/cli185/scratch/$USER

ONE all user accessible spaces, BUT not writable by other than creator.

    /gpfs/wolf2/cades/cli185/world-shared

LOGIN node: baseline.ccs.ornl.gov

i.e. access by baseline.ccs.ornl.gov

Note: ...../proj-shared, ...../world-shared, and ......./scratch are for what they are supposed to.

I-2. ENV suggestions

(1) It's suggested to edit your own .bashrc, when first time login.

(2) Suggested .bashrc for ELM currently as following,

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific environment
if ! [ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ](/ORNL-Ecosystem-Projects/Documentations/wiki/-"$PATH"-=~-"$HOME/.local/bin:$HOME/bin:"-)
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions

#############  NOTE: the following are built using mpi wrappers for E3SM land model  #########

module purge

export MYPROJ_HOME=/ccsopen/proj/cli185
export MYPROJ_SHARED=/gpfs/wolf2/cades/cli185/proj-shared
export MYWORLD_SHARED=/gpfs/wolf2/cades/cli185/world-shared
export MYPROJ_SCRATCH=/gpfs/wolf2/cades/cli185/scratch

echo "MYPROJ_HOME: $MYPROJ_HOME"
echo "MYPROJ_SHARED: $MYPROJ_SHARED"
echo "MYWORLD_SHARED: $MYWORLD_SHARED"
echo "MYPROJ_SCRATCH: $MYPROJ_SCRATCH"

# the following 2 modules are gcc-8.5.0 based
module load python/3.11-anaconda3
module load cmake/3.26.3

# above module load  will actually load gcc/12.2.0 and openmpi/4.0.4
module load gcc/12.2.0
module load openmpi/4.0.4
export OPENMPI_DIR=${OLCF_OPENMPI_ROOT}

module load openblas/0.3.23
export BLASLAPACK_LIBDIR=${OLCF_OPENBLAS_ROOT}/lib

# (optional) NCO tool package
module load nco

# libs for ELM and PFLOTRAN
module load hdf5/1.14.3
module load netcdf-c/4.9.2-h5f
module load netcdf-fortran/4.6.1-h5f
module load netcdf-cxx/4.2
module load parallel-netcdf/1.12.3
export HDF5_PATH=${OLCF_HDF5_ROOT}
export NETCDF_C_PATH=${OLCF_NETCDF_C_ROOT}
export NETCDF_FORTRAN_PATH=${OLCF_NETCDF_FORTRAN_ROOT}
export NETCDF_CXX_PATH=${OLCF_NETCDF_CXX_ROOT}
export PNETCDF_PATH=${OLCF_PARALLEL_NETCDF_ROOT}

export PETSC_DIR=$MYPROJ_HOME/petsc-3.x-noopt
export ALQUIMIA_PATH=$MYPROJ_HOME/alquimia-pflotran/v2021/alquimia

# D.R. compiled perl5 with Lib2XML, which required by ELM namelist build
export PATH=/ccsopen/home/zdr/opt/perl/bin:$PATH

I-3. E3SM codes & Input data

for CADES-baselibe machine settings

There is specific branch for our system, https://github.com/E3SM-Project/E3SM.git, branch fmyuan/ornl-cades-baseline-machine-settings trunk E3SM for cades-baseline

for example, if you'd like model to be in your proj-shared directory like:

cd /gpfs/wolf2/cades/cli185/proj-shared/$USER/models/
git clone https://github.com/E3SM-Project/E3SM.git

cd E3SM
git cherry-pick c17b050504ac88ea7ef9a333cfff347a108ee4d1
(this will 'cades-baseline' machine settings)

E3SM external submodule initiate and update

There are a bunch of external models as E3SM's submodules, which must be updated after git clone source codes above.

git submodule init
git submodule update -f --recursive

VERY likely, the updating will have troubles when you first time to do so.

The suggested methods to solve this issue is to set up a ssh authorization key in your github.com account, and then copy its public key to your cades-baseline home. It's little bit inconvenient. Alternatively, simply edit the file under 'E3SM/.git/config', replacing all '[email protected]:' to 'https://github.com/', or using the following perl command:

cd .git
perl -pi -e 's/com:/com\//g' ./.git/config
perl -pi -e 's/git@/https:\/\//g' ./.gitconfig

E3SM input data on CADES-baseline

/gpfs/wolf2/cades/cli185/world-shared/e3sm/inputdata/

We try to keep a minimal but full copy of E3SM inputdata in this folder.

Additionally, for site-level or project-specific purpose, the following site-specific input data may be used.

/gpfs/wolf2/cades/cli185/proj-shared/pt-e3sm-inputdata

+++++++++++++++++++++++++++++++++++

II. CASE 1: single-point user-defined datm-clm (3 Examples).

-----------------------------------------------------------------

NOTE: For ELM CUSTOMIZED configuration with options of CPL_BYPASS & User-defined PFT

NOTE: For this example and other to work, it is needed to clone pt-e3sm-inputdata repository