Building and running the ACCESS 1.3 coupled model on raijin

Build

The build details are based on Hailin Yan's notes, adapted for raijin. Note that the MOM and CICE builds are the same for the CMIP5 versions of ACCESS1.0 and ACCESS1.3. However the current version of ACCESS1.3 (basis of ACCESS 1.4 development) uses OASIS-MCT.

The instructions here use OASIS3 via the module ~access/modules/oasis3 whereas we originally built it as a separate step in the coupled model build. Scripts all include

module use ~access/modules
module load oasis3/3

The module sets the environment variables for the compiler search paths so that explicit include and library paths are not necessary (similarly for netCDF).

Preparation

Create the directory ACCESSHOME (you can use whatever name you prefer), then check out the /bin directory from cmip5 trunk:

mkdir ACCESSHOME
cd ACCESSHOME
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/bin@[80]

This directory includes a set of executables that you can use. If you want to build any of the executables yourself, create a “submodels” directory under ACCESSHOME

mkdir ACCESSHOME/submodels

CICE4.1

cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/cice/branches/auscom/release-1.0-raijin cice4.1_cmip5
cd cice4.1_cmip5
./comp_access_cice4_RAIJIN 6

Here 6 is the number of cpus allocated for CICE model in the standard ACCESS configuration. Unfortunately CICE must be built for a fixed number.

MOM4p1:

cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/mom4/branches/mom4p1_cmip5_raijin
cd mom4p1_cmip5_raijin/compile
./comp_auscom_mom4p1.RAIJIN

UM

cd ACCESSHOME/submodels
mkdir UM
cd UM

svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/compile
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/ummodel_hg3-CABLE
svn co https://access-svn.nci.org.au/svn/um/branches/dev/han32j/VN7.3_ACCESS_coupled_dhb599@[5203] umbase_hg3-CABLE

cd compile

qsub compile_hg3_CABLE

Alternatively, the UM executable can be built with the UMUI with job vagzb (see ACCESS_AMIP_testcases). In this case the executable must be manually copied to the ACCESSHOME/bin directory.

Running the model

All necessary input files for the historical simulation are in directories ~access/data/ACCESS_CMIP5 (model control files), ~access/data/ancil/CMIP5 (CMIP5 emissions) and ~access/data/ancil/access_v2 (land sea mask, vegetation properties etc).

The coupled model control scripts for the original ACCESS 1.3 CMIP5 run are saved at https://access-svn.nci.org.au/trac/cmip5/browser/trunk/exp/hg2-r11Mhd?rev=75. These have been modified to use the files under ~access. The init script has also been modified to allow starting a new run from any year of an existing run (provided the appropriate restart files are in ~access/data/ACCESS_CMIP5/restart/ (this restriction can likely be removed when the /g/data filesystem is available on raijin). At the moment the files for starting the historical run at 1951 are there.

To set up the model run,

mkdir -p $HOME/ACCESS/exp
cd $HOME/ACCESS/exp
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/exp/a13_hist
cd a13_hist

In script a13_hist, change ACCESS_BINDIR to ACCESSHOME/bin where ACCESSHOME is the directory you used for the build. In a13_hist.init, set the run period (iniyear and finalyear). Also, change the executable specification

  cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20131017.RAIJIN 	$ocn_exe
  cp -f $ACCESS_BINDIR/cice_MPI1.RAIJIN_6p_201310128   $ice_exe
  cp -f $ACCESS_BINDIR/um_hg2-[1].1.exe-20131017        $atm_exe

to match the filenames of your versions. There are two instances of these copy commands in a13_hist.init.

Then to start the run

qsub a13_hist

The model runs in 3 month chunks, each taking about 70 minutes.

Run log files will appear in this directory. The model run directory is

/short/$PROJECT/$USER/work/um_coupled/a13_hist

and results are archived to

/short/$PROJECT/$USER/archive/a13_hist

Note that to change the experiment name, the directory and the 3 files must have names changed consistently.

The experiment setup will have to be modified further to go past 2005.