access_ACCESS1.3_raijin - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
Building and running the ACCESS 1.3 coupled model on raijin
Build
The build details are based on Hailin Yan's notes, adapted for raijin. Note that the MOM and CICE builds are the same for the CMIP5 versions of ACCESS1.0 and ACCESS1.3. However the current version of ACCESS1.3 (basis of ACCESS 1.4 development) uses OASIS-MCT.
The instructions here use OASIS3 via the module ~access/modules/oasis3 whereas we originally built it as a separate step in the coupled model build. Scripts all include
module use ~access/modules
module load oasis3/3
The module sets the environment variables for the compiler search paths so that explicit include and library paths are not necessary (similarly for netCDF).
Preparation
Create the directory ACCESSHOME (you can use whatever name you prefer), then check out the /bin directory from cmip5 trunk:
mkdir ACCESSHOME
cd ACCESSHOME
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/bin@[80]
This directory includes a set of executables that you can use. If you want to build any of the executables yourself, create a “submodels” directory under ACCESSHOME
mkdir ACCESSHOME/submodels
CICE4.1
cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/cice/branches/auscom/release-1.0-raijin cice4.1_cmip5
cd cice4.1_cmip5
./comp_access_cice4_RAIJIN 6
Here 6 is the number of cpus allocated for CICE model in the standard ACCESS configuration. Unfortunately CICE must be built for a fixed number.
MOM4p1:
cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/mom4/branches/mom4p1_cmip5_raijin
cd mom4p1_cmip5_raijin/compile
./comp_auscom_mom4p1.RAIJIN
UM
cd ACCESSHOME/submodels
mkdir UM
cd UM
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/compile
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/ummodel_hg3-CABLE
svn co https://access-svn.nci.org.au/svn/um/branches/dev/han32j/VN7.3_ACCESS_coupled_dhb599@[5203] umbase_hg3-CABLE
cd compile
qsub compile_hg3_CABLE
Alternatively, the UM executable can be built with the UMUI with job vagzb (see ACCESS_AMIP_testcases). In this case the executable must be manually copied to the ACCESSHOME/bin directory.
Running the model
All necessary input files for the historical simulation are in directories ~access/data/ACCESS_CMIP5 (model control files), ~access/data/ancil/CMIP5 (CMIP5 emissions) and ~access/data/ancil/access_v2 (land sea mask, vegetation properties etc).
The coupled model control scripts for the original ACCESS 1.3 CMIP5 run are saved at https://access-svn.nci.org.au/trac/cmip5/browser/trunk/exp/hg2-r11Mhd?rev=75. These have been modified to use the files under ~access. The init script has also been modified to allow starting a new run from any year of an existing run (provided the appropriate restart files are in ~access/data/ACCESS_CMIP5/restart/ (this restriction can likely be removed when the /g/data filesystem is available on raijin). At the moment the files for starting the historical run at 1951 are there.
To set up the model run,
mkdir -p $HOME/ACCESS/exp
cd $HOME/ACCESS/exp
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/exp/a13_hist
cd a13_hist
In script a13_hist, change ACCESS_BINDIR to ACCESSHOME/bin where ACCESSHOME is the directory you used for the build. In a13_hist.init, set the run period (iniyear and finalyear). Also, change the executable specification
cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20131017.RAIJIN $ocn_exe
cp -f $ACCESS_BINDIR/cice_MPI1.RAIJIN_6p_201310128 $ice_exe
cp -f $ACCESS_BINDIR/um_hg2-[1].1.exe-20131017 $atm_exe
to match the filenames of your versions. There are two instances of these copy commands in a13_hist.init.
Then to start the run
qsub a13_hist
The model runs in 3 month chunks, each taking about 70 minutes.
Run log files will appear in this directory. The model run directory is
/short/$PROJECT/$USER/work/um_coupled/a13_hist
and results are archived to
/short/$PROJECT/$USER/archive/a13_hist
Note that to change the experiment name, the directory and the 3 files must have names changed consistently.
The experiment setup will have to be modified further to go past 2005.