access_ACCESS1.0_raijin - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
Building and running the ACCESS 1.0 coupled model on raijin
Build
The build details are based on Hailin Yan's notes, adapted for raijin. Note that the MOM and CICE builds are the same for the CMIP5 versions of ACCESS1.0 and ACCESS1.3. However the current version of ACCESS1.3 (basis of ACCESS 1.4 development) uses OASIS-MCT.
The instructions here use OASIS3 via the module ~access/modules/oasis3 whereas we originally built it as a separate step in the coupled model build. Scripts all include
module use ~access/modules
module load oasis3/3
The module sets the environment variables for the compiler search paths so that explicit include and library paths are not necessary (similarly for netCDF).
Preparation
Create the directory ACCESSHOME (you can use whatever name you prefer), then check out the /bin directory from cmip5 trunk:
mkdir ACCESSHOME
cd ACCESSHOME
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/bin@[80]
Create “submodels” directory under ACCESSHOME
mkdir ACCESSHOME/submodels
CICE4.1
cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/cice/branches/auscom/release-1.0-raijin cice4.1_cmip5
cd cice4.1_cmip5
./comp_access_cice4_RAIJIN 6
Here 6 is the number of cpus allocated for CICE model in the standard ACCESS configuration. Unfortunately CICE must be built for a fixed number.
MOM4p1:
cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/mom4/branches/mom4p1_cmip5_raijin
cd mom4p1_cmip5_raijin/compile
./comp_auscom_mom4p1.RAIJIN
UM
cd ACCESSHOME/submodels
mkdir UM
cd UM
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/compile
svn co https://access-svn.nci.org.au/svn/um/branches/dev/han32j/VN7.3-HadGEM2-1.1 umbase_hg2-[1].1
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/ummodel_hg2-[1].1
cd compile
qsub compile_hadgem2-[1].1
After compilation all the executables are in ACCESSHOME/bin. Names have a date suffix, e.g.
% ls -lt /short/p66/mrd599/ACCESS_BUILD/bin
-rwxr-x--- 1 mrd599 p66 28191506 Oct 17 15:36 um_hg2-[1].1.exe-20131017
-rwxr-x--- 1 mrd599 p66 34463096 Oct 17 15:06 mom4_MPI1.exe.20131017.RAIJIN
-rwxr-x--- 1 mrd599 p66 98971 Oct 17 14:58 mppnccombine.RAIJIN
-rwxr-x--- 1 mrd599 p66 8993556 Oct 17 14:55 cice_MPI1.RAIJIN_6p_20131017
Running the model
All necessary input files for the historical simulation are in directories ~access/data/ACCESS_CMIP5 (model control files), ~access/data/ancil/CMIP5 (CMIP5 emissions) and ~access/data/ancil/access_v2 (land sea mask, vegetation properties etc).
The coupled model control scripts for the original ACCESS 1.0 CMIP5 run are saved at https://access-svn.nci.org.au/trac/cmip5/browser/trunk/exp/hg2-r11Mhd?rev=75. These have been modified to use the files under ~access. The init script has also been modified to allow starting a new run from any year of an existing run (provided the appropriate restart files are in ~access/data/ACCESS_CMIP5/restart/ (this restriction can likely be removed when the /g/data filesystem is available on raijin). At the moment the files for starting the historical run at 1951 are there.
Experiment IDs for ACCESS-CM 1.0 are as follows:
# Experiment ID | # Description |
---|---|
hg2-r11M1p | 1% per year CO2 |
hg2-r11M4c | 4x CO2 |
hg2-r11Mhd | Historical to 2006, then RCP45 |
hg2-r11Mpd | Pre-Industrial Control |
hg2M-RCP85 | RCP85 |
Historical
To set up the model run,
mkdir -p $HOME/ACCESS/exp
cd $HOME/ACCESS/exp
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/exp/a10_hist
cd a10_hist
In script a10_hist, change ACCESS_BINDIR to ACCESSHOME/bin where ACCESSHOME is the directory you used for the build. In a10_hist.init, set the run period (iniyear and finalyear). Also, change the executable specification
cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20131017.RAIJIN $ocn_exe
cp -f $ACCESS_BINDIR/cice_MPI1.RAIJIN_6p_201310128 $ice_exe
cp -f $ACCESS_BINDIR/um_hg2-[1].1.exe-20131017 $atm_exe
to match the build dates of your versions. There are two instances of these copy commands in a10_hist.init.
Then to start the run
qsub a10_hist
The model runs in 3 month chunks, each taking about 70 minutes.
Run log files will appear in this directory. The model run directory is
/short/$PROJECT/$USER/work/um_coupled/a10_hist
and results are archived to
/short/$PROJECT/$USER/archive/a10_hist
Note that to change the experiment name, the directory and the 3 files must have names changed consistently.
The experiment setup will have to be modified further to go past 2005.
Pre-Industrial Control #piControl
The pre-industrial control experiment works much the same as the Historical version
mkdir -p $HOME/ACCESS/exp
cd $HOME/ACCESS/exp
svn co https://access-svn.nci.org.au/svn/cmip5/branches/dev/saw562/a10_piControl
cd a10_piControl
In a10_piControl
set ACCESS_BINDIR, and in a10_piControl.init
set the executable build dates as above.
The model starts from archived CSIRO restart files stored on Raijin at
/g/data1/p66/ACCESSDIR/har599/ACCESS/output/hg2-r11Mpd/restart
. You can set the file to start from by changing the iniyear
variable in the file a10_piControl.init
.
Submit the run with
qsub a10_piControl
By default the run will progress in three-month segments, automatically resubmitting itself after a segment has completed, until it reaches the year given in the finalyear
variable.
RCP 45/85
The RCP scenarios can be enabled by swapping out some of the namelist files from the historical experiment.
The CNTLATM and INITHIS UM namelist files need to be replaced with the corresponding RCP versions. The different versions of these files can be found in the directory /projects/access/data/ACCESS_CMIP5/um_control/hg2-[1].1-M2_Hist
.
The RCP scenarios should use the files (replace RCP45 with RCP85 if required):
INITHIS.RCP45_ocff_hg2r11_dust
CNTLATM.template-RCP45_so2_ocff_hg2r11
These two files set the emssions forcings for the model.
To swap them out, change the lines in the *.init
file:
cp INITHIS.Hist_ocff_hg2r11_dust INITHIS
and
cp CNTLATM.template-Hist_so2_ocff_hg2r11 CNTLATM
to copy the template files appropriate for your experiment