access_ACCESS_AMIP_testcases - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
ACCESS 1.0 and 1.3 AMIP testcases
The ACCESS 1.0 and 1.3 coupled and atmospheric (AMIP) configurations were used in the CAWCR submission to CMIP5. See https://wiki.csiro.au/confluence/display/ACCESS/ for more information. The model configuration and forcing is described in several papers published in a special issue of the Australian Meteorological and Oceanographic Journal in 2013, http://www.bom.gov.au/amoj/papers.php?year=2013.
The ACCESS1.0 atmospheric configuration is very similar to HadGEM2 as used in the Met Office CMIP5 simulations. ACCESS1.3 is based on a HadGEM3 configuration and also includes the CABLE land surface scheme. For full details on the model configurations see Bi et al (http://www.bom.gov.au/amoj/docs/2013/bi1_hres.pdf) and Kowalczyk et al (http://www.bom.gov.au/amoj/docs/2013/kowalczyk_hres.pdf). For details on the model forcing see Dix et al (http://www.bom.gov.au/amoj/docs/2013/dix_hres.pdf).
Note that the ACCESS land-sea mask and orography is different to both HadGEM2 and HadGEM3. See LandMask and HighResolutionAustralianOrography for details.
The jobs described here were originally developed and run on vayu but have been modified to run on raijin. Compiler differences mean that the original results will not be reproduced exactly with a new executable, but the original vayu executables can still be run on raijin. Note however, that it is intended for the user to build their own executable, prior to running the model.
The UMUI jobs described here are available on accescollab. Whilst accesscollab is still in operation, accessdev is preferred for ACCESS development and as the UMUI host machine in the longer term. We have provided equivalent UMUI jobs for accessdev. These jobs are intended to be run in place as the run jobs are hard-wired to point to the executable compiled by the build job. If the user makes a copy of the UMUI job to submit then this requires editing.
For access 1.0, the build and run jobs are vabva and vabvm respectively. These are identical to the versions on accesscollab (saaca and saacm described below). These jobs are owned by Martin Dix and can be located by searching/filtering the owner ID mrd599 or the experiment ID vabv.
For access 1.3, the build and run jobs are vagzb and vagza respectively. These are identical to the versions on accesscollab (saada and saadb described below). These jobs are owned by Jhan Srbinovsky and can be located by searching/filtering the owner ID jxs599 or the experiment ID vagz.
For the access 1.3 single column model (SCM), the build job is vagzs. This is identical to the version on accesscollab (saads). This job can be located by searching/filtering the owner ID jxs599 or the experiment ID vagz. The SCM model is run via an executable script. Further details about the SCM are here
ACCESS 1.0
Build job saaca uses the UM vn 7.3 trunk + branch https://access-svn.nci.org.au/trac/um/browser/branches/dev/han32j/VN7.3-HadGEM2-1.1. Note that the job builds a coupled model executable that can also be run in AMIP mode. The hand edit
accesscollab:~access/umui_jobs/hand_edits/coupled/access_coupled.ed
sets the appropriate preprocessor directives and the compiler overrides
~access/umui_jobs/overrides/coupled/raijin_coupled_file.ovr
~access/umui_jobs/overrides/coupled/raijin_coupled.ovr
set the appropriate compiler options.
The build uses the standard system modules
module load intel-cc/12.1.9.293
module load intel-fc/12.1.9.293
module load openmpi/1.6.5
module load netcdf/4.2.1.1
and the access modules (from ~access/modules)
module load fcm/2.3.1
module load gcom/4.4
module load dummygrib
module load oasis3
The executable will be /short/$PROJECT/$USER/UM_ROUTDIR/$USER/saaca/ummodel/bin/hg2r11_access.exe.
Run job is saacm. Note that these jobs are owned by mrd599. To run this job you will need to change the path to the executables to use the name of your build job (presently set to saaca).
ACCESS 1.3
Build job saadb uses the 7.3 trunk + branch https://access-svn.nci.org.au/trac/um/browser/branches/pkg/Rel/ACCESS1.3. As with the 1.0 build, this job also builds a coupled model executable.
The run job is saada.
Job saads builds a single column version of the ACCESS 1.3 model. See SCM_testcases
ACCESS 1.1
This configuration is HadGEM2 + CABLE. It was not submitted to CMIP5, but used for isolating the effect of CABLE from the other changes in the 1.3 configuration.
Original CMIP5 executables used by CAWCR
The original CMIP5 executables used on vayu were built with intel-fc/11.1.046 and OpenMPI/1.4.3. Although that compiler version is not available on raijin, the compiler libraries were statically linked so the executables still run and produce results identical to the original vayu simulations (note that in general this reproducibility depends on the compiler options used - see wiki:ACCESS1.0_reproducibility).
The original ACCESS 1.0 executable was /short/p73/har599/UM_ROUTDIR/har599/xajbf/ummodel/bin/xajbf_new.exe_07Nov11. Copied to directory ~/access/data/AccessModelExperimentLibrary/access1.0/bin.
ACCESS1.0 run job was xajba (in the UMUI DB on cherax). This was copied to accesscollab as sabma and modified to run on raijin as sabme. Note that this has been converted to UMUI vn7.3.1 to support the extra precision in the GHG values.
The original ACCESS 1.3 executable was /short/p66/hxy599/ACCESS/bin/um_hg3.exe-CABLE-amip, copied to ~access/data/AccessModelExperimentLibrary/access1.3/bin.
ACCESS1.3 run job was uaakq (cherax), copied to sabmc and modified as sabmf.
Note that these jobs are owned by access and should appear in your list when the UMUI is started.
Accessdev versions are vacue and vacuf, owned by mrd599.
The standard ACCESS1.0 experiment started at 1979-01-01 from a restart file from an earlier AMIP experiment. The ACCESS1.3 simulation started at 1978-01-01 from the same initial file as used for the coupled model simulations. This had a year spin-up before the CMIP5 data was saved.
Comparing to original experiments
Post-processed data from the AMIP runs is available via the Earth System Grid. Raw model output is available on raijin
- 1.0 AMIP run /g/data/p66/ACCESSDIR/AMIP/hg2_MOSES/xajba/atm (with dump files in /g/data/p66/ACCESSDIR/har599/ACCESS/output/xajba)
- 1.3 AMIP run /g/data/p66/ACCESSDIR/AMIP/hg3_CABLE/uaakq/atm
For convenience the monthly mean surface air temperature (tas), sea level pressure (psl) and precipitation (pr) post-processed CMIP5 output files are in ~access/data/AccessModelExperimentLibrary/access1.0/std_output (and similarly for 1.3).
- The original experiments were done on VAYU and will not produce identical results on RAIJIN.
Validating results
Use the cumf command as below to compare your result /short/$PROJECT/$USER/$RUNID/$RUNIDa.pah9jan to the existing one ~access/data/AccessModelExperimentLibrary/access1.0/std_output/vabvm/vabvma.pah9jan (1.0 AMIP run): ~access/data/AccessModelExperimentLibrary/access1.3/std_output/vagza/vagzaa.pah8jan (1.3 AMIP run):
export UMDIR=~access/umdir
~access/umdir/vn8.5/normal/utils/cumf ~access/data/AccessModelExperimentLibrary/access1.0/std_output/vabvm/vabvma.pah9jan /short/$PROJECT/$USER/$RUNID/$RUNIDa.pah9jan
The following outputs will show after cumf command finishes:
Summary in: /short/p66/hxy599/tmp/cumf_summ.hxy599.d14302.t161217.7979
Full output in: /short/p66/hxy599/tmp/cumf_full.hxy599.d14302.t161217.7979
Difference maps (if available) in: /short/p66/hxy599/tmp/cumf_diff.hxy599.d14302.t161217.7979
Check the summary file, you will see the similar content as below. The 0 difference on DATA FIELDS indicates the results are identical.
COMPARE - SUMMARY MODE
-----------------------
Number of fields in file 1 = 2024
Number of fields in file 2 = 2024
Number of fields compared = 2024
FIXED LENGTH HEADER: Number of differences = 0
INTEGER HEADER: Number of differences = 0
REAL HEADER: Number of differences = 0
LEVEL DEPENDENT CONSTANTS: Number of differences = 0
LOOKUP: Number of differences = 0
DATA FIELDS: Number of fields with differences = 0
files compare, ignoring Fixed Length Header
Note on STASH output
Met Office configurations typically use a 360 day calendar. Using the regular calendar makes the calculation of climate means within the model run considerably more expensive. To work around this we have a locally developed modification to calculate the monthly means. This was used in all the CMIP5 runs (AMIP and coupled) and the code branch is included in both jobs here. See GregorianClimateMeans for details
Run times
The runs job are all set to run in 3 month chunks. After the first 3 month chunk is complete, enable the hand-edit ~access/umui_jobs/hand_edits/common/crun.ed, process and resubmit. The job should then continue to resubmit automatically until complete.
A 3 month run requires about 3500 s walltime on 64 processors on raijin.