access_CAP_raijin - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
Generating ancillaries on raijin
Version 8.3 of the Met Office Central Ancillary Package is installed on raijin. This is based on a slightly modified version of the Met Office trunk https://access-svn.nci.org.au/trac/cap/browser/branches/dev/Share/ANCIL8.3_nci.
For build details see CentralAncillaryProgram
To run it
module use ~access/modules
module load cap/8.3
The module sets all the environment variables required to run CAP.
This version of the ancillary program is controlled by XML files, e.g. ~access/apps/cap/8.3/data/xml/n96.xml/v11/n96.xml
. As well as the options to generate each field, this file also has job options like time and memory limits. Note that these files are the original Met Office versions and the job options will not be correct for raijin.
Default job limits are 1000 seconds and 2 GB. This is sufficient for generating most mask files, but orography requires higher limits.
Many ancillary tasks require more memory than is available to a single processor job, so an extra option has been added so you can request enough processors to get the required memory but still run on only 1. Only the orography generation can run in parallel (and the parallel version only works correctly for global grids at the moment).
Extra options can be specified as
<ancilTopOptions list='-memory 20 -time 2000 -mpp 16 -queue express'></ancilTopOptions>
Memory is in GB, time is walltime in seconds. mpp specifies the total number of processors for the job. To actually run on less than this add -nprocrun N
.
An example of a file modified for raijin is ~/access/apps/cap/8.3/sample/n96.xml
.
To generate a mask
AncilScr_RunAncil -M -d n96.xml -a mask
This will create and submit a job to generate an N96 mask. Job is AncilScr_Mask in the run directory. Note that at the moment the script just returns with
Return code is 0
and no message about the submitted job (though it does give an error if the submission fails for some reason).
By default, output will appear in $WORKDIR/ancil.vn8.3/n96
. Job output files are also written there.
Some extra options are
- -A directory specify output directory
- -v for verbose output
User Specific Grids
The following example shows how to generate ancillaries on a user specified (non-standard) grid
module use ~access/modules
module load cap/8.3
export DATASETS="mask vegfrac"
export MASKOPTS="-m4 -k1 -3"
export VEGFOPTS="-g5 -h -p1.1 -qmodis4 -3 -A1 -B1 -CHWSD -DCH" # option -5 needs orography ancil
AncilScr_top -search 200 -model msas -grid `pwd`/grid_msas.txt -output `pwd`/cap
The grid_msas.txt file defines the grid
&GRID
POINTS_LAMBDA_TARG=984,POINTS_PHI_TARG=792,
LAMBDA_ORIGIN_TARG=113.000,PHI_ORIGIN_TARG=-10.9586,
DELTA_LAMBDA_TARG=0.0416664,DELTA_PHI_TARG=0.0416663,
PHI_POLE=90.0,LAMBDA_POLE=0.0,ROTATED=F,
GLOBAL=F
/
Typical time and memory requirements
N96
- Mask
- Orography 120 s (16 PE), 20 GB
- Veg/Soil 1200 s (1 PE), 5 GB
N512
- Orography 130 s (16 PE), 22 GB
Validation
Compare N48 ENDGAME fields against ancillaries distributed with vn9.0 example job in ~access/data/UM_Input/vn9.0/ancil/atmos/n48e
.
$UMDIR/vn9.1/linux/utilities/um-cumf qrparm.mask ~access/data/UM_Input/vn9.0/ancil/atmos/n48e/land_sea_mask/igbp/v2/qrparm.mask
Field 1 : Stash Code 30 : LAND MASK (No halo) (LAND=TRUE) : Level 8888
Field 1 has 0 LOGICAL Differences
OK
$UMDIR/vn9.1/linux/utilities/um-cumf qrparm.orog ~access/data/UM_Input/vn9.0/ancil/atmos/n48e/orography/globe30/v2/qrparm.orog
Note that validation against Met Office orography has the use the original bad GLOBE30 data which should not be used for anything else!!
Field 1 : Stash Code 33 : OROGRAPHY (/STRAT LOWER BC) : Level 88
88
MAXIMUM DIFFERENCE= 2.441406250000000E-004 AT PT. 177
RMS FIELD1 : 874.050158655528
RMS FIELD2 : 874.050158094218
Difference: 5.613100029222551E-007 RMS_difference: 6.504486615169443E-006
Field 2 : Stash Code 34 : STANDARD DEVIATION OF OROGRAPHY : Level 8888
MAXIMUM DIFFERENCE= 6.13146972656250 AT PT. 4539
RMS FIELD1 : 233.532721668381
RMS FIELD2 : 233.782032100133
Difference: 0.249310431751667 RMS_difference: 0.413822105003696
Larger than expected difference in the SD here still needs to be explained.