Generating ancillaries on raijin

Version 8.3 of the Met Office Central Ancillary Package is installed on raijin. This is based on a slightly modified version of the Met Office trunk https://access-svn.nci.org.au/trac/cap/browser/branches/dev/Share/ANCIL8.3_nci.

For build details see CentralAncillaryProgram

To run it

module use ~access/modules
module load cap/8.3

The module sets all the environment variables required to run CAP.

This version of the ancillary program is controlled by XML files, e.g. ~access/apps/cap/8.3/data/xml/n96.xml/v11/n96.xml. As well as the options to generate each field, this file also has job options like time and memory limits. Note that these files are the original Met Office versions and the job options will not be correct for raijin.

Default job limits are 1000 seconds and 2 GB. This is sufficient for generating most mask files, but orography requires higher limits.

Many ancillary tasks require more memory than is available to a single processor job, so an extra option has been added so you can request enough processors to get the required memory but still run on only 1. Only the orography generation can run in parallel (and the parallel version only works correctly for global grids at the moment).

Extra options can be specified as

   <ancilTopOptions list='-memory 20 -time 2000 -mpp 16 -queue express'></ancilTopOptions>

Memory is in GB, time is walltime in seconds. mpp specifies the total number of processors for the job. To actually run on less than this add -nprocrun N.

An example of a file modified for raijin is ~/access/apps/cap/8.3/sample/n96.xml.

To generate a mask

AncilScr_RunAncil -M -d n96.xml -a mask

This will create and submit a job to generate an N96 mask. Job is AncilScr_Mask in the run directory. Note that at the moment the script just returns with

Return code is 0

and no message about the submitted job (though it does give an error if the submission fails for some reason).

By default, output will appear in $WORKDIR/ancil.vn8.3/n96. Job output files are also written there.

Some extra options are

• -A directory specify output directory
• -v for verbose output

User Specific Grids

The following example shows how to generate ancillaries on a user specified (non-standard) grid

module use ~access/modules
module load cap/8.3
export DATASETS="mask vegfrac"
export MASKOPTS="-m4 -k1 -3"
export VEGFOPTS="-g5 -h -p1.1 -qmodis4 -3 -A1 -B1 -CHWSD -DCH" # option -5 needs orography ancil
AncilScr_top -search 200 -model msas -grid `pwd`/grid_msas.txt -output  `pwd`/cap

The grid_msas.txt file defines the grid

&GRID
 POINTS_LAMBDA_TARG=984,POINTS_PHI_TARG=792,
 LAMBDA_ORIGIN_TARG=113.000,PHI_ORIGIN_TARG=-10.9586,
 DELTA_LAMBDA_TARG=0.0416664,DELTA_PHI_TARG=0.0416663,
 PHI_POLE=90.0,LAMBDA_POLE=0.0,ROTATED=F,
 GLOBAL=F
/

Typical time and memory requirements

N96

• Mask
• Orography 120 s (16 PE), 20 GB
• Veg/Soil 1200 s (1 PE), 5 GB

N512

• Orography 130 s (16 PE), 22 GB

Validation

Compare N48 ENDGAME fields against ancillaries distributed with vn9.0 example job in ~access/data/UM_Input/vn9.0/ancil/atmos/n48e.

$UMDIR/vn9.1/linux/utilities/um-cumf qrparm.mask ~access/data/UM_Input/vn9.0/ancil/atmos/n48e/land_sea_mask/igbp/v2/qrparm.mask

Field     1 : Stash Code    30 : LAND MASK (No halo) (LAND=TRUE)      : Level 8888
Field     1 has     0 LOGICAL Differences
OK

 $UMDIR/vn9.1/linux/utilities/um-cumf qrparm.orog  ~access/data/UM_Input/vn9.0/ancil/atmos/n48e/orography/globe30/v2/qrparm.orog 

Note that validation against Met Office orography has the use the original bad GLOBE30 data which should not be used for anything else!!

Field     1 : Stash Code    33 : OROGRAPHY (/STRAT LOWER BC)          : Level 88
88
MAXIMUM DIFFERENCE=   2.441406250000000E-004 AT PT.                    177
RMS FIELD1 :    874.050158655528
RMS FIELD2 :    874.050158094218
Difference:   5.613100029222551E-007 RMS_difference:   6.504486615169443E-006

Field     2 : Stash Code    34 : STANDARD DEVIATION OF OROGRAPHY      : Level 8888
MAXIMUM DIFFERENCE=    6.13146972656250      AT PT.                   4539
RMS FIELD1 :    233.532721668381
RMS FIELD2 :    233.782032100133
Difference:   0.249310431751667      RMS_difference:   0.413822105003696

Larger than expected difference in the SD here still needs to be explained.