access_APS1_ACCESS_C - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

APS1 ACCESS-C Experiments

Wenming LU [email protected], Michael Naughton [email protected]

ACCESS-C Background

ACCESS City model, ACCESS-C for short, is one of the ACCESS NWP systems operational in the Australia Bureau of Meteorology.

There are in ACCESS-C six pre-defined domains namely, BN, SY, VT, AD, PH, DN, which cover main cities, Brisbane, Sydney,

Melbourne-Hobart, Adelaide, Perth and Darwin. The general information of these domains is listed in the following table,

City	Domain	Lat. Range	Lon. Range	Grids	Res.	fcst_hrs
`Brisbane`	`BN`	-31.00S, -22.04S	148.00E, 156.03E	250x224	4km	36
`Sydney`	`SY`	-38.00S, -29.98S	147.00E, 155.03E	224x224	4km	36
`VicTas`	`VT`	-46.00S, -33.04S	139.00E, 150.99E	362x334	4km	36
`Adelaide`	`AD`	-39.50S, -29.53S	132.00E, 141.98E	278x306	4km	36
`Perth`	`PH`	-37.00S, -28.04S	112.00E, 122.98E	250x306	4km	36
`Darwin`	`DN`	-16.00S, -7.97S	128.50E, 139.98E	224x320	4km	36

APS1 ACCESS-C UMUI jobs

APS1 ACCESS-C is UM VN7.6 UMUI jobs. There are three sets of UMUI jobs for each domain,

A: standard UMUI jobs
B: number of convection calls per physical timestep is increased to 4 instead of 1 in the standard UMUI jobs
C: half timestep of that in the standard UMUI jobs

City	Domain	A	B	C
`Brisbane`	`BN`	`xbcjb`	`xbeab`	`xbeeb`
`Sydney`	`SY`	`xbcjs`	`xbeas`	`xbees`
`VicTas`	`VT`	`xbcjv`	`xbeav`	`xbeev`
`Adelaide`	`AD`	`xbcja`	`xbeaa`	`xbeea`
`Perth`	`PH`	`xbcjp`	`xbeap`	`xbeep`
`Darwin`	`DN`	`xbcjd`	`xbead`	`xbeed`

These UMUI components are located on vayu:/g/data/access/AccessModelExperimentLibrary/aps1_access_c/beans/um_components.

For reference, the resources needed by the forecast step for each domain are listed below,

City	Domain	Num. of Cores	Wall Time (min)	CPU time (min)	Input Data Size(B)	Output Data Size(GB)
`Brisbane`	`BN`	100	28 (22*)	2756	294M/3.9G	16.0
`Sydney`	`SY`	100	25 (20*)	2420	272M/3.5G	15.0
`VicTas`	`VT`	100	52 (38*)	4760	620M/5.5G	33.0
`Adelaide`	`AD`	100	38 (28*)	3375	438M/4.9G	23.0
`Perth`	`PH`	100	37 (27*)	3605	396M/4.7G	16.0
`Darwin`	`DN`	100	28 (25*)	2480	309M/4.1G	18.0

Note*: Wall time with 196 cores used.
In the column of Input Data Size(B), the sizes of initial condition and lateral boundary condition files (IC/LBC) are listed.

Sample Input Data and Forecast Output Data

There are some sample data for the experimental run (20130204 00Z) of the APS1 ACCESS-C on vayu.

Input data:
Dump file for producing initial condition: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/input_data/iau0-dump.201302040000
Pi files for producing LBC: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/input_data/qwxbeta_pi[009-040]_2013020400_utc_fc.um
Please note that those data will not be used when running the standard domains;

the sample initial condition (IC) and lateral boundary condition (LBC) have been provided (see the Output Forecast Data);

for other domains, um_reconfig and makebc jobs are needed to create the IC and LBC files
Output Forecast Data
BN: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/BN, including IC, LBC and forecast data
SY: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/SY, including IC, LBC
VT: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/VT, including IC, LBC
AD: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/AD, including IC, LBC
PH: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/PH, including IC, LBC
DN: /g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/DN, including IC, LBC

UM Run Script

We provide a script access_c_um.sh for running these cases, located in the scripts directory

vayu:/g/data/access/AccessModelExperimentLibrary/aps1_access_c/scripts

To try a test run, please copy run script to your own place and change the following items if you need to

2nd line, $PROJECT to your valid project, such as dp9, du7 etc
line 29, export MY_PFX=BN #change BN to AD SY VT DN PH as you wish
line 30, export UMUI_ID=xbcjb #leave as is, or change xbcjb to another UMUI job id from table above of UMUI jobs

Next, submit the run script:

qsub access_c_um.sh

The forecast output will be stored at /short/$PROJECT/$USER/aps1_access_c/$DOMAIN given $DOMAIN is

one of BN AD SY VT DN PH you specify. For BN, the sample forecast data are store at

/g/data/access/AccessModelExperimentLibrary/aps1_access_c/data/output_data/BN for your comparison.

Temorary Note: UM Run Script on raijin

The APS1 ACCESS-C is under working process to be ported to raijin. The folder vayu:/g/data/access/AccessModelExperimentLibrary/aps1_access_c/

is rsynced to raijin:/short/dp9/wml548/aps1_access_c/. The executable for the UM forecasting and reconfiguration has been compiled

on raijin and tested fine.

raijin:/short/dp9/wml548/aps1_access_c/scripts/access_c_um.sh has also been updated for raijin. This script has been tested on BN domain

and the forecast data are located in raijin:/short/dp9/wml548/aps1_access_c/BN. Please refer to the previous section for the modifications

to make your own experiments.

Full ACCESS-C Run by CYLC

For researchers who need to be able to control the full set of steps of the ACCESS-C system, for example to run other dates or

to cycle through periods of running, the CYLC version of the APS1 ACCESS-C is provided. The steps to copy and run the CYLC

version are as follows.

First, run

/g/data/access/AccessModelExperimentLibrary/aps1_access_c/scripts/install_aps1_access_c_cylc.ksh

If the script runs successfully, it will create the following two directories in your home directory,

aps1_access_c: contains binaries, scripts, configurations, environment files and log files
cylc/aps1_access_c: CYLC suite definition and CYLC related scripts and configurations

To run CYLC requires loading necessary modules

module use /g/data/access/modules

module load cylc

*Note: Please delete any already loaded modules which conflict with this CYLC module.

cylc db register aps1_access_c $HOME/cylc/aps1_access_c

where aps1_access_c is the name of the suite in CYLC, $HOME/cylc/aps1_access_c is the PATH to the suite definition file.

The next step is to run the suite:

gcylc aps1_access_c

You should be able to see the following window

Please click the run button and type the required dates as in the following box (for this example run, input data is supplied for the 20130204 00Z case)

Finally the GUI control of the ACCESS-C system pops up as follows

The following section describes the steps in the full ACCESS-C CYLC sample suite.

Steps in the ACCESS-C CYLC Suite

ACCESS-C is a relatively simple forecast-only system compared with the Bureau Global and Regional operational assimilation and

forecast systems. We use a tree-like structure to display all tasks and descriptions. Note that tasks are listed in logically

running order, which is not the case in the (current) CYLC displays which use alphabetical order at each level of the tree.

-->setup
AC_setup : setup the folders for all jobs; Please go to DATADIR=/short/$PROJECT$USER/aps1_access_c_S to check the output
wait_for: triggered by completion of AC_setup; check the availability of the input data for creating initial and LB conditions
AC_get_initial_conditions: triggered by wait_for; get the dump file
AC_get_LB_conditions: triggered by wait_for; get the those pi file
--> brisbane
make_lbc: producing the lateral boundary condition
recon: reconfiguration step to produce the initial condition at ACCESS-C resolution
recon_lores: reconfiguration step to produce the initial condition at the original resolution (usually lower than ACCESS-C resolution)
finalise_recon: copy wind fields from the IC from step recon_lores to that from recon
um_run: UM forecasting job; equivalent for what access_c_um.sh does
um2grb: convert forecast data from PP format to grib as requested
ncdf: convert forecast data from PP format to netCDF as requested
opendap: archive forecast data to OPeNDAP folder for easy access
--> sydney
--> victas
--> adelaide
--> perth
--> darwin
-->finalise
AC_cleanup: clean up unnecessary files after forecasts finish

*Note: Jobs under each domain are same, i.e. as shown for brisbane domain.

Sample Plots

BN 20130204 00Z time+2 Forecast: `Mslp_Precip(1 Hour)`; time+1 Forecast: `Sfc Temp`, `Sfc Dewpoint`, `Sfc Wind`

Note: For all plots please click the image for display in the full size.

Sample Plots For Extreme Weather Cases

VT 20130321 00Z time+1 to time+6 Forecast: `Mslp_Precip(1 Hour)`

Oswald BN 20130125 00Z time+2 Forecast: `Mslp_Precip(1 Hour)`; time+1 Forecast: `Sfc Real. Humidity`, `850hPa Wind`, `Geopotential Height Wind`