access_aps1_access_c_detail - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

Model in Details

The ACCESS-C operational standard experiment aps1_access_c consists of four parts,

  • System files: located on raijin:$HOME/roses/aps1_access_c, copied by the installation script.
  • Suite files: located on accessdev:$HOME/roses/aps1_access_c, copied by the installation script.
  • Data output: located on raijin:/short/$PROJECT/$USER/aps1_access_c_S and

raijin:/short/$PROJECT/$USER/ncdata/aps1_access_c_S;

those are the forecast data produced by the UM run jobs.

  • Source code: located on raijin:/short/$PROJECT/$USER/aps1_access_c_src, copied by the installation script

In this sectiom, we will explain the components and the structure of the system files in details; we also

describe the tasks/steps in the suite definition in details for the better understanding of the experiment; and

finally we introduce the compilation of the source code for advanced research activity.

ACCESS-C in Details

As explained, aps1_access_c system file, suite and source codes are copied to either raijin or accessdev and

output files are produced when running the suite. The details of these information are illustrated for higher

level of understanding of the whole system. By grabbing those knowledge, users may implement advanced research

by configuring settings, modifying the source codes and some other methods to enhance the forecasting accuracy, running

effieciency etc.

  • System files:raijin:$HOME/roses/aps1_access_c
  • beans:
  • makebc: executable/scripts for making lateral boundary condition
  • mars: executable/scripts for converting/archiving data
  • um: executable/scripts for um forecasting
  • um_components: advanced users may modify the configuration files inside the two following components

for research * xbcjx: um component called by recon and um_run jobs; shared by 6 domains. * xbdmx: um component called by recon_lores; shared by 6 domains.

  • umr: executable/scripts for making initial condition
  • util: utility Python scripts
  • bin:
  • env.aps1_access_c: settings for the system, such as data input and output directory, log directory,

working directory, archiving options, locations of dump and pi files etc;

you may configure settings as you wish

  • produceLinks.ksh: script to create soft links to Cylc scripts
  • Cylc scripts: Cylc job script; you may check what is really done by investigating scripts closely
  • conf (only advanced users may need to modify some of the configuration files if users understand

what they are doing):

  • ANCILmaster:

  • PS25:

  • STASHmaster:

  • ancil: ancillary files, including land-sea-mask, landfrac and orog, for 6 domains; do not need to be modified

  • spectral:

  • log.S: log files from all tasks of the suite when running

  • Cylc Suite: accessdev:$HOME/roses/aps1_access_c

  • suite.rc: suite definition file which defines jobs, job flow, cycle time, resources for jobs etc;

the command rose suite-run picks up this file suite.rc to run the model

  • other files but not important in this case

  • Output data: raijin:/short/$PROJECT/$USER/

  • raijin:/short/$PROJECT/$USER/aps1_access_c_S contains all input and output data running the model;

in this folder each run is given an individual directory as YYYYMMDDHH, such as 2014011800 to

accommodate inputs and outputs. In this directory of YYYYMMDDHH,

  • ic: dump file for producing IC for each domain
  • lbc: pi files for producing LBC for each domain; usually soft links to pi files to save disk usage
  • BN: IC file astart.BN, LBC file lbc.BN and forecast pp files
  • SY: similar to BN
  • VT: similar to BN
  • AD: similar to BN
  • PH: similar to BN
  • DN: similar to BN
  • Note: The directory raijin:/short/$PROJECT/$USER/aps1_access_c_S/YYYYMMDDHH will be moved to

raijin:/short/$PROJECT/$USER/aps1_access_c_S/previous_dates/YYYYMMDDHH.$$ after suite completes.

$$ is the run ID for the specific job does the move

  • raijin:/short/$PROJECT/$USER/ncdata/aps1_access_c_S contains all forecast data in nectCDF format;

go to YYYYMMDDHH/DOMAIN, e.g., 2014011800/BN for Brisbane forecasting data at 2014011800Z run.

  • Source codes: raijin:/short/$PROJECT/$USER/aps1_access_c_src;

see the APS1 ACCESS-C Compilation of Source Code for compilation instruction

  • umbase: Modify source codes in here for testing new science
  • umrecon: Contains configuration file to compile recon executable
  • ummodel: Contains configuration file to compile um executable

ACCESS-C Cylc Suite in Details

ACCESS-C is a relatively simple forecast-only system compared with the Bureau Global and Regional

operational assimilation and forecast systems. The suite definition file suite.rc is located at

accessdev:$HOME/roses/aps1_access_c/. The suite definition file defines all the jobs of aps1_access_c

standard experiment. You may modify the suite definition file for the following purposes,

  • add or remove tasks from the suite.rc
  • update the cycle time by modifying initial cycle time and final cycle time;

in the standard experiment, initial cycle time and final cycle time are both set at

2014011800: it only does one run at 2014011800Z

  • tune the resources, such as number of cores and memory, for each task in the directive item

We use a tree-like structure to display all tasks in suite.rc and their descriptions.

Note that tasks are listed in logically running order, which may not be the case in some Cylc views which may

use alphabetical order.

  • -->setup
  • AC_setup : setup the folders for all jobs; Please go to

DATADIR=/short/$PROJECT$USER/aps1_access_c_S to check the output

short adminsitration job finishing in seconds

  • wait_for: triggered by completion of AC_setup; check the availability of

the input data for creating initial and LB conditions

short job finishing in seconds

  • AC_get_initial_conditions: triggered by wait_for; get the dump file

short job finishing within minutes

  • AC_get_LB_conditions: triggered by wait_for; get the pi files

(in most cases produce soft links to pi files to save disk usage)

short job finishing within a minute

  • --> brisbane
  • make_lbc: producing the lateral boundary condition

long job finishing within 30-60 minutes

  • recon: reconfiguration step to produce the initial condition at ACCESS-C resolution

medium job finishing within few minutes

  • recon_lores: reconfiguration step to produce the initial condition at

the original resolution of the nested NWP system, i.e., ACCESS-R (12km) in this case

medium job finishing within few minutes

  • finalise_recon: copy wind fields from the IC of step recon_lores to that of recon

short job finishing within minutes

  • um_run: UM forecasting job

long job finishing within 30-60 minutes

  • um2grb: convert forecast data from PP format to grib as requested

medium job finishing within few minutes

  • ncdf: convert forecast data from PP format to netCDF as requested

medium job finishing within few minutes

  • --> sydney

  • --> victas

  • --> adelaide

  • --> perth

  • --> darwin

  • -->finalise

  • AC_cleanup: clean up unnecessary files after forecasts finish

short job finishing within seconds

Note:

  • Jobs under each domain are same, i.e. as shown for brisbane domain.
  • Please refer to the Cylc scripts located in raijin:$HOME/roses/aps1_access_c/bin for more information.

In every script, usually the purpose of the script is described at the header section. In addition,

the input data required by the job and the output produced by the job are also recorded at the header

section for reference.

APS1 ACCESS-C Compilation of Source Code

Researchers may need to modify the source codes for testing new Physics, improving efficiency and so on.

The source code is copied by the ~access/AccessModelExperimentLibrary/aps1_access_c/install_aps1_access_c.ksh

to raijin:/short/$PROJECT/$USER/aps1_access_c_src. The modification on source codes must be done in umbase

place and users may follow the instructions to do the compilation,

  1. Update the environment,

module purge

module use ~access/modules

module load intel-cc/12.1.8.273

module load intel-fc/12.1.8.273

module load openmpi/1.6.5

module load netcdf/4.3.0

module load fcm/2.3.1

  1. Go to raijin:/short/$PROJECT/$USER/aps1_access_c_src/ummodel and type

fcm build

to produce UM executable. Look for the executable in raijin:/short/$PROJECT/$USER/aps1_access_c_src/ummodel/bin,

the name of the executable is UM7.6_UK4_EXEC_01.exe.

  1. Go to raijin:/short/$PROJECT/$USER/aps1_access_c_src/umrecon and type

fcm build

to produce UM Recinfiguration executable. Look for the executable in raijin:/short/$PROJECT/$USER/aps1_access_c_src/umrecon/bin,

the name of the executable is qxreconf.