access_ACCESS1.0_rose - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

Running ACCESS 1.0 Coupled Model under Rose

Introduction

The Australian Community Climate and Earth-System Simulator (ACCESS) models have been developed by the Centre for Australian Weather and Climate Research (CAWCR) and are based on the several models. The coupled version of the model includes several sub models, UM for the atmosphere, MOM for the ocean, CICE for sea ice and either MOSES (ACCESS 1.0) or CABLE (ACCESS 1.3) for the land.

Example Suite

The rose suite au-aa025 is an example suite of the ACCESS coupled Model 1.0. The standard script environment for building and running the model is documented in ACCESS1.0_raijin.

This suite has the following characteristics

  • Historical Run
  • Runs from 1951 to 2010 (or later if desired)
  • Includes build steps for MOM4, UM 7.3 and CICE models
  • Data is processed in 3 month chunks
  • Atmospheric resolution N96, 38L
  • Full GHG, aerosol, volcanic, solar forcings
  • Fixed vegetation

Prerequisites

To run the suite, you will need read permissions for a number of repositories. New users will probably need to request at least some of these. To check your access, run the following commands:

svn ls https://access-svn.nci.org.au/svn/um
svn ls https://access-svn.nci.org.au/svn/cmip5
svn ls https://access-svn.nci.org.au/svn/cice
svn ls https://access-svn.nci.org.au/svn/mom4

If any generate errors, email [email protected]

The code versions of the various models are the same as described in ACCESS1.0_raijin.

Editing the Model

Before editing or running the suite, it needs to be checked out into your local directory on accessdev.nci.org.au. To do this, run the following command.

rosie checkout au-aa025

This will check out the the suite from the repository into your local directory $HOME/roses/au-aa025 with the following directory structure.

total 28
drwxr-xr-x 4 pbd562 p66 4096 Oct 23 10:53 app/
drwxr-xr-x 2 pbd562 p66 4096 Oct 23 10:53 bin/
drwxr-xr-x 2 pbd562 p66 4096 Oct 23 10:53 meta/
-rw-r--r-- 1 pbd562 p66  195 Oct 23 10:53 rose-suite.conf
-rw-r--r-- 1 pbd562 p66   67 Oct 23 10:53 rose-suite.info
-rw-r--r-- 1 pbd562 p66 7338 Oct 23 10:53 suite.rc

If you plan to edit the suite, it's best practice to copy the suite into a new suite. This can be done with the following command.

rosie copy au-aa025

This will create a new suite in the repository with a new id. Default settings for the suite are created and opened in your default text editor. Save any changes and close the file.

To edit the model run the command. (Replace "au-aa025" with the new ID if you ran "rosie copy").

rose edit -C ~/roses/au-aa025

This will open the Rose GUI for editing the suite. Options include environmental variables, namelists and files that should be created at run time.

Editing the Start/End Time and Project

Clicking on jinja2 in the left panel will show top level configuration items that can be edited. These includes the start and end times of the experiment. It is important that you enter a project that you are connected to in the PROJECT field or your job will fail. Users with multiple projects can change to another project by changing this variable in this panel or by editing $HOME/roses/au-aa025/rose-suite.conf on accessdev.

accessedit3.png,700

Editing the Namelists

Namelists for the ocean and sea ice model can be edited from the Rose interface. Clicking on a namelist in the left panel will display the various variables that are currently in that namelist.

accessnamelistedit.png,700

Setting up the Post Processing of Data

Some basic post-processing (e.g. um to netCDF conversion) is set up in the pp1 task, which runs after each access task finishes.

There is also an optional CMIP5 post-processor available.

  • This is turned on using the flag L_CMIP5_PP in the tab: suite conf:jinja2.

  • The environment for the CMIP5 post processor and experiment metadata is set in the tab: post-processing:env:CMIP5_PP_ENV.

  • Configuration of what data is post-processed is handled in the tab: post-processing:namelist:cmip5_tables

This is a list of CMIP5 tables and whether they are output by the post-processor or not. New CMIP5 tables can be included by cloning one of the existing entries. These correspond to the standard CMIP5 tables (see http://cmip-pcmdi.llnl.gov/cmip5/docs/standard_output.pdf), but split up further by the use of suffixes.

Some tables have suffixes: _cable or _moses. These contain variables that require different calculation for model versions using the different land surface schemes. e.g. Amon_moses should be used for ACCESS1.0 since the moses land scheme is used the Amon table contains the rest of the variables which don't require different calculations.

In addition some tables have the suffix _limited or _all. _limited indicates that for CMIP5 the variables included were only requested for a limited range of times for particular experiments (because the amount of data required to output them for all time would be too large). Tables with _all contain variables requested for all times.

Running the Coupled Model

On accessdev

rose suite-run -vv -C ~/roses/au-aa025

accesscylc.png,700

The suite first compiles the executables for each model.

After compilation all the executables are in ACCESSHOME/bin. Names have a date suffix, e.g.

% ls -lt /short/p66/ibc599/cylc-run/au-aa025/share/ACCESS/bin   
-rwxr-xr-x 1 ibc599 p66    98987 Mar  3 12:55 mppnccombine.RAIJIN
-rwxr-xr-x 1 ibc599 p66 34481040 Mar  3 13:45 mom4_MPI1.exe.20140303.RAIJIN
-rwxr-xr-x 1 ibc599 p66  8993556 Mar  3 13:47 cice_MPI1.RAIJIN_6p_20140303
-rwxr-xr-x 1 ibc599 p66 28272234 Mar  3 14:04 um_hg2-[1].1.exe-20140303

After the suite has compiled the various models it will begin running the model. Each run is done in 3 month chunks, each taking about 70 minutes.

To scan accessdev for your running suites and bring up the GUI:

rose suite-scan
rose suite-gcontrol --name=au-aa025

For help:

rose help

Viewing the Output

Data files from the model are

/short/$PROJECT/$USER/cylc-run/au-aa025/share/data/history/

Suite in Detail

  • au-aa025\
  • rose-suite.info: Suite information file which describes what the suite is and who is able to edit/read the files.
  • rose-suite.conf: Suite configuration file which defines start and end time.
  • suite.rc: suite definition file which defines jobs, job flow, cycle time, resources for jobs etc;
  • bin\
  • access: Top Level script that runs the coupled model
  • access.init: Main script that is called by access.
  • access.fin: Finish script that is called by access. Does basic post processing of data.
  • *.py Bin directory contain various python scripts that are used for calculations involving time.
  • app\
  • access
    * rose-app.conf: Rose application file that contains namelist information, environment variabels to be set at run time and files that are generated at the start of each run. * meta\
    • rose-meta.conf: Rose meta data file for the access application. This includes meta data for MOM and CICE namelists.

Log files

There are two types of log files that can be used for solving problems when compiling the UM.

Suite Logs

Suite log files contain top level information about what jobs failed and information about job submission. They are located on Accessdev in

$HOME/cylc-run/SUITENAME/suite/out

$HOME/cylc-run/SUITENAME/suite/log

$HOME/cylc-run/SUITENAME/suite/err

Job Logs

Job log files contain more specific infortion about running a task. They are located on both Accesdev and Raijin in

$HOME/cylc-run/SUITENAME/log/job/access.YYYYMMDD.1.err

$HOME/cylc-run/SUITENAME/log/job/access.YYYYMMDD.1.out

Rose Bush

Log files can be alternatively be viewed via Rose Bush on the Internet (https://accessdev.nci.org.au/rose-bush/).

Validation

To validate your run, it is sufficient to compare the first month's atmosphere output with a standard run:

module load cdo
cdo diffv /short/$PROJECT/$USER/cylc-run/au-aa025/share/data/history/atm/au-aa025.pa-1951-01.nc ~access/data/AccessModelExperimentLibrary/access1.0/std_output/cylc-run/au-aa025/share/data/history/atm/au-aa025.pa-1951-01.nc

If the outputs are the same, you will see the following:

Warning (cdfScanVarAttributes) : NetCDF: Variable not found - time_bnd0
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - time_bnd0
cdo diffn: Processed 112136832 values from 446 variables over 2 timesteps ( 5.04s )

Small differences in the output files can arise due to differences in library versions. A simple check of their significance can be made by inspecting your data in ncview (and this is recommended anyway).

module load ncview
ncview filename.nc

Look in particular at the ranges and patterns.

Help

Frequently Asked Questions about Rose and Cylc

Attachments