access_access_g_vn8.2_25km_detail - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
#!html
<h3 style="text-align: center; color: blue"> NECTAR Climate and Weather Science Laboratory</h3>
<h3 style="text-align: center; color: blue"> Work Package 1</h2>
<h3 style="text-align: center; color: blue"> ACCESS Simulation and Modelling</h3>
<h3 style="text-align: center; color: darkgreen"> ACCESS Model Experiment Library</h3>
<h4 style="text-align: center; color: black">ACCESS Global model standard experiment access_g-vn8.2_25km</h4>
<h4 style="text-align: center; color: black">Details</h4>
The model suite is implemented as a 'Rose/cylc' suite within the ACCESS infrastructure framework at NCI.
Within the Rose/cylc environment, a suite is a set of files and configurations in a particular directory structure stored as a single entity in a suite data base (implemented using SVN ). Users interface with suites through a Rose utility called rosie. To allow systematic management of potentially many thousands of suites among hundreds of users in a global research environment, rosie assign a 8 character code: 2 letters and 3 digits, prepended by a country code ("au-" for Australia) for each suite.
- rose-suite.info - Contains top level information about the suite
- rose-suite.conf - Contains the Jinja2 template variables that read into suite.rc
- suite.rc - Cylc file that contains the scheduler information and task dependencies
- app/
- forecast/
- rose-app.conf - Namelist information
- f_short/
- rose-app.conf - Namelist information
- bin/
- envfile
- UMScr_RunUM8.2
suite.rc file is the main main file as far as describing the model tasks and how they are to be run to cylc.
The file app//rose-app.conf is the main file containing the settings for particular model run. Contrasting with previous method of running the Unified Model, all the namelist settings pertinent to a model run is kept in this single user-editable file. This file is also editable within a GUI environment using another rose utility rose config-edit which operates with, or without meta-data.
NOTE: Some values in rose-app.conf are given in the form of environment variables which are defined to cylc via suite.rc or at runtime.
Using rose config-edit with meta-data provide users an environment similar previous UMUI utility, whilst experienced or expert users may choose to turn-off meta-data, or simply use a text editor.
-
Domain Decomposition = 14x36 (set in suite.rc)
- UM_ATM_NPROCX=14
- UM_ATM_NPROCY=36
-
TIMESTEP = 10 minutes [entry in rose-app.conf: steps_per_period=144] (fc_short uses 5 minute timestep)
- The suite is written so that if a model forecast run fails, a rerun is made with shorter timestep.
-
RUN LENGTH = 12 Hours [entry in rose-app.conf: run_target_end=30,12,20 ]
- Equate to 72 timesteps.
- To demonstrate the cycling behaviour of the au-aa087 cylc suite, the suite is set to run for 5th March and 6th March, 00z 2014 ( 2014030500 & 2014030600 ).
- Input data directory 2014030418 and 2014030518 holds input data to the two model runs.
-
When completed, the suite takes up about 186GB of disk space in user's /short/${PROJECT}>/<${USER}/cylc-run/au-aa-87/
- 8GB each for two input data directories
- 85GB each for output of the two model run.
-
Each model runs uses 504 CPU cores (14x36) and elapsed time of about 1400 seconds (totaling about 200cpu hours. note that there can be significant variation in time taken between identical runs repeated at different times).
-
Information on run output are further expanded in 'output files' section further down.
UM Model runs normally start from an initial conditions file (sometimes referred to as model dump). In the case of au-aa087, it also make use of an "analysis-increment" file ( non-mandatory option).
In cases where model run is preceded by "reconfiguration" run to prepare the initial condition, it will also require ancillary data.
For au-aa087, all the large input data files are kept in ~access area as further elaborated below:
Location of input data files are set in the bin/envfile.
export INPUTDATA=/projects/access/AccessModelExperimentLibrary/access_g_vn8.2_25km/data/00hd_3
export INITIAL_DUMP_DIR=$INPUTDATA/dump
export INITIAL_DUMP_FILE=iau-start-dump.${ref}
export AN_DIR=$INPUTDATA/an
export AN_FILE=analysis-increments_${REFERENCE_TIME}
export SMC_DIR=$INPUTDATA/smc
export SMC_FILE=an_smc.${REFERENCE_TIME}
Ancillary data is found in the directory defined by ANCILDIR variable in bin/envfile
export ANCILDIR=~access/AccessModelExperimentLibrary/access_g_vn8.2_25km/data/um/ancil
\
Information regarding the location of ACCESS-G2 start-files archived on Bureau of Meteorology filesystems is available here: G2_data_at_BoM.
The suite create a directory /short/$PROJECT/$USER/cylc-run/$CYLC_SUITE_NAME in which the model forecast run is done and where most output files are generated.
In total, the suite as it is currently configured generated about 186 GB of output.
EXAMPLE:
raijin> cd /short/$PROJECT/$USER/cylc-run/au-aa087
raijinb> du
8145708 ./share/2014030418/run/staging
8145712 ./share/2014030418/run
8145716 ./share/2014030418
8141604 ./share/2014030518/run/staging
8141608 ./share/2014030518/run
8141612 ./share/2014030518
8 ./share/2014030500/tmp/a_pb_2014030500
932516 ./share/2014030500/tmp/a_pb
996228 ./share/2014030500/tmp/a_pc
188 ./share/2014030500/tmp/xbjin.32766
8 ./share/2014030500/tmp/a_pa_2014030500
473316 ./share/2014030500/tmp/a_pa
16 ./share/2014030500/tmp/a_pc_2014030500
2402284 ./share/2014030500/tmp
101272 ./share/2014030500/run/um-output/dataw/pe_output
1916056 ./share/2014030500/run/um-output/dataw
160 ./share/2014030500/run/um-output/datam/history_archive
264 ./share/2014030500/run/um-output/datam
1916324 ./share/2014030500/run/um-output
39327956 ./share/2014030500/run/staging/ls-fields-dir
530904 ./share/2014030500/run/staging/svar-in
53246944 ./share/2014030500/run/staging
55163272 ./share/2014030500/run
4 ./share/2014030500/archive/cx
4 ./share/2014030500/archive/analysis-fields/pb
4 ./share/2014030500/archive/analysis-fields/pc
1320804 ./share/2014030500/archive/analysis-fields
10088900 ./share/2014030500/archive/cycle_dumps0
4 ./share/2014030500/archive/varobs
4 ./share/2014030500/archive/model-dumps
4 ./share/2014030500/archive/bgerr
9937180 ./share/2014030500/archive/fieldsfiles/pi
4 ./share/2014030500/archive/fieldsfiles/pa
1482580 ./share/2014030500/archive/fieldsfiles/pe
219572 ./share/2014030500/archive/fieldsfiles/pf
4 ./share/2014030500/archive/fieldsfiles/pb
4 ./share/2014030500/archive/fieldsfiles/pc
4243980 ./share/2014030500/archive/fieldsfiles/pd
15883328 ./share/2014030500/archive/fieldsfiles
16 ./share/2014030500/archive/cycle_dumps
27293068 ./share/2014030500/archive
84858628 ./share/2014030500
4 ./share/data/2014030500
4 ./share/data/2014030600
12 ./share/data
932828 ./share/2014030600/tmp/a_pb
996228 ./share/2014030600/tmp/a_pc
16 ./share/2014030600/tmp/a_pc_2014030600
188 ./share/2014030600/tmp/xbjin.11100
473588 ./share/2014030600/tmp/a_pa
8 ./share/2014030600/tmp/a_pb_2014030600
8 ./share/2014030600/tmp/a_pa_2014030600
2402868 ./share/2014030600/tmp
101276 ./share/2014030600/run/um-output/dataw/pe_output
1918280 ./share/2014030600/run/um-output/dataw
160 ./share/2014030600/run/um-output/datam/history_archive
264 ./share/2014030600/run/um-output/datam
1918548 ./share/2014030600/run/um-output
39327952 ./share/2014030600/run/staging/ls-fields-dir
531828 ./share/2014030600/run/staging/svar-in
53250080 ./share/2014030600/run/staging
55168632 ./share/2014030600/run
4 ./share/2014030600/archive/cx
4 ./share/2014030600/archive/analysis-fields/pb
4 ./share/2014030600/archive/analysis-fields/pc
1320800 ./share/2014030600/archive/analysis-fields
10088900 ./share/2014030600/archive/cycle_dumps0
4 ./share/2014030600/archive/varobs
4 ./share/2014030600/archive/model-dumps
4 ./share/2014030600/archive/bgerr
9956592 ./share/2014030600/archive/fieldsfiles/pi
4 ./share/2014030600/archive/fieldsfiles/pa
1484868 ./share/2014030600/archive/fieldsfiles/pe
220196 ./share/2014030600/archive/fieldsfiles/pf
4 ./share/2014030600/archive/fieldsfiles/pb
4 ./share/2014030600/archive/fieldsfiles/pc
4250180 ./share/2014030600/archive/fieldsfiles/pd
15911852 ./share/2014030600/archive/fieldsfiles
16 ./share/2014030600/archive/cycle_dumps
27321588 ./share/2014030600/archive
84893092 ./share/2014030600
186039064 ./share
4 ./work/staging.2014030500/axs599/glb/00hf/smc
8 ./work/staging.2014030500/axs599/glb/00hf
12 ./work/staging.2014030500/axs599/glb
16 ./work/staging.2014030500/axs599
20 ./work/staging.2014030500
176 ./work/forecast.2014030600
4 ./work/PostArchive.2014030500
172 ./work/forecast.2014030500
4 ./work/PostArchive.2014030600
4 ./work/staging.2014030600/axs599/glb/00hf/smc
8 ./work/staging.2014030600/axs599/glb/00hf
12 ./work/staging.2014030600/axs599/glb
16 ./work/staging.2014030600/axs599
20 ./work/staging.2014030600
400 ./work
186039468
Standard output from um model execution go to /short/$PROJECT/$USER/cylc-run/au-aa087/share//run/um-output/dataw/pe_output/xbjin.fort6.peXXX where XXX is between 0-503 (because the jobs is broken up into 504 MPI threads).
A good start to verify that the model executes properly is to look at xbjin.fort6.pe0 file. A fort.6.pe0 file from a successful run (from /short/$PROJECT/$USER/cylc-run/au-aa087/share/2014030600/run/um-output/dataw/pe_output/fort.6.pe0) have been copied to ~access/AccessModelExperimentLibrary/access_g_vn8.2_25km/output/au-aa087_xbjin.fort6.pe0 on raijin.
Your run should return a fort6.pe0 file with numbers identical to that file. You can difference the whole file, or for quick comparison, the 'Absolute Norm' values printed out by UM's GCR's solver is a good indicator of bit-reproducibility. "grep 'Absolute Norm' ~access/AccessModelExperimentLibrary/access_g_vn8.2_25km/output/au-aa087_xbjin.fort6.pe0" and compare it against 'Absolute Norm' values from your run's xbjin.fort6.pe0
Output from cylc tasks go to $HOME/cylc-run/au-aa087/log/job/
Output there are mainly related to cylc tasks and scheduling information.
The forecast task in au-aa087 is configured to write out a number of output data files in the form /short/$PROJECT/$USER/cylc-run/au-aa087/share/$yyyymmdd00/run/um-output/datam/xbjina_pa00h,_pb00h and _pc00h at hourly or 3hourly interval depending on type of fields. Those files are in UM format which subsequent tasks "a_pa", "a_pb" and "a_pc" convert into meteorological grib format which are then moved to /short/$PROJECT/$USER/cylc-run/au-aa087/share/$yyyymmdd00/archive/
The xbjina_pX00H files are removed when the a_pX tasks completed.
A copy of /short/$PROJECT/$USER/cylc-run/au-aa087/share/2014030500/archive/ from a successfully completed run is kept on raijin:~access/AMEL/access_g_vn8.2_25km/output/au-aa087_share_2014030500_archive/ for reference.
Various tools exist for analysing and plotting output data. A quick and simple tool called xconv is available on raijin (~access/bin/xconv) for viewing the output fields. For example:
raijin> ARCHD=~access/AMEL/access_g_vn8.2_25km/output/au-aa087_share_2014030500_archive
raijin> ~access/bin/xconv -i $ARCHD/fieldsfiles/pi/xbjina_pi000_2014030500_utc_glb.um
- See AMEL N512 Global NWP Suite for description of a full NWP suite
- For more information on UM, register to use UKMO Collaboration Wiki (http://collab.metoffice.gov.uk/view ) and go to http://collab.metoffice.gov.uk/twiki/bin/view/Support/UnifiedModelDocumentation
- For more information about rose, go to http://metomi.github.io/rose/doc/rose.html
- For more information about cylc suite engine, go to http://cylc.github.io/cylc/
- Access Wiki is the portal for various information pertinent to ACCESS development work.
<-- Back to Research UM vn8.2 25km Global Model Start Page
<-- Back to ACCESS Model Experiment Library
[WikiStart <--- Back to Main ACCESS page ]