access_NewSun_009 - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
#!html
<h1 style="text-align: center; color: green"> CAWCR-BoM ACCESS NWP Ngamai Migration Working Group</h1>
Meeting 9: Wednesday 4th September 2013, 9E Meeting Room
Present: Robin Bowen, Joerg Henrichs, Ed Habjan, Ilia Bermous, Jim Fraser, Joan Fernon, Wenming Lu, Martin Dix, Chris Tingwell, Michael Naughton, Asri Sulaiman, Yi Xiao
Apologies: Zhihong Li, Robert Jukic, Ivor Blockley, Peter Steinle
- Follows notes from previous meeting.
- DISK usage
- Current disk usage close to maximum
- Can stop suite which has gone to mid Aug, (verification already done with acceptable result).
- Consider deleting pi files and re-run suite to generate as needs arise.
- ACTION: On-going monitoring and management.
- Plots
- Verification essentially OK
- Gary's Diagnostics
- Exploratory task
- ACTION: Gary to report.
- NMOC AG1
- Joan's suite have started cycling.
- Started with input from 25/6, now up to 30/6. Running several days run/day.
- MARS7 not yet ready for archiving
- opdata capacity is 80Tb, sufficient for 2 months worth of output.
- pi files will be deleted once MARS archiving is done
- MARS7 is expected to be available by next week. It will be a disk only system. With 40Tb capacity and only subset of the fields from pi files to be archived, should be sufficient for several months of archiving by all the suites.
- Discussion will be made with Richard Oxbrow regarding new SAM capacity.
- LSDSS disk are already mounted on ngamai.
- Verification of NMOC AG1 suite require MARS -- will need to wait until it is available.
- Now ran to 25 Jun - 8 Aug. Will stop at 11 Aug.
- Joan is close to starting NMOC's AR1
- consolidating directories
- save to svn
- ciwt done 1 month of preliminary verification
- Virtually identical result to solar apart from expected variation in wind biases.
- MSLP SI absolutely identical
- OPS and VAR performance also virtually identical.
- Can declare the suite as OK.
- No need to continue further with CAWCR's ACCESS-R on ngamai.
- Gary's plots will be also be useful on ACCESS-R results. Mike to find someone to take this up.
- The suite have also incorporated speed-ups from Ilia and Joerg.
- Speed of reconfiguration still a concern with large runtime variations
- R12 dump files are much larger than global suite's.
- Reconfiguration may take between 3 - 20 minutes.
- This problem was not observed on solar.
- Timing information on individual task from Joan's runs.
- Individual task all appear a lot faster than solar.
- Overall G_scs_run runtime on ngamai is 59.0 +/- 12.7 min (on solar was 94.9 +/- 11.5 min). See http://webnm-dev.bom.gov.au/nmoc/jrf/G_scs_run.timing.gif
- Frames are being used for LBCs generation in Joan's suite
- No benefit have been observed in using 1/2 hourly rather than hourly LBC's in ACCESS-C suite
- Should update run settings to go to use hourly ourput - significant saving of disk space and run time.
- Joan to coordinate with Chris to implement the change
- Preferably done before start of Joan's ACCESS-R suite.
- Joerg has been investigating run time variations
- Initial runs have variation up to 100%
- Over hundreds of runs, variation is about 10%
- There appear to be some messaging bottle-neck
- Modified messages to reduce size
- Remove combining of several halo exchanges.
- Modified messages to reduce size
- Problem do not occur on raijin - suspect difference in IB network.
- Ilia have tried runs with "sleep" between runs
- No improvement observed.
- Second run still have up to 100% variation.
- First run can have up to 20% variation.
- Ed said Oracle will investigate several areas
- source code
- mpi library
- System configurations
- buffers
- Infiniband drivers.
- There are a great number of runtime settings possible with intel mpi
- Open MPI appear less so, but according to Joerg, there are also a large number of changeable runtime settings.
- It may be worthwhile to communicate with Paul Selwood of UKMO as well as MPI developers.
- Joerg will also look at re-configuration issues next week.
- Now running July to present.
- It is possible to compare to legacy runs, but not that straightforward to do.
- Differences observed in trial verifications of March/April results
- Due to ntile settings?
- Convective settings should be identical, different in re-runs.
- No difference apart from machine for July.
- Holly doing rainval plots
- Chris Bridge & Xiaoxi Wu to run obs verification on surface fields.
- Run Gary's plots
- Basically ok, minor issues still being ironed out.
- To start planning the migration of all the UI's from solar to ngamai, together with SVN repositories and Trac databases.
- This porting meeting will now be conducted every fortnight.
- update and revisit issues list for next meeting
- review status of project documentation
- It is now appropriate to broaden scope to cover the need of migrating everything that currently runs on solar such as research suites.
- Look to reconvene UM Working Group meetings.
- Find out MARS status from Tan Le
- Follow up on "Build Process"
- links to preliminary documentation added to new section in https://trac.nci.org.au/trac/access/wiki/NewSun
[ 8/9/2013 ] azs, first cut. [ 8/9/2013 ] rab, fix some typos. [ 8/9/2013 ] mjn, few changes.