#!html
<h1  style="text-align: center; color: green"> CAWCR-BoM ACCESS NWP Ngamai Migration Working Group</h1>

Meeting 12: Wednesday 16th October 2013, 9E Meeting Room

Present: Martin Dix, Jim Fraser, Ed Habjan, Chris Tingwell, Michael Naughton, Robin Bowen, Asri Sulaiman, Joan Fernon, Yi Xiao, Ilia Bermous

Apologies: Joerg Henrichs, Wenming Lu, Zhihong Li,,

• Update on MARS status
• List from previous meeting notes
• Task List
• AOB

• Robin reported: It was decided that MARS-7 will not be ready and the project is put on the back-burner.
• MARS-1 will be used for ngamai operations.
  • "Forensic" work on MARS-1 have resulted in substantial improvement
• Joan have streamlined the flow of critical data by cutting down fields and levels of fields from AG1 and AR1 for archiving to MARS.
  • Ivor will be testing the new scripts.
• Improvements to MARS-1 may in fact make it possible to try out full ingestions of data with fall-back to reduced set if needed.
• Three more tape drives are expected, which should improve MARS-1 even further.
• New MARS support staff seconded from ITOPS have done good work in improving MARS-1 operations.
• ACTION: Continue monitoring AG1 MARS ingestion.
• Refer to Jim Fraser's updates sent via email.

• NMOC updates on ACCESS-G porting is available on NMOC BoM wiki: http://wiki.bom.gov.au/foswiki/NMOC/OperationsDevelopment/NgamaiAccessGPort (BoM Internal)
• ACCESS-G takes about 57 minutes to run on ngamai, which is about 40 minutes improvement over solar.NMOC run timings: http://webnm-dev.bom.gov.au/nmoc/jrf/G_scs_run.timing.gif
• Verification skill score looks ok, with skill scores of ngamai ACCESS-G rerun slightly better that previous, probably due to addition of NOAAA-19 data. NMOC Verification plots: http://intweb.bom.gov.au/nob/nmoc/verify/verify_charts/daily_plots/verf_plot_trial_ngamai_access_r.html
• Plots from Kevin Keay is available, though not yet uploaded; these will be discussed further by NWP-WG
• Ivor has started on AG downstream task on ngamai.
• ATC-1 nesting within AG has not been restarted.
• Gary's Diagnostics are still work-in-progress

• Improvement from AR1 runtime on ngamai have been "sensational".
• This time critical application now takes (on-average) 69 minutes to run - a 50 minutes improvement.
• MPI 1.6.5, tuning of "Transparent Huge Pages" have contributed to the improved and more consistent run times.
• Joerg's SSP jumped from 400 to 500 points
• Mis-config NMOC var fix applied.
• Verify Broadscore over Australia looks good.
• Moving towards Surface Verifications when time permits -- this is a system issue, not directly related to porting.

• Joan has started running NMOC's ACCESS-C version.
• Some script fixes required after incorporating Milton's tidy-ups
• More stricter script criteria implemented.
• Will start cycling once MARS issues are settled.
• No timing information yet on ngamai ACCESS-C
• Expected to start cycling with data from 10/10/2013.----
• Wenming's AC1 accidentally reverted back to using old STASHMASTER file - this has been fixed.

• APS0 systems are being turned off
  • AC0 already turned off
  • AG0 and AR0 to follow

• Work on NMOC's ATC1 starting.
• Xiao to provide handover notes.
• Martin have run reconfiguration for big domain on ngamai
• Extraction of subset is ok.
• Work done using UM N7.6.
• Not much structural changes required.
• Joan and Robin to work on replacing existing reconfig in ATC.
• Some complication is expected.
• Do not implement until Xiao gets back from leave.
• Robin to provide other miscellaneous program updates, including tcbgs program.

• Virtual server within a VMWare installation at SDC will be available shortly
• Will be used for SMS scheduler controlling ngamai operational systems.
• This will replace current SMS system on RTDS4.
• Expected to have improved stability.
• Fail-over capability.
• New system will have stdout and stderr available in real-time
• Work to "harmonise" NMOC's sms implementation is very desirable and scheduled for 1Q 2014.

• Disk space quotas are now implemented on ngamai
• rto quota on ngamai is set to 5Tb
• Use "lfs quota" command to check usage status
  • eg. lfs quota -u $DATADIR
• Ngamai quota policies and link to current usage available from: http://ngamai.bom.gov.au/~jjm/quotas.html
• Keeping ngamai HOME area to manageable size is crucial
  • Currently backup of solar's HOME can take up to 36 hours.
  • 4/5 hours backup time is more reasonable.
  • Currently ngamai HOME usage is already at 1.5Tb ( about 1/4 of solar's)
• mjn's and Xiao's jobs was hit by ngamai's quota implementation - now fixed.
  • Due to oracle implementation mistake the quota size was set a magnitude too low.
• Standard ngamai users will be set with Tier-1 quotas. A few people with specific needs are given Tier-2 quotas.

• Ilia has successfully verified the improved and consistent runtimes with the implementation of LUSTRE bug workaround and kernel tuning with "Transparent Huge Pages" (From email sent after meeting).
• Information on kernel tuning have been supplied to NCI and they have also turned off "Transparent Huge Pages" on raijin.

• Migration have been conducted succesfully with a few issues emerging.
• Most of those issues have been addressed.
• Email to be sent to access_nwp_users asking users to test all their usage and identify as many issues as possible before solar becomes unavailable.
• Remind users to use $DATADIR for UM Builds.
• All sample UMUI UM build jobs will use $DATADIR for intermediate files.
• Some users will need to migrate to Solar due to continued use of assimilation infrastructure.

• Implementation of dependent libraries has been added to /apps tasks
• Robin to follow up with the aim to have it available by the time Xiao gets back from leave
• SREP demo planned for JAN 2014

• S/W stack update was made and tested successfully.

• Raijin's version in use
• Documentation and ngamai build -- work in progress

• Martin and Wenming following up -- work in progress

• Ngamai version now working
• Work-in-progress for Raijin's version.

• Robin implementing weekly emails
• Impact on research ?
• Noel and Maree (visiting scientist) will use solar to the end for ATC
• Migrating is not feasible.
• AGREPS suites
• APS-2
• Other D/A suites?
• Switching off of some nodes during November
  • reduce power and air-conditioning requirement.
  • usage is supposed to be reducing.
• NMOC looking at mid-November for the date of switchover of operational systems.
  • May be done earlier
• Ngamai to be declared fully operational by end of week beginning 21/10/13.

• Work on documentation page continuing (https://trac.nci.org.au/trac/access/wiki/Access_NWP_Build_Procedures)

• Need peer verifications of the steps outlined in the documentation.

• Test with build of AG1 generated executable of different size (but gives identical result? )

• Simple test run jobs required for each build

• Currently need to test within NWP suite.

• Ilia building VAR and OPS on raijin

• Need to perform source extraction on ngamai (repository not available from raijin)

• May build on ngamai, and transfer executable.

• Solution to the un-availability of OPS and VAR source outside of BoM needs attention.

• Ilia's test of running UM7.5 R12 Ngamai executable on Raijin gave bit-identical results

  • 186 Time step, 8x16 decomposition, 45 minutes runtime

• Previous test of vn8.4 executable built on raijin did not run on ngamai.

• Missing dynamic library

• Porting of VAR 27.2 and 29.2

  • executable crashed requiring disabling of vectorization
    • Problem with intel compiler reported to compiler developers.

• Job for building UM VN8.2 on ngamai available

• To report on performance

• Test of Intel compiler 14.0.0 reported problems with optimisation at "-O2" or "-O3" and only work with "-O0".

• ksh problem was narrowed down to a ksh bug
  • Fix is available - to be applied
  • In the meantime workaround is being used.

 # * * * 11am Wed 30th October 2013, 10E Meeting Room * * *

#!html
<h2  style="text-align: left; color: red">  NOTE: NEW MEETING VENUE </h1>

-> Back to Ngamai Porting index page

--> Back to Main access page