access_NewSun_011 - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
#!html
<h1 style="text-align: center; color: green"> CAWCR-BoM ACCESS NWP Ngamai Migration Working Group</h1>
Meeting 11: Wednesday 2nd October 2013, 9E Meeting Room
Present: Joerg Henrichs, Martin Dix, Jim Fraser, Wenming Lu, Chris Tingwell, Michael Naughton, Robin Bowen, Asri Sulaiman, Joan Fernon(phone)
Apologies: Ed Habjan, Ilia Bermous, Zhihong Li, Yi Xiao
- List from previous meeting notes
- Task List
- AOB
-
Ran 3 months to real time, but a problem was discovered (now fixed).
- Verification was not run because of MARS unavailability.
- Problem was noticed by Xiao in ACCESS-TC verification.
- Troubleshooting was aided by Gary's difference plot.
- Tracked down to an abort in SST data update task
- SST's not updated from July values at start of trial period.
- Failure was due to missing directory not created; error was not trapped by SMS.
- Not noticable over short timeframe.
- Impact on result is negligible over Australia.
- Impact shows up clearly in NH.
- Overall the problem was quickly spotted and solved.
-
Restart run commenced from 1st Sept with problem fixed.
- Expect to catch up to real time in ~ 1 week.
-
Gary's Diagnostics - Robin to follow up
- Python program which goes through all output fields and produces statistics, difference plots, etc.
- Can include other programs from various people.
- Plan to document for general use.
- Will re-run from 1st September.
- Ivor working on post-processing tasks.
- Ingestion to MARS to start soon.
- Limited fields/levels and timesteps for limited verification to reduce load on MARS-1.
- MARS-7 still being debugged.
- Data from ACCESS-TC with 3 digit MARS IDs have not been flushing, clogging the cache.
- This was discovered by Arn and Tan -- flushing will improve MARS-1 capacity by 20%.
- Verification using files on disk being tried out by Xiaoxi Wu.
-
Tan Le to analyse users use of MARS fields.
- Identify fields which can be archived to tape more quickly.
- Email CAWCR users to survey usage (rab).
-
Copying of output from Daily run to NCI ( as part of RDSI project ) may further reduce MARS load as some users can access data at NCI.
-
rab, jrf and Joan to follow up
-
Recompilation of MARS-7 being done.
-
Crucial meeting on future of MARS-7 to be held Thursday 3rd October - May decide to revert to old MARS and in interim instead of MARS-7 at SDC.
-
Arn will be on leave in November which will impact MARS development work.
- No new updates from Wenming; all is well.
- NMOC - Joan about to get started on operational ACCESS-C version.
- Joan may be able to start looking at ATC later in October, continuing to Nov.
- For now, ACCESS-C is higher priority.
- Xiao will be away for 2.5 weeks from 18 Oct.
- Will work on new ACCESS-TC & APS2 ACCESS-G when she returns.
- Improvement proposed for reconfiguration step for various ACCESS-TC domains:
- Run re-config over whole domain, then use subset as needed.
- Solution to problem with no land points in ACCESS-TC domains.
- Martin to supply the job.
- Xiao problem with obs task on ngamai has been fixed.
- More monitoring is being done on Ngamai to spot node problems more quickly.
- James Mandilas / Rob Jukic preparing for operational support to allow mid-November NMOC operational switch over.
- NMOC plan to be ready for operstional switchover to Ngamai from 1st November.
-
Kernel tuning changes to disable defragmentation of "Transparent Huge Pages" to be applied to all ngamai computing nodes.
-
The changes should fix Xiao problem with slow reconfiguration execution, from 120-1200s to around 103s.
-
Problem with 2nd UM run substantially slower than 1st run in 2 run test job has been tracked down to a LUSTRE problem which occurs when 2nd job simply overwrites the files created by 1st job. If a fresh directory is used for 2nd job, elapse time variation disappears.
-
A meeting between BoM/ORACLE and NCI sysadmins is being scheduled.
-
Work on documentation page continuing (https://trac.nci.org.au/trac/access/wiki/Access_NWP_Build_Procedures)
-
"Official" UMUI job for building operational UM VN7.5 Global Executable is now available (qaaba)
-
Job for ACCESS-R and ACCESS-C executable to be done.
-
Dan Cook of Oracle is looking at ksh issue on Ngamai.
-
SCI cgi monitor not working due to Perl issue.
-
rab following up -- work in progress.
- Email announcing migration of access applications from solar to ngamai was sent to all solar users; migration scheduled for the period 4:00pm Friday 4th October to 7:00am Monday 7th October.
- Test of all main components was done, but not everything.
-
Rose/Cylc set up on Ngamai
-
rab looking at arranging pre-requisite python package installations.
-
Xiao to install Rose-Cylc packages and try out SREP implementation after her leave.
-
Work to allow compilation on Ngamai compute nodes being addressed.
-
gcc has been installed.
-
"make" required, not yet installed.
-
/apps installation should now be complete.
-
UM Small execs
-
Copy from Raijin now being used.
-
Documentation on their build is requested.
-
Build on ngamai is still desirable but not urgent.
-
CAP program on Ngamai: Wenming and Martin Dix to follow up.
-
Verify is workig on Ngamai, to be done for Raijin.
-
rab sending weekly emails re Solar de-commissioning.
-
Documentation from meetings complementary to this group:
-
Desirable but not critical.
-
Some documents now on S-Drive which may be made available through this NCI trac wiki.
-
GANTT charts are not readable without the software.
-
NMOC web pages (Available within BoM Internal Network only )
-
NMOC pages from here: http://wiki.bom.gov.au/foswiki/NMOC/SoftwareSystems/NgamaiPortIntro
-
"/apps" (modules) : http://nmoc-svnop.bom.gov.au:8080/projects/modules/wiki
-
NMOC NWP Configuration Management
-
"Station list" information now in sync with DA group.
- To be kept in SVN after design/structure is completed.
-
Executable Build procedures being developed (see "Build Procedures" section in this notes).
- Work continuing; NMOC to try out when ready.
-
Fortnightly meetings of this ACCESS NWP Ngamai Migration Working group to be continued until shutdown of Solar.
# * * * 11am Wed 16th October 2013, 9E Meeting Room * * *
[azs, Fri 11/10/2013] First draft. [azs, Mon 14/10/13] Updates with feedback from Robin and Joerg.
[mjn, Mon 14/10/2013] Minor editing.