access_MichaelNaughtonEmail14Mar2012.2 - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

#!html
<h1 style="text-align: center; color: blue"> ROSE: response to comments on Met Office ROSE experiments database plans</h1>

> -----Original Message-----
> From: Michael Naughton [mailto:[email protected]] 
> Sent: 14 March 2012 05:26
> To: Matthews, David
> Cc: Asri Sulaiman
> Subject: FW: Plans for ROSE experiments database GUI 
> [SEC=UNCLASSIFIED]
> 
> Dave,
> 
> Thanks for sharing your plans with us.
> 
> Here are some comments Asri and I came up with this morning:
> 
> Overall the ROSE suite database design seems like a 
> compatible step on from the way UMUI experiment database has 
> operated in the past, which will enable fairly smooth 
> transition for users. That seems sensible enough way to go to us.
> 
> You don't mention it, but presumably there will be a GUI 
> interface similar to existing UMUI experiments interface with 
> point-and-click access to suite-xxx commands.

Yes, that's the idea.

> We don't really understand why there is a second 
> "document-oriented database" for the suite discovery/identity 
> info planned to be kept separate from the svn suite 
> repository itself?  Doesn't this just duplicate the same 
> metadata?  Is it a speed thing, to speed up 
> discovery/identity functions of the rose-experiments-database-gui?

Yes, it duplicates the metadata. However, I don't see how you could
implement any kind of discovery system without it. For example,
if I wanted to find all the suites that had been used to make up
a particular climate ensemble. This information will, hopefully,
be stored in the metadata but it would take forever to trawl
through thousands of suites in a repository examining the metadata
for each of them. Duplicating this discovery metadata in a separate
database should be easy to implement and should allow us to build
a very powerful interface.

> Why FCM?  In general, we would like our users and developers 
> to become familiar with using SCM package such as SVN 
> directly, since it can also be used to help with day-to-day 
> tasks.  However, it seems there is tendency at UKMO to shield 
> user from the full intricacy of SVN and present a cut-down, 
> more simplified, FCM interface.

Absolutely. In an ideal world I'd agree we wouldn't need to do
this. However, I still believe that this was, and remains, the
right thing for us to do. The Subversion interface is very
poor at supporting our working practises. Amongst other things,
FCM massively simplifies branching, merging, switching, etc. It
also prevents users from ending up with mixed working copies
(mixed revisions and/or mixed branches) which are horrible in
my opinion and easy to create by accident. If we had not
provided the simplified FCM interface then I think that:
a) FCM would not be as widely used as it is.
b) Our working practises would be much more chaotic.
c) Very few people would use branches.

> What purpose is there really 
> in having intermediate FCM layer:
> 
>    rose-experiments-database-gui >> suite-commands >> 
> fcm-commands >> svn-commands >> rose-svn-experiments-repository

We want to use Subversion because it's what we for the source
code and it makes sense to use the same.

Given that we're using Subversion it makes sense to use FCM
since that's what people are used to. Furthermore, given that
we want to support being to be able to do things like create
branches, etc, we'll need the simplified interface FCM
provides (see above).

Unfortunately, some aspects of working with suites are not the
same as working with source code so we need to support some
additional commands at the Rose level. I can't really see
any way to avoid this.

> We still think a general GUI interface to svn is worth 
> further investigation, i.e. like rabbit-vcs, to enable direct 
> browsing of svn repositories; this would probably be useful 
> for UM and other source code repositories as well as rose experiments.

I've no problem with this so long as it interfaces to the
FCM commands (to avoid the problems inherent in the raw
svn commands). Maybe something for discussion at the working
group at the end of May? This also needs thinking about in
the context of the wider Configuration Management strategy
which we're giving some thought to at present.

> BTW: We finally got PyGTK installed on our machines here, 
> both in the Bureau (Melbourne) and at NCI (Canberra), so 
> we're able to start up rose config editor and gcylc.  This 
> was not so straightforward (at the Bureau) because we do not 
> have system-level access to our machines, so we had to do 
> user-level installs of required packages.

Sorry it been difficult. What systems/OS are you using?

> So, to date all we've done is look-and-see functions with 
> config-edit and cylc, haven't tried to set up and run 
> anything yet, but hopefully will get to start trying this fairly soon.
> 
> One area I've thought to give feedback on is separation of 
> compile & run steps and um-reconfig & um-run steps in the 
> design of rose suites.  I was going to look more closely into 
> the mechanics before commenting, but haven't got to do so.
> 
> In our SMS suites we separate umr and um into separate nqs 
> job steps, even though they get relate to same umui job.  We 
> think of the umr reconfig step as one of the preparatory 
> steps like like ancil or makebc, logically separate from the 
> um step since it relies on a separate executable.  Clearly 
> there's link between reconfig and um steps in the requirement 
> that their settings agree, but ideally it would be good to 
> clarify and minimise, if possible eliminate, these 
> dependencies (e.g. makebc doesn't seem to depend on physics 
> settings?).  I would think a rose reconfig suite could point 
> to a rose model suite to get the variables it requires to know about.

I agree it would be much better it the reconfig and um
applications could be configured independently. However, our
understanding is that this won't be possible to achieve (not
in the short term at least). Therefore, our solution to this
is that they are separate tasks but share the same application
configuration. See the "um.suite" example suite to see how this
works:
http://collab.metoffice.gov.uk/twiki/bin/view/Support/RoseExamples#Example%20Suites%20%28rose-examples/cy

> Likewise, same thinking could be applied to the compilation 
> and running phases for all the packages: um, reconfig, ops, 
> var, etc.  It would seem to be much cleaner to keep the 
> compile suites separate from the run suites, since they're 
> really quite separate steps and don't really derive any 
> benefit from being locked together.  Again, to handle the 
> physics ifdef's (until they can be fully removed), the rose 
> um compile suite would need to point to corresponding rose um 
> run suite to pick up the required settings, but for the rest 
> there's probably no compile-run dependency at all.

I guess this is just a suite design issue and it depends what
you're doing. If it's better to keep compilation in separate
suites then you can design the suites in that way. I suspect
you're right and that NWP trial suites may well work that way.
I'm not so sure about the climate suites. Either way, the
system will support either.

> Some also's:
> 
> We're also interested to hear how you're progressing with 
> rose NWP suite, including when we might be able to pick up 
> test version to play with ourselves.

We have a suite but it needs further review. If you want to
see it you can but it may be better to wait until we've had
a chance to improve it a bit first.

> We're also putting rose developments into plans we're making 
> for our ACCESS modelling developments, including for our 
> ACCESS coupled model, using different ocean and land surface 
> components to HADGEM.
> 
> We kept your file just to ourselves today, since you said 
> it's pre-release.  Colleagues from our TI area will also be 
> interested to read your document, so there'll probably be 
> more comments when the collab-wiki version comes through.

Thanks. Hopefully we'll publish it very soon.

> People are happy with the way you're providing information on 
> the development as you for interested collaborators to follow 
> along, and interact.

Thanks - that's good to know.

I understand that we should have some external newsgroup
functionality in place soon so hopefully we can have some of
these discussions in a more open way which should be good.

Regards,
Dave

> Cheers,
> 
> Mike and Asri