Using prebuilds with rose suites

Using prebuilds

UM prebuilds are a way for multiple UM jobs to share the same files when building. If a job is using a prebuild, the build job will check each file being compiled (& its build settings) against the prebuild and only recompile if the source or options have changed. This can greatly speed up builds if only a few settings have been changed. It also greatly reduces the number of files created in the build output directories.

This is described in the FCM documentation at http://metomi.github.io/fcm/doc/user_guide/make.html#concept.use.

Prebuilds are in /g/data/access/cylc-run/vnX.Y_prebuilds so that they're accessible for both the extract step on accessdev and the build on raijin/. At the moment we create 6 different configurations

• fcm_make_nci_um_high_noomp
• fcm_make_nci_um_high_omp
• fcm_make_nci_um_high_omp_drhook
• fcm_make_nci_um_high_omp_single_precis
• fcm_make_nci_um_rigorous_noomp
• fcm_make_nci_um_safe_omp

In practice it's likely that NWP configurations will use fcm_make_nci_um_high_omp_single_precis and climate configurations will use fcm_make_nci_um_safe_omp. The others are used by rose stem.

To use a prebuild, simply specify the appropriate directory for prebuild in app/fcm_make/rose-app,conf. E.g.

prebuild=/g/data/access/cylc-run/vn10.6_prebuilds/fcm_make_nci_um_safe_omp

The fcm-make.cfg file created by the build will have a line like

use = /g/data/access/cylc-run/vn10.6_prebuilds/fcm_make_nci_um_safe_omp

Many Met Office suites have a PREBUILD environment variable in rose-suite.conf that can be set to the appropriate directory.

Warning about compiler versions

FCM checks for changes in code and compiler options but can't check whether you're using the same compiler version as originally used to create the prebuild. You should be careful to use the same compiler and MPI modules as in the NCI rose stem configuration for that model version. See access/UMVersion_Status.

Creating prebuilds

See also section 11 of UMDP X4, https://code.metoffice.gov.uk/doc/um/vn10.3/papers/umdp_X04.pdf

X4 recommends setting the prebuild to use the source code directly from the repository rather than from a working copy.

Recommended command is something like

 rose stem -S PREBUILDS=false --group=fcm_make --source=fcm:[email protected] --name=vn10.5_prebuilds --config=/home/599/mrd599/um_vn10.5/rose-stem

At the moment we don't have an fcm-make group in our rose-stem configuration so run the complete group=nci and just hold all the model reconfig and run tasks. E.g.

rose stem -S PREBUILDS=false --group=nci --source=fcm:[email protected] --name=vn10.5_prebuilds --config=/home/599/mrd599/vn10.5_prebuilds/rose-stem -- --hold

The prebuild directory is set in rose-stem/site/nci/variables.rc as

{%- set PREBUILD_NCI_ROOT_DIR   = '/g/data/access/cylc-run/vn10.5_prebuilds' %}

(the version number is automatically updated by the UM release script).

To create the prebuilds in this directory it's necessary to modify the fcm_make tasks in rose-stem/site/nci/runtime.rc and also the fcm_make_um app. E.g. in runtime.rc, define a new variable in each task

    [fcm_make_nci_um_high_omp](/ACCESS-NRI/accessdev-Trac-archive/wiki/fcm_make_nci_um_high_omp)
        [[environment](/ACCESS-NRI/accessdev-Trac-archive/wiki/[environment)]
	    NCI_PREBUILD_DIR=[PREBUILD_NCI_ROOT_DIR ]/fcm_make_nci_um_high_omp 

    [fcm_make2_nci_um_high_omp](/ACCESS-NRI/accessdev-Trac-archive/wiki/fcm_make2_nci_um_high_omp)
        [[environment](/ACCESS-NRI/accessdev-Trac-archive/wiki/[environment)]
            NCI_PREBUILD_DIR=[PREBUILD_NCI_ROOT_DIR ]/fcm_make_nci_um_high_omp 

Because we're running with PREBUILDS=false, the PREBUILD variable isn't defined so can't use that.

In app/fcm_make_um/rose_app.conf add

dest-orig=$NCI_PREBUILD_DIR
dest-cont=$NCI_PREBUILD_DIR

befeore the [env] section.

The resulting fcm-make.cfg file has

extract.location{diff}[um] = fcm:[email protected]

rather than the specific directory a normal rose-stem build would have

extract.location{diff}[um] = /home/599/mrd599/vn10.5_prebuilds

From vn10.7 on, also set

MAKE_PREBUILDS=true

in rose-stem/rose-suite.conf.