access_RoseSuitePrebuilds - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
Using prebuilds with rose suites
Using prebuilds
UM prebuilds are a way for multiple UM jobs to share the same files when building. If a job is using a prebuild, the build job will check each file being compiled (& its build settings) against the prebuild and only recompile if the source or options have changed. This can greatly speed up builds if only a few settings have been changed. It also greatly reduces the number of files created in the build output directories.
This is described in the FCM documentation at http://metomi.github.io/fcm/doc/user_guide/make.html#concept.use.
Prebuilds are in /g/data/access/cylc-run/vnX.Y_prebuilds
so that they're accessible for both the extract step on accessdev and the build on raijin/. At the moment we create 6 different configurations
- fcm_make_nci_um_high_noomp
- fcm_make_nci_um_high_omp
- fcm_make_nci_um_high_omp_drhook
- fcm_make_nci_um_high_omp_single_precis
- fcm_make_nci_um_rigorous_noomp
- fcm_make_nci_um_safe_omp
In practice it's likely that NWP configurations will use fcm_make_nci_um_high_omp_single_precis
and climate configurations will use fcm_make_nci_um_safe_omp
. The others are used by rose stem.
To use a prebuild, simply specify the appropriate directory for prebuild
in app/fcm_make/rose-app,conf
. E.g.
prebuild=/g/data/access/cylc-run/vn10.6_prebuilds/fcm_make_nci_um_safe_omp
The fcm-make.cfg
file created by the build will have a line like
use = /g/data/access/cylc-run/vn10.6_prebuilds/fcm_make_nci_um_safe_omp
Many Met Office suites have a PREBUILD environment variable in rose-suite.conf that can be set to the appropriate directory.
Warning about compiler versions
FCM checks for changes in code and compiler options but can't check whether you're using the same compiler version as originally used to create the prebuild. You should be careful to use the same compiler and MPI modules as in the NCI rose stem configuration for that model version. See access/UMVersion_Status.
Creating prebuilds
See also section 11 of UMDP X4, https://code.metoffice.gov.uk/doc/um/vn10.3/papers/umdp_X04.pdf
X4 recommends setting the prebuild to use the source code directly from the repository rather than from a working copy.
Recommended command is something like
rose stem -S PREBUILDS=false --group=fcm_make --source=fcm:[email protected] --name=vn10.5_prebuilds --config=/home/599/mrd599/um_vn10.5/rose-stem
At the moment we don't have an fcm-make group in our rose-stem configuration so run the complete group=nci and just hold all the model reconfig and run tasks. E.g.
rose stem -S PREBUILDS=false --group=nci --source=fcm:[email protected] --name=vn10.5_prebuilds --config=/home/599/mrd599/vn10.5_prebuilds/rose-stem -- --hold
The prebuild directory is set in rose-stem/site/nci/variables.rc
as
{%- set PREBUILD_NCI_ROOT_DIR = '/g/data/access/cylc-run/vn10.5_prebuilds' %}
(the version number is automatically updated by the UM release script).
To create the prebuilds in this directory it's necessary to modify the fcm_make tasks in rose-stem/site/nci/runtime.rc
and also the fcm_make_um app. E.g. in runtime.rc, define a new variable in each task
[fcm_make_nci_um_high_omp](/ACCESS-NRI/accessdev-Trac-archive/wiki/fcm_make_nci_um_high_omp)
[[environment](/ACCESS-NRI/accessdev-Trac-archive/wiki/[environment)]
NCI_PREBUILD_DIR=[PREBUILD_NCI_ROOT_DIR ]/fcm_make_nci_um_high_omp
[fcm_make2_nci_um_high_omp](/ACCESS-NRI/accessdev-Trac-archive/wiki/fcm_make2_nci_um_high_omp)
[[environment](/ACCESS-NRI/accessdev-Trac-archive/wiki/[environment)]
NCI_PREBUILD_DIR=[PREBUILD_NCI_ROOT_DIR ]/fcm_make_nci_um_high_omp
Because we're running with PREBUILDS=false, the PREBUILD variable isn't defined so can't use that.
In app/fcm_make_um/rose_app.conf
add
dest-orig=$NCI_PREBUILD_DIR
dest-cont=$NCI_PREBUILD_DIR
befeore the [env]
section.
The resulting fcm-make.cfg file has
extract.location{diff}[um] = fcm:[email protected]
rather than the specific directory a normal rose-stem build would have
extract.location{diff}[um] = /home/599/mrd599/vn10.5_prebuilds
From vn10.7 on, also set
MAKE_PREBUILDS=true
in rose-stem/rose-suite.conf
.