Minimising disk usage with rose/cylc suites

This page provides some tips for minimising the disk use of your rose/cylc suites on both accessdev and raijin.

Model builds

Prebuilds

Using a prebuild saves both time and disk space because your job only builds those files that are different to the standard version. See access/RoseSuitePrebuilds.

fcm make archive option

fcm make has an option --archive,http://metomi.github.io/fcm/doc/user_guide/command_ref.html#fcm-make

In archive mode, intermediate files will be put into TAR-GZIP archives on completion, e.g. extract system: .fcm-make/cache/extract/, and build system: build/include/ and build/o/.

If the make is rerun the archives are unpacked before rebuilding anything so the time penalty is negligible.

To use this add the line

args=--archive

to your suite fcm_name/rose-app.conf file at the top before the [env] section. For a typical full build (no prebuild used), disk space is reduced from 150 MB accessdev and 355 MB raijin to 60 MB accessdev and 256 MB raijin. Builds that include a lot of branches will benefit even more.

Log files and work directories

Rose has a builtin app rose_prune which can delete or archive old suite log files and remove old work directories, http://metomi.github.io/rose/doc/rose-rug-task-run.html#rose-task-run.util.rose_prune

E.g. the standard GA7 AMIP suite u-ah530 has a housekeeping app with rose-conf

meta=rose_prune
mode=rose_prune

[prune]
archive-logs-at=-P1Y
prune-datac-at=-P3M
prune-remote-logs-at=-P1M
prune-work-at=-P3M

This model runs in one month chunks so at the end of a month the work directory and log files for the previous month are removed on raijin. On accessdev, the job/log/ directories from the run a year earlier are tarred and gzipped, e.g. directory job/log/19910101T0000Z to job-19910101T0000Z.tar.gz. In this case the disk usage is reduced by a factor of about 20. Note that the files are still viewable with rose-bush.