This page will help you to get to the point of running some basic test cases using MOM6 and SIS2. If you have any problems please check the issues pages and see the FAQ. There are a few broad steps:

1. Downloading model source code and tools
2. Downloading input dataF
3. Compiling the models
4. Running a test case
5. Checking your results

Most of the source code is hosted on GitHub. Check that git is setup on your machine and that you can "talk" to GitHub. See Set Up Git and Git configuration and environment for details.

All files need to be visible to the compute nodes on your machine. For example on the gaea platform the files are best placed on the lustre file system. Be aware that many parts of the lustre file system are swept (i.e., your files are erased). To avoid having files swept, it is best to use the unswept area, e.g.:

cd /lustre/f1/unswept/$USER/

Note that the lustre file system is NOT backed up. The home file system is backed up but of limited size, and since it is not visible from the compute nodes, using home is more difficult.

In general if you are making changes to the source code it's important to use git to push to GitHub regularly. This is especially true when your source and edits are on a non-backed up file system (like lustre).

If you are running the model on a desktop/laptop then it shouldn't matter where you put the source.

The source code is split across several repositories. These are arranged in a hierarchy with MOM6-examples at the top as a "super repository", and the other repositories configured as submodules. To get a better understanding of the repository layout see Understanding the repository layout.

Even without a GitHub account you can clone (download) the repositories by executing this single command at the command-line in a shell:

git clone --recursive https://github.com/NOAA-GFDL/MOM6-examples.git MOM6-examples

That said, we strongly recommend having a GitHub account so you can version control your own modifications and contribute to MOM6.

The above will clone (download) the MOM6-examples repository and all its sub-modules into a directory called MOM6-examples/.

From here on, all commands will take place within the MOM6-examples/ directory you just created.

By way of explanation, the above one line "recursive clone" is equivalent to the follow steps:

git clone https://github.com/NOAA-GFDL/MOM6-examples.git MOM6-examples
cd MOM6-examples
git submodule init
git submodule update --recursive

where the last line in turn is equivalent to

git submodule update src/FMS
git submodule update --init --recursive src/MOM6
git submodule update src/SIS2
git submodule update tools/matlab/gtools
git submodule update tools/python/MIDAS

Knowing this, you can avoid downloading various of the submodules. For instance, you could just clone MOM6-examples and none of the submodules by stopping before the git submodule init.

Note that MOM6 also has a sub-module (CVmix). If you haven't used a recursive clone or update and src/MOM6/pkg/CVmix-src is empty, it should be initialized and updated independently.

cd src/MOM6
git submodule init
git submodule update

Not all of the necessary component models for a fully coupled model can be downloaded as described above. If you would like to run any coupled model configurations (i.e. anything other than stand-alone MOM6), then see Obtaining other components.

MOM6 relies on tools that are part of the Flexible Runtime Environment (FRE) to help with the build process. Those tools are provided in the NOAA-GFDL/mkmf repository which you should find in MOM6-examples/src/mkmf/.

The workflow we describe in these wiki pages generally compiles the model in a user created directory MOM6-examples/build/:

cd MOM6-examples
mkdir build

After downloading all of the above the directory tree should look like:

% cd MOM6-examples
% tree -d -L 2
.
|-- build
|-- coupled_AM2_LM2_SIS
|   |-- AM2_MOM6i_1deg
|   `-- CM2G63L
|-- coupled_AM2_LM3_SIS
|   `-- AM2_MOM6i_1deg
|-- coupled_AM2_LM3_SIS2
|   |-- AM2_SIS2B_MOM6i_1deg
|   `-- AM2_SIS2_MOM6i_1deg
|-- ice_ocean_SIS
|   |-- Amery
|   |-- *
|   `-- OM4_025
|-- ice_ocean_SIS2
|   |-- Baltic
|   |-- *
|   `-- SIS2_icebergs
|-- ocean_only
|   |-- DOME
|   |-- *
|   `-- torus_advection_test
|-- src
|   |-- FMS
|   |-- MOM6
|   |-- SIS2
|   |-- atmos_null
|   |-- coupler
|   |-- ice_ocean_extras
|   |-- icebergs
|   |-- land_null
|   `-- mkmf
`-- tools
    |-- analysis
    |-- matlab
    |-- python
    `-- tests

Some experiments have input data sets that are placed in an INPUT/ subdirectory of the example. For example tree ocean_only/global yields:

% tree ocean_only/global
ocean_only/global
|-- INPUT
|   |-- 31Layer_zgrid.nc -> .datasets/global/siena_201204/INPUT/31Layer_zgrid.nc
|   |-- GOLD_IC.2010.11.15.nc -> .datasets/global/siena_201204/INPUT/GOLD_IC.2010.11.15.nc
|   |-- OM3_zgrid.nc -> .datasets/global/siena_201204/INPUT/OM3_zgrid.nc
|   |-- README -> .datasets/global/siena_201204/mosaic.unpacked/README
|   |-- atmos_hgrid.nc -> .datasets/global/siena_201204/mosaic.unpacked/atmos_hgrid.nc
|   |-- atmos_mosaic.nc -> .datasets/global/siena_201204/mosaic.unpacked/atmos_mosaic.nc
|   |-- atmos_mosaic_tile1Xland_mosaic_tile1.nc -> .datasets/global/siena_201204/mosaic.unpacked/atmos_mosaic_tile1Xland_mosaic_tile1.nc
|   |-- atmos_mosaic_tile1Xocean_mosaic_tile1.nc -> .datasets/global/siena_201204/mosaic.unpacked/atmos_mosaic_tile1Xocean_mosaic_tile1.nc
|   |-- geothermal_heating_cm2g.nc -> .datasets/global/siena_201204/INPUT/geothermal_heating_cm2g.nc
|   |-- grid_spec.nc -> .datasets/global/siena_201204/mosaic.unpacked/grid_spec.nc
|   |-- gustiness_qscat.nc -> .datasets/global/siena_201204/INPUT/gustiness_qscat.nc
|   |-- land_hgrid.nc -> .datasets/global/siena_201204/mosaic.unpacked/land_hgrid.nc
|   |-- land_mask.nc -> .datasets/global/siena_201204/mosaic.unpacked/land_mask.nc
|   |-- land_mosaic.nc -> .datasets/global/siena_201204/mosaic.unpacked/land_mosaic.nc
|   |-- land_mosaic_tile1Xocean_mosaic_tile1.nc -> .datasets/global/siena_201204/mosaic.unpacked/land_mosaic_tile1Xocean_mosaic_tile1.nc
|   |-- mosaic.nc -> .datasets/global/siena_201204/mosaic.unpacked/mosaic.nc
|   |-- ocean_forcing_daily.nc -> .datasets/global/siena_201204/INPUT/ocean_forcing_daily.nc
|   |-- ocean_forcing_monthly.nc -> .datasets/global/siena_201204/INPUT/ocean_forcing_monthly.nc
|   |-- ocean_hgrid.nc -> .datasets/global/siena_201204/mosaic.unpacked/ocean_hgrid.nc
|   |-- ocean_mask.nc -> .datasets/global/siena_201204/mosaic.unpacked/ocean_mask.nc
|   |-- ocean_mosaic.nc -> .datasets/global/siena_201204/mosaic.unpacked/ocean_mosaic.nc
|   |-- ocean_precip_monthly.nc -> .datasets/global/siena_201204/INPUT/ocean_precip_monthly.nc
|   |-- ocean_vgrid.nc -> .datasets/global/siena_201204/mosaic.unpacked/ocean_vgrid.nc
|   |-- seawifs_1998-2006_GOLD_smoothed_2X.nc -> .datasets/global/siena_201204/INPUT/seawifs_1998-2006_GOLD_smoothed_2X.nc
|   |-- sgs_h2.nc -> .datasets/global/siena_201204/INPUT/sgs_h2.nc
|   |-- tideamp.nc -> .datasets/global/siena_201204/INPUT/tideamp.nc
|   `-- topog.nc -> .datasets/global/siena_201204/mosaic.unpacked/topog.nc
|-- MOM_input
|-- MOM_memory.h
|-- MOM_override
|-- MOM_parameter_doc.all
|-- MOM_parameter_doc.short
|-- available_diags.000000
|-- diag_table
|-- input.nml
|-- ocean.stats.gnu
|-- ocean.stats.intel
`-- ocean.stats.pgi

The files are actually links relative to a non-existent link .datasets. This link can be pointed to either a local installation of the data sets or a shared installation (.e.g /archive/gold/datasets or /lustre/f1/pdata/gfdl_O/datasets).

Follow the instructions below to create a .datasets link.

cd MOM6-examples/

ln -sf /lustre/f1/pdata/gfdl_O/datasets .datasets

You only have to do this once.

cd MOM6-examples/
ln -sf /archive/gold/datasets .datasets

You only have to do this once.

You can construct your own datasets directory that the link .datasets points to by downloading and unpacking each of the tar files at ftp://ftp.gfdl.noaa.gov/pub/aja/datasets/ into your own central location and linking .datasets to that location.

Note that not all tar files are needed at once but some tests might use data from more than one file. We try to reuse data between tests and the directory names within the datasets location are not necessarily those of the tests in MOM6-examples.

We need to build different executables depending on the coupled mode of the model. All modes require MOM6 and FMS source code. When used in ice-ocean mode, whether with SIS or SIS2, we must link in a "null" land- and atmosphere-models. In fully coupled mode you also need the full land and atmosphere source codes.

The phases of compilation are:

1. Create a path_names file that contains the relative path to the source files. This uses the list_paths tools which ships with FRE but is also provided in the mkmf package that you installed in the above Downloading build tools and templates section.
2. Create a Makefile that will compile your library or executable. This uses the mkmf tool which ships with FRE.
3. Compile your library or executable.

We typically apply above steps either component-by-component (the usual method within FRE scripts) or, in these pages, applied to FMS and then everything else.

The first step is to ensure that you have all the necessary prerequisite software installed. As a minimum you'll need: a fortran compiler (such as gfortran), an MPI implementation (such as openmpi or mpich2), and netcdf libraries and headers. Here are some instructions for specific platforms:

• Gaea: set up a compiler environment.
• Ubuntu: set up the GNU toolchain.
• Fedora: set up the GNU toolchain.

Feel free to add instructions for your platform if they differ considerably from the above.

The following instructions are for GFDL's HPC "Gaea" using the Intel compiler within the Cray's Intel programming environment. They will need adapting to work on other platforms.

It is best to compile the shared code separately from the model code (proves quicker when building multiple versions of the model). From the MOM6-examples/ directory:

mkdir -p build/intel/shared/repro/
(cd build/intel/shared/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths ../../../../src/FMS; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/ncrc-intel.mk -p libfms.a -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names)

The above creates the file build/intel/shared/repro/Makefile which you can use to build the fms library:

(cd build/intel/shared/repro/; source ../../env; make NETCDF=3 REPRO=1 libfms.a -j)

which creates the file build/intel/shared/repro/libfms.a. This is a library that we will link to after compiling the MOM6 source code. Note that to create a debug compilation, change "repro" to "debug". To use the GNU compiler instead of Intel then change "intel.mk" to "gnu.mk".

If these step fails then it's very likely that your build template - "intel.mk" in this case - does not match your environment, please see Setting up build configuration

With the FMS library compiled, you can now build the MOM6 code:

mkdir -p build/intel/ocean_only/repro/
(cd build/intel/ocean_only/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths ./ ../../../../src/MOM6/{config_src/dynamic,config_src/solo_driver,src/{*,*/*}}/ ; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/ncrc-intel.mk -o '-I../../shared/repro' -p MOM6 -l '-L../../shared/repro -lfms' -c '-Duse_libMPI -Duse_netCDF -DSPMD' path_names)

One can also use dynamic_symmetric instead of dynamic.This step creates the "pathnames" file, which lists the source code that will be compiled. One generally does NOT need to recreate the Makefile or path_names files. But if any new module has been added, or there have been changes to the dependencies between modules (e.g., changed use statements), THEN one needs to recreate the path_names file.

Finally, compile the MOM6 ocean-only model with:

(cd build/intel/ocean_only/repro/; source ../../env; make NETCDF=3 REPRO=1 MOM6 -j)

If successful you should now have an ocean executable: build/intel/ocean_only/repro/MOM6.

Passing -I ../../shared/repro via the -o option to mkmf is making sure that the modules files associated with libfms.a can be found. Similarly passing '-L../../shared/repro -lfms' via the -l option to mkmf is specifying where to look for libfms.a.

Again, to compile in debug mode, change "repro" to "debug" in every place above.

To run the circle_obcs case, you need to replace config_src/dynamic with config_src/dynamic_symmetric above.

This step is an alternative to the above "Compiling MOM6 without SIS2".

mkdir -p build/intel/ice_ocean_SIS2/repro/
(cd build/intel/ice_ocean_SIS2/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths ./ ../../../../src/MOM6/config_src/{dynamic,coupled_driver} ../../../../src/MOM6/src/{*,*/*}/ ../../../../src/{atmos_null,coupler,land_null,ice_ocean_extras,icebergs,SIS2,FMS/coupler,FMS/include}/)
(cd build/intel/ice_ocean_SIS2/repro/; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/ncrc-intel.mk -o '-I../../shared/repro' -p MOM6 -l '-L../../shared/repro -lfms' -c '-Duse_libMPI -Duse_netCDF -DSPMD -Duse_AM3_physics -D_USE_LEGACY_LAND_' path_names )

Finally, compile the MOM6 sea-ice ocean coupled model with:

(cd build/intel/ice_ocean_SIS2/repro/; source ../../env; make NETCDF=3 REPRO=1 MOM6 -j)

A successful compilation will yield the file build/intel/ice_ocean_SIS2/repro/MOM6.

How a test is run depends a lot on your platform. On most systems you'll need to use the scheduling system to get access to compute nodes. For example, see Running on Gaea below. If this is not the case, or you already have an interactive session on a compute node, then simply execute the following:

cd ocean_only/double_gyre/
mkdir -p RESTART
mpirun -n 8 ../../build/intel/ocean_only/repro/MOM6

The mkdir -p RESTART is needed so that the model can write out a restart state at the end of the test. Also you may need to replace "mpirun" with the name of your MPI run executable. The smaller test cases, like double gyre, can be run on a single core with:

cd ocean_only/double_gyre/
mkdir -p RESTART
../../build/intel/ocean_only/repro/MOM6

On gaea you will be logged into a front-end node on which you compiled. The model executable can only execute in parallel on the compute nodes. The executable is loaded onto the compute nodes with the aprun command which is invoked from a batch node. Job scripts (batch scripts) run on the batch nodes and are normally submitted from the front-end nodes. In brief:

• Front-end nodes - can see /home and /lustre/f1. Use for editing, compiling and submitting jobs.
• Batch nodes - can see /home and /lustre/f1. Use for launching the executable with aprun.
• Compute nodes - can only see /lustre/f1.
• Use aprun instead of mpirun.

To run the model interactively you can obtain an interactive session on a batch node with (for example):

msub -I -X -m n -l walltime=02:00:00 -l nodes=2:ppn=32

to get a session with 64 (2x32) processors for 2 hours.

msub -I -X -m n -l walltime=02:00:00 -l nodes=15:ppn=32

to get a session with 480 (15x32) processors for 2 hours. Note: may need to drop the -X option if there is a problem with the display settings.

Although this shell can see /home the executable will not be able to. Therefore the directory in which you run aprun should be on /lustre/f1, which is why we suggest working on the /lustre file-system when setting up MOM6.

Once you have run the test case you will get some output (and/or a netcdf HDF5 error that will indicate that you did not create the RESTART/ folder above) in the directory from which you ran``mpirun(i.e. inMOM6-examples/ocean_only/double_gyre/` for the above example).

The numerical results will depend on the specifics of your platform (chip, operating system, compiler vendor, versions, etc.) and so "bit for bit" reproducing the results obtained on GFDL platforms is not guaranteed. Some basics are expected for the checked-in experiments:

• The generated output documenting parameters and configuration (MOM_parameter_doc.*) should be indentical (except for choices of processor layout).
• The absence of any U_velocity_truncations or V_velocity_truncations files, which if present indicate numerical instabilities.

If you want an example of regression files used by GFDL for testing, you can look in a demonstration regression repository. If you clone this regressions repository, or better yet, maintain one of your own for you platform, then you could check your results by comparing the newly generated ocean.stats file to the committed one:

diff ocean.stats ocean.stats.intel

If you are running on gaea and the ocean.stats files differ, then try removing the Depth_list.nc file and ensure that you ran the initial test with 8 processors (i.e -np 8). If you are not running on gaea then it's expected that there will be small differences in the ocean.stats files. This is because no two compilers or computers can be relied upon to do all floating point calculations in a consistent way.

To look at netCDF output, you might need to combine the netcdf files, if you ran on more than one processor. To combine the netCDF files on gaea, do the following:

module load fre
mppnccombine <fname>.nc

where "<fname>" is the root for the files to be combined. If you do not have access to the mppnccombine command then you can find the source code for the tool at https://github.com/NOAA-GFDL/FRE-NCtools.

The content of the netcdf files is controlled by the model input file diag_table. See Diagnostics from the MOM6 documentation. You will need a netcdf browser (such as ncview) or the appropriate netcdf libraries within your analysis software (e.g. scipy.io.netcdf or netCDF4 for python).

See Running and changing the "benchmark" test for another example of running an ocean-only model and how to change the configuration.