Fedora compiler environment - adcroft/MOM6-examples GitHub Wiki
Supplementary instructions for Fedora Linux environments
Noted differences between Fedora Core (FC) 21 and Fedora Core 23. Useful commands to find the software we need to compile MOM6.
- Package management: FC21: yum, FC23: dnf
- List files in a package: FC21: rpm -ql [pkg], FC23: dnf repoquery -l [pkg]
- FC21: pkg-config configuration files are not available for openmpi, see below.
These instructions closely follow the instructions outlined for GFDL's HPC platforms.
Setup a compiler environment on Fedora
Execute the following commands to install the compilation prerequisites on an Fedora machine:
sudo dnf install csh make gcc-gfortran openmpi openmpi-devel netcdf netcdf-devel netcdf-fortran netcdf-fortran-devel netcdf-fortran-openmpi netcdf-fortran-openmpi-devel
Other recommended tools to use git (and unzip), X windows via SSH and syntax colored vim (restart shell session: logout and login again):
sudo dnf install git unzip xauth xterm vim-enhanced
Install the MOM6 source tree
You are ready to use git to download and setup the code repository on the AWS instance.
Create a blank env file
The following commands make use of a file build/gnu/env that can contain any local environment changes need for compilation:
mkdir -p build/gnu
echo > build/gnu/env
Example contents of env using bash
# Need openmpi path before all other paths to get correct nc-config for netcdf with openmpi
export PATH=/usr/lib64/openmpi/bin:${PATH}
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/lib64/openmpi/lib/pkgconfig
Create a copy of the linux-gnu template and modify
For FC21:
cd src/mkmf/templates
cp linux-gnu.mk linux-gnu-fedora21.mk
# The only real difference is supplying include file and library paths instead
# of using pkg-config. Replace (-) lines with (+) lines. Use different paths
# if you installed netcdf or openmpi in other locations. This demonstrates use
# of openmpi over mpich2.
-FFLAGS += $(shell pkg-config --cflags-only-I mpich2-c)
+FFLAGS += -I/usr/include/openmpi-x86_64 -I/usr/include/openmpi-x86_64/
-CFLAGS += $(shell pkg-config --cflags-only-I mpich2-c)
+CFLAGS += -I/usr/include/openmpi-x86_64 -I/usr/include/openmpi-x86_64/
-LIBS := $(shell nc-config --flibs) $(shell pkg-config --libs mpich2-f90)
+LIBS := $(shell nc-config --flibs) -L/usr/lib64/openmpi/lib
For FC23:
cd src/mkmf/templates
cp linux-gnu.mk linux-gnu-fedora23.mk
# Since we chose to using openmpi in this example, change references of mpich2-* to ompi-*
# in linux-gnu-fedora23.mk
NOTE: Change the template to linux-gnu-fedora21.mk for FC21 as needed in the following examples.
Building libfms.a
To build the Makefile use:
mkdir -p build/gnu/shared/repro/
(cd build/gnu/shared/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths ../../../../src/FMS; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/linux-gnu-fedora23.mk -p libfms.a -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names)
To build the FMS library (libfms.a):
csh/tcsh:
(cd build/gnu/shared/repro/; source ../../env; make NETCDF=3 REPRO=1 FC=mpif90 CC=mpicc libfms.a -j)
bash:
(cd build/gnu/shared/repro/; . ../../env; make NETCDF=3 REPRO=1 FC=mpif90 CC=mpicc libfms.a -j)
Notice the addition of FC=mpif90 and CC=mpicc above, which is necessary to find the MPI header files.
Building the ocean-only MOM6 executable
To create the Makefile for compiling MOM6:
mkdir -p build/gnu/ocean_only/repro/
(cd build/gnu/ocean_only/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths ./ ../../../../src/MOM6/{config_src/dynamic,config_src/solo_driver,src/{*,*/*}}/ ; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/linux-gnu-fedora23.mk -o '-I../../shared/repro' -p 'MOM6 -L../../shared/repro -lfms' -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names)
To build the MOM6 executable:
csh/tcsh:
(cd build/gnu/ocean_only/repro/; source ../../env; make NETCDF=3 REPRO=1 FC=mpif90 LD=mpif90 MOM6 -j)
bash:
(cd build/gnu/ocean_only/repro/; . ../../env; make NETCDF=3 REPRO=1 FC=mpif90 LD=mpif90 MOM6 -j)
This time note the addition of LD=mpif90.
Run MOM6 ocean-only
For any example in ocean_only/ that does NOT have an INPUT/ directory, e.g. double_gyre:
csh/tcsh:
cd ocean_only/double_gyre/
mkdir RESTART
(source ./../../build/gnu/env ; mpirun -np 4 ../../build/gnu/ocean_only/repro/MOM6)
bash:
cd ocean_only/double_gyre/
mkdir RESTART
(. ./../../build/gnu/env ; mpirun -np 4 ../../build/gnu/ocean_only/repro/MOM6)
A successful run should look similar and should match output from a run on an Ubuntu 15.10 system.
This was run on an Amazon AWS instance type m4.xlarge.
MOM Day 0.000 0: En 1.424361E-13, MaxCFL 0.00000, Mass 5.288178268008E+18
Total Energy: 4126FC96B8ADCFE7 7.5322736070108123E+05
Total Mass: 5.2881782680077681E+18, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00)
MOM Day 1.000 72: En 4.580432E-06, MaxCFL 0.00023, Mass 5.288178268008E+18
Total Energy: 42B607A80A852EA4 2.4222139843886641E+13
Total Mass: 5.2881782680077681E+18, Change: -1.7202096100332227E+01 Error: -1.72021E+01 (-3.3E-18)
MOM Day 2.000 144: En 8.958206E-06, MaxCFL 0.00040, Mass 5.288178268008E+18
Total Energy: 42C58AE53347EF57 4.7372590026718680E+13
Total Mass: 5.2881782680077681E+18, Change: -6.7986416459937118E+01 Error: -6.79864E+01 (-1.3E-17)
MOM Day 3.000 216: En 1.166708E-05, MaxCFL 0.00045, Mass 5.288178268008E+18
Total Energy: 42CC0E8B5CE0D7C2 6.1697586479535516E+13
Total Mass: 5.2881782680077681E+18, Change: 1.2942244735786460E+01 Error: 1.29422E+01 ( 2.4E-18)
MOM Day 4.000 288: En 1.542296E-05, MaxCFL 0.00044, Mass 5.288178268008E+18
Total Energy: 42D28B60D589706D 8.1559337444801703E+13
Total Mass: 5.2881782680077681E+18, Change: -1.6385457801725806E+01 Error: -1.63855E+01 (-3.1E-18)
MOM Day 5.000 360: En 1.826022E-05, MaxCFL 0.00053, Mass 5.288178268008E+18
Total Energy: 42D5F4B97D5D6AA0 9.6563312817578500E+13
Total Mass: 5.2881782680077681E+18, Change: 3.7420734548125125E+01 Error: 3.74207E+01 ( 7.1E-18)
MOM Day 6.000 432: En 2.397431E-05, MaxCFL 0.00062, Mass 5.288178268008E+18
Total Energy: 42DCD397A5E1A4E8 1.2678043163201962E+14
Total Mass: 5.2881782680077681E+18, Change: -3.8078863737074414E+01 Error: -3.80789E+01 (-7.2E-18)
MOM Day 7.000 504: En 2.927962E-05, MaxCFL 0.00073, Mass 5.288178268008E+18
Total Energy: 42E19A5109E63388 1.5483585791222025E+14
Total Mass: 5.2881782680077681E+18, Change: -3.5114098448941945E+01 Error: -3.51141E+01 (-6.6E-18)
MOM Day 8.000 576: En 3.486441E-05, MaxCFL 0.00081, Mass 5.288178268008E+18
Total Energy: 42E4F5D995BB638B 1.8436920009398034E+14
Total Mass: 5.2881782680077681E+18, Change: -6.0992962031751858E+00 Error: -6.09930E+00 (-1.2E-18)
MOM Day 9.000 648: En 4.195565E-05, MaxCFL 0.00090, Mass 5.288178268008E+18
Total Energy: 42E9393C9CAC7A8E 2.2186896584802044E+14
Total Mass: 5.2881782680077681E+18, Change: 5.1411840814233216E+01 Error: 5.14118E+01 ( 9.7E-18)
MOM Day 10.000 720: En 4.604729E-05, MaxCFL 0.00100, Mass 5.288178268008E+18
Total Energy: 42EBAEF71B9AAA47 2.4350627232699422E+14
Total Mass: 5.2881782680077681E+18, Change: -1.9384762671898812E+01 Error: -1.93848E+01 (-3.7E-18)
20160512 184045.816: Memuse(MB) at Memory HiWaterMark= 2.965E+01 3.172E+01 8.756E-01 3.020E+01
MPP_DOMAINS_STACK high water mark= 3680
Supplemental instructions on how to start a single instance on the Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
For Fedora Core 23 (FC23), several pre-constructed base images are available from the Community AMIs. The ami name will be different for different compute regions and zones. By searching on "Fedora-Cloud-Base-23" you can see all the different zones and types available. Linux Amazon Machine Images use one of two types of virtualization: paravirtual (PV) or hardware virtual machine (HVM). The main difference between PV and HVM AMIs is the way in which they boot and whether they can take advantage of special hardware extensions (CPU, network, and storage) for better performance. For example, the image ami-20203441 is a HVM image which can take advantage of the special hardware extensions.
Reference: Nic Hannah, http://nh.id.au/run-an-ocean-model-in-30-mins.html
Author note: Once you have an account established with Amazon AWS and are familiar with how to request and login to the instance, it does literally take 30 minutes or less to run the above example.