General Tips
Some general tips for running CAM MPAS. Because CAM and CESM options can be hard to find….
Turn off the short turn archiver
By default, once your run finishes ESMF will move your output files and your log files to /glade/scratch/user/archive. (i.e. DOUT_S = True). You can turn it off by:
./xmlchange DOUT_S = false
Change the run time and run option
./xmlchange STOP_N=1
./xmlchange STOP_OPTION=’ndays’
Other options for STOP_OPTION are ndays, nmonths, nsteps etc.
Change the requested wallclock time when you submit on PBS
The default is 12:00:00.. which significantly adds to the time jobs are held in the PBS queue.
./xmlchange JOB_WALLCLOCK_TIME=hh:mm:ss
General Instructions
git clone [email protected]:ESCOMP/CAM.git
cd CAM
git checkout cam_development
(Checkout externals - MPAS repository will be pulled)
./manage_externals/checkout_externals
Grid information
You can find all the grids that are currently supported in CAM-MPAS by running:
./cime/scripts/query_config --grids | grep “alias: mpasa”
I’m not sure what it means when a grid is registered or not, or what this output means as you still need to set the grid file to ncdata in user_nl_cam file. The current output of the above command is below:
alias: mpasa480_mpasa480 (only for compsets that are not _POP )
alias: mpasa120_mpasa120 (only for compsets that are not _POP )
alias: mpasa60_mpasa60 (only for compsets that are not _POP )
alias: mpasa30_mpasa30 (only for compsets that are not _POP )
alias: mpasa15_mpasa15 (only for compsets that are not _POP )
alias: mpasa12_mpasa12 (only for compsets that are not _POP )
alias: mpasa15-3_mpasa15-3 (only for compsets that are not _POP )
But you’ll need to find the above grids someplace. A few places you can look:
CESMDATA Input dir:
/glade/p/cesmdata/inputdata/atm/cam/inic/mpas
Michael’s cesmdata input dir:
/glade/work/duda/cesmdata/inputdata/atm/cam/inic/mpas/
My cam-mpas meshes directory
The mpasa15-3_L32.notopo.inic.nc currently is invalid, so avoid using it at this time. The meshes marked with ‘notopo’ can be used for aquaplanet runs. Only a few meshes in this directory have terrain, but the meshes with no topography and the few with topography should be able to meet most of your needs.
/glade/work/mcurry/meshes/cam-mpas/
You can find decompositions for the ‘notopo’ files in /glade/work/mcurry/meshes. You’ll just have to find the corresponding graph.info file (i.e. mpasa15_L32.notopo.init.nc decomp is x1.2621442.graph.info).
Compsets
FKESSLER
Check out CAM and grab externals
git clone [email protected]:ESCOMP/CAM.git
cd CAM
git checkout cam_development
# Checkout externals - MPAS repository will be pulled
./manage_externals/checkout_externals
Create a new FKESSLER Case
./cime/scripts/create_newcase --compset FKESSLER --res mpasa120_mpasa120 --pecount 36x1 --run-unsupported --input-dir /glade/p/cesmdata/inputdata --project NMMM0013 --case YOUR_DIRECTORY/FKESSLER
(Of course replace NMMM0013 with your account key.)
Change to the case directory and set CAM config options.
cd /glade/scratch/duda/FKESSLER
./xmlchange CAM_CONFIG_OPTS="-nadv_tt 6 -chem none -phys kessler -nlev 26"
./xmlchange STOP_N=15
./xmlchange STOP_OPTION='ndays'
Set up the case, this will generate namelist files
./case.setup
Edit the the user_nl_cam file with the following:
ncdata = '/glade/work/duda/cesmdata/inputdata/atm/cam/inic/mpas/mpasa120_L26_FKESSLER.init.nc'
fincl1 = 'PS','PRECL','Q','CLDLIQ','RAINQM','T','U','V','OMEGA'
mpas_block_decomp_file_prefix = '/glade/work/duda/cesmdata/inputdata/atm/cam/inic/mpas/mpasa120.graph.info.part.'
(Optional) Changing the number tacers
If you want to change the number of tracers say for the FKESSLER run you can change -nadv_tt to another value, but you’ll need to set test_tracer_names in user_nl_cam to be test_tracer_names = ''.
Build and run the case
./case.build
./case.submit
FHS94 - Held Suarez 1994
An easy case to run as the grid you use won't need initial conditions data, instead CAM uses the FHS94 analytic data for the run.
Check out CAM and grab externals
git clone [email protected]:ESCOMP/CAM.git
cd CAM
git checkout cam_development
# Checkout externals - MPAS repository will be pulled
./manage_externals/checkout_externals
Create a new FHS94 Case
./cime/scripts/create_newcase --compset FHS94 --run-unsupported --input-dir /glade/p/cesmdata/inputdata --project NMMM0013 --res mpasa480_mpasa480 --pecount 36x1 --case YOUR_DIRECTORY/fhs94
(Of course replace NMMM0013 with your account key.)
Change to fhs94 case and alter CAM_CONFIG_OPTS to use the correct number of vertical levels (default is 26) that are within your grid file
./xmlchange --append CAM_CONFIG_OPTS=’-nlev 32’
# Change these values as you desire
./xmlchange STOP_N=1
./xmlchange STOP_OPTION=’ndays’
./xmlchange DOUT_S=’False’
./xmlchange JOB_WALLCLOCK_TIME=00:30:00
Run ./case.setup and alter user_nl_cam
./case.setup
Set the following values in user_nl_cam:
ncdata = '/glade/work/mcurry/meshes/cam-mpas/mpasa480_L32_notopo_grid_c201216.nc'
mpas_block_decomp_file_prefix = '/glade/work/mcurry/meshes/cam-mpas/mpasa480.graph.info.part'
mpas_len_disp = 480000
mpas_dt = 2880
Run ./case.build and ./case.submit
./case.build
./case.submit
Creating CAM-MPAS Grids
The following steps allow you to create a CAM-MPAS grid with topography (if you have a netcdf file with a PHIS field) or an aquaplanet.
1) Load the following modules:
ncarenv/1.3
intel/19.0.5
ncarcompilers/0.5.0
mpt/2.22
netcdf-mpi/4.7.3
pnetcdf/1.12.1
pio/2.4.4
nco/4.7.9
2) Create a clone of my fork of the MPAS-Model repository:
git clone https://github.com/mgduda/MPAS-Model.git
3) Checkout the init_atmosphere/cam_mpas_grid branch:
cd MPAS-Model
git checkout init_atmosphere/cam_mpas_grid
4) Compile the init_atmosphere core to produce the init_atmosphere_model program:
make -j4 ifort CORE=init_atmosphere USE_PIO2=true
5) Edit the namelist.init_atmosphere file so that it looks like the following:
&nhyd_model
config_init_case = 13
config_start_time = '0000-01-01_00:00:00'
config_theta_adv_order = 3
config_coef_3rd_order = 0.25
/
&dimensions
config_nvertlevels = 32
config_nsoillevels = 1
config_nfglevels = 1
config_nfgsoillevels = 1
/
&vertical_grid
config_specified_zeta_levels = 'L32.txt'
config_nsmterrain = 0
config_smooth_surfaces = true
config_dzmin = 0.3
config_nsm = 30
/
&decomposition
config_block_decomp_file_prefix = 'x1.40962.graph.info.part.'
/
6) Create a text file named 'L32.txt' with the following:
0.0
249.011837562501
703.531419818076
1292.10981710262
1989.09940291778
2779.67781449656
3653.84453807477
4604.25829188187
5625.23125557809
6712.18749270867
7861.34152240467
9069.49372565253
10333.8930366799
11652.1408801514
13022.1216721823
14441.951129401
15909.9369194232
17424.54810654
18984.3910164145
20588.1898827207
22234.7711205107
23923.0503939826
25652.0218679986
27420.7491881842
29228.3578455617
31074.0286623913
32956.9921953429
34876.5238964129
36831.9399053092
38822.5933722713
40847.8712295661
42907.1913447131
45000.0
7) (Optional) If you have a netcdf file that contains the same number of cells as the grid you want to convert and the file has a PHIS field you can use it to add topography to your CAM-MPAS grid. Copy a topo file with a PHIS field to your working directory, and rename the 'ncols' dimension to 'nCells':
cp /glade/scratch/duda/topo/mpas_120_nc3000_Co060_Fi001_MulG_PF_Nsw042.nc .
ncrename -d ncol,nCells mpas_120_nc3000_Co060_Fi001_MulG_PF_Nsw042.nc
8) If you followed step number 7 and have a netcdf file with a PHIS field then you can edit the streams.init_atmosphere file so that it looks like the streams example below. If you do not have a grid with a PHIS you can delete the cam_topo stream entirely. With no cam_topo stream specified, an aquaplanet CAM-MPAS grid will be generated.
<streams>
<immutable_stream name="input"
type="input"
filename_template="/glade/work/duda/static_moved_to_campaign/x1.40962.grid.nc"
input_interval="initial_only" />
<immutable_stream name="output"
type="output"
filename_template="mpasa120.grid.nc"
packages="initial_conds"
output_interval="initial_only" />
<stream name="cam_topo"
type="input"
filename_template="mpas_120_nc3000_Co060_Fi001_MulG_PF_Nsw042.nc"
input_interval="none">
<var name="PHIS"/>
</stream>
<!-- streams below here are ignored when creating a 3-d CAM-MPAS grid -->
<immutable_stream name="surface"
type="output"
filename_template="sfc_update.nc"
filename_interval="none"
packages="sfc_update"
output_interval="86400" />
<immutable_stream name="lbc"
type="output"
filename_template="lbc.$Y-$M-$D_$h.$m.$s.nc"
filename_interval="output_interval"
packages="lbcs"
output_interval="3:00:00" />
</streams>
9) Run the init_atmosphere_model program with 1 MPI task. If you'd like to run this in parallel, specify the path to an x1.40962.graph.info.part.N file in the namelist.init_atmosphere file.
The resulting mpasa120.grid.nc file should be usable with CAM-MPAS.