Milestone: CMEPS 0.5 Appendix Graduate Student Test Evaluation SST Experiment - ESCOMP/UFSCOMP GitHub Wiki

This test has closed! For information about how to join Graduate Student Tests that are currently open, please see the GST page on the UFS Portal.

This test assesses how easy it is to access, run, and modify source code to perform a sensitivity experiment with a coupled atmosphere-ocean-ice model. It is an example of a code evaluation following the "graduate student test" protocol developed for the Unified Forecast System (UFS).

The model code is described here. The sea surface temperature (SST), which is provided by the ocean model component (Modular Ocean Model 6 or MOM6) to other system components, will be increased by 2°C by modifying a specific source file in the ocean model "cap".

The term "cap" is used because it is a small software layer that sits on top of model component code, making calls into it and exposing its data structures in a standard way.

As a result of the increased SST provided by the MOM6 ocean model, the surface heat flux components (especially latent heat flux, or evaporation from the ocean surface) in the surface boundary will change and impact the coupled model results. The changes in the model result will not be same for all regions due to the local effects (e.g. latitude, distance to the ocean), and two-way feedback mechanism between atmosphere-ocean and ice-ocean components and their non-linear interaction.

In order to perform the test, the model is run without the modification for 5 days, and then the output is compared to the output from the same configuration with the increased SST. As described on the CMEPS 0.5 milestone page, both runs are performed by running the cold start sequence for one hour first, and then using the mediator restart option (MEDIATOR_READ_RESTART=TRUE) to run the warm start sequence for 5 days.

Note: The January 2012 initial condition is the default and is used in the rest of the document.

Please refer to Build and Run section on CMEPS 0.5 milestone page for specific instructions.

To perform this test, you will need:

• an account on either theia, cheyenne, or Stampede2 computers and an allocation where you can charge the runs.
• a GitHub account and permission to access the FV3GFS repository, as explained in the Download, Build, and Run section.

1. Create a new case called "REF" and perform a run following the instructions on the Download, Build and Run page.

2. The results of the simulation can be checked quickly with the methods explained in the Looking at Outputs section.

1. Create a new case called "SST_2".

2. Goto case directory and copy MOM cap methods file to the SourceMods directory.

$ cd ufs.s2s.c384_t025.sst_2
$ cp ../../../components/mom/./MOM6/config_src/nuopc_driver/mom_cap_methods.F90 SourceMods/src.mom/.

3. Next the SST will be modified in the MOM6 cap. Documentation for the MOM6 cap shows the complete list of export fields from that component, including SST. To expedite the modification, the specific changes that need to be made are outlined below.

Edit the mom_cap_methods.F90 file under SourceMods/src.mom directory and modify the SST variable as shown. The green lines with + show added and the red lines with - show removed code.

Screenshot for difference between original source file and the modified one

4. Change model parameters to make 1-hour cold start run. Don't forget to change the ice initial condition file. The model also needs to be build again to create executable.

5. After a successful cold start run, run the model for 5-days with warm start. Please also note that you need to increase the job wall clock time (JOB_WALLCLOCK_TIME and USER_REQUESTED_WALLTIME) for a longer run. You need to be in the case directory to issue these commands. For example,

$ ./xmlchange JOB_WALLCLOCK_TIME=06:00:00
$ ./xmlchange USER_REQUESTED_WALLTIME=06:00:00

As a result of the modified SST field, the surface flux components over the ocean (especially latent heat flux) are modified and it leads to the change in the planetary boundary layer (PBL). The effect of the modified SST growth in time is that it starts to dominate the solution of the modeling system.

As a part of the tutorial, the atmospheric model output will be examined briefly to reveal the effect of the modified SST.

Alternative 1:

A simple way to look at the difference is to use the combination of netCDF Operators (NCO) and Ncview. To use those tools, the following modules need to be loaded.

# On Cheyenne: 
module load nco/4.7.9
module load ncview/2.1.7
# On Stampede2:
module load nco/4.6.9
module load netcdf
module load hdf5 udunits ncview/2.1.7
# On Theia:
module load intel
module load nco
module load ncview

Then, NCO (ncdiff command) can be used to calculate the difference between two different NetCDF files.

ncdiff /scratch/01118/tg803972/ufs.s2s.c384_t025.sst_2/run/ufs.s2s.c384_t025.sst_2.mom6.sfc._2012_01_04_75600.nc /scratch/01118/tg803972/ufs.s2s.c384_t025.jan.ref/run/ufs.s2s.c384_t025.jan.ref.mom6.sfc._2012_01_04_75600.nc diff.nc

The resulting file (diff.nc) can be visualized by ncview.

# Do not forget to use -X option when you login using ssh to forward X-window
ncview diff.nc

If you get an error like the following when you run the ncview command, just log out and log in again.

Copyright (C) 1993 through 2015, David W. Pierce
Ncview comes with ABSOLUTELY NO WARRANTY; for details type `ncview -w'.
This is free software licensed under the Gnu General Public License version 3; type `ncview -c' for redistribution details.

Note: could not open file /home1/01118/tg803972/.ncviewrc for reading
Error: Can't open display: localhost:58.0

A simple output of the ncview command can be seen as the following:

Screenshot of ncview

This method can be applied to any other pair of NetCDF files to look at different datasets and variables, which are not supported by a given NCL script. It is also possible to modify NCL scripts to do that but it requires experience in programming and NCL scripting.

Alternative 2:

In FV3GFS, the surface variables are stored in the files with a sfcf prefix and they are written at hourly intervals for the first 12-hours of simulation and at 3-hour intervals for the rest. Each time step is stored in a separate file. This data output behavior is controlled with the FV3GFS namelist file called (input.nml and diag.table) and can be modified to get the desired output.

For this experiment, the latent heat flux from the atmospheric model output will be processed using the NCAR Command Language (NCL). The simple NCL scripts needed to create 2d spatial difference plots can be downloaded from here.

The script needs to be slightly modified to create the latent heat plot. First edit the parameters section of the script (plot_2d_atmos.ncl) as follows:

...
  time_step = 120 ; time step to create plot (in hours)
  variable = "lhtfl_ave" ; variable that is used to create plot (options: lhtfl_ave, tprcp)
  basedir = "/scratch/01118/tg803972" ; scratch folder that stores the run
...  

The time_step parameter indicates the time step (in hours). In this example, it is set to 120 (end of the 5-days simulation) but it could be changed to analyze other dates (see the run directory for the list of available time steps). lhtfl_ave is selected as a variable to create the plot but it is also possible to select other variables if they are defined in the NCL script. Next, the basedir needs to be changed to point to the SCRATCH directory that stores the simulation results. This directory will be different and needs to be changed in your case. Note that the NCL script automatically selects the correct file based on the selected option in the parameters section.

The case names must be changed in the dataset section. The default script uses ufs.s2s.c384_t025.jan.jun26.ref as a name for REF and ufs.s2s.c384_t025.jan.jun26.sst_2 for SST_2 simulations but these could be changed to the case names used in the simulations.

...
  dsets = (/ (/ "CMEPS-REF  ", basedir+"/ufs.s2s.c384_t025.jan.jun26.ref/run"   /), \
             (/ "CMEPS-SST_2", basedir+"/ufs.s2s.c384_t025.jan.jun26.sst_2/run" /) /)
... 

The NCL script can be run with the following commands:

# Load NCL module 
# On Cheyenne: 
module load intel/18.0.5 ncl/6.6.2
# On Stampede2:
module load ncl_ncarg/6.3.0
# On Theia:
module load ncl

# Run the script
ncl plot_2d_atmos.ncl

This will produce a pdf file (plot_2d_atmos_lhtfl_ave.pdf) with the output as shown in Fig. 1a.

Figure 1a: Map of latent heat flux (lhtfl_ave, W/m2) and the difference for July 2012 initial condition on 2012-01-05.

Just by changing the variable name to tprcp, running the provided NCL script creates a spatial distribution of the total precipitation and the difference between two runs (Fig. 1b).

Figure 1b: Map of total precipitation (tprcp, mm/day) and the difference for July 2012 initial condition on 2012-01-05.