CTSM_Case With CUPID - ESCOMP/CTSM GitHub Wiki

4) Running a CTSM Case with CUPiD

In this tutorial you will:

  1. Clone CTSM to your work directory
  2. Create and customize a new case
  3. Submit your case
  4. Examine the output and log files created
  5. Look at diagnostic plots generated by CUPID

1) Log onto derecho

NOTE: This tutorial assumes that you've already cloned CTSM with CUPID enabled and have conda environments installed.

1a) Log onto derecho

Log onto a Derecho login node in your terminal. Make sure you are using the bash shell.

[NOTE]: All but the last set of commands are assumed to run in a terminal on Derecho in the bash shell

1b) now that we're on a new machine we need to set your CUPID_ROOT again

cd /glade/work/$USER/ctsm_cupid_2026/CTSM/tools/CUPiD
export CUPID_ROOT=`pwd -P`

2) Create and customize your case

2a) Create a new case

I like to keep my case directories in a common location (here in work). We'll use an environment variable to keep track of it's location.

cd /glade/work/$USER/ctsm_cupid_2026/CTSM/cime/scripts/
export CASES_ROOT=/glade/work/$USER/ctsm_cupid_2026/cases
mkdir -p $CASES_ROOT

export CASE=ctsm5.4.021_clm6_BGCcrop_crujra_4x5_2000

Now create you new case and move into your case directory

export PROJECT=P93300041
./create_newcase --case $CASES_ROOT/$CASE \
  --compset I2000Clm60BgcCropCrujra \
  --res f45_g37  \
  --user-mods-dirs clm/CUPiD \
  --project ${PROJECT} \
  --run-unsupported

cd $CASES_ROOT/$CASE
  • What kind of compset is this?
  • What CESM components are just stubs?
  • What CESM components just read in data?
  • What CESM components are active?
  • What resolution are you running?

2b) Customize your case

Here we set some of the CASE XML variables to control how the case will run...

./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=5
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=1:00:00

# Have it use an existing spunup intiial conditions file and interpolate to your 4x5 degree resolution grid
echo "use_init_interp = .true." >> user_nl_clm
echo "finidat = '/glade/campaign/cesm/development/cross-wg/diagnostic_framework/CESM_output_for_testing/ctsm5.4.002_clm6_BGCcrop_crujra_4x5_HIST/rest/2000-01-01-00000/ctsm5.4.002_clm6_BGCcrop_crujra_4x5_HIST.clm2.r.2000-01-01-00000.nc' " >> user_nl_clm

Check that those changes came through (using the case xmlquery command):

./xmlquery -p JOB,STOP

Will show something like this...

Results in group case.cupid
        JOB_WALLCLOCK_TIME: 01:00:00
.
.
.
        STOP_DATE: -999
        STOP_N: 5
        STOP_OPTION: nyears

2c) Add CUPiD modifications here

Running CUPiD as part of the case submission process requires setting a number of xml variables.

  • We set this up for the tutorial case by using the flag --user-mods-dirs clm/CUPiD when we created this case.
  • You can look at what's being set up with usermods by looking in the shell_commands file that was copied to your case
  • The xml settings for CUPiD are being controlled with shell_commands
  • NOTE: You could also put your changes above in the shell_commands file for the case, and run it so they are applied
  • NOTE: Most of the CUPID XML variables are in the env_postprocessing.xml file

First we'll make sure we turn on LDF output

./xmlchange CUPID_RUN_LDF=TRUE

Then look at all the CUPiD xml settings

./xmlquery -p CUPID

You'll get output that shows all the settings grouped in different sections. Here we highlight a few of them in the cupid_analysis section:

Results in group cupid_analysis
        CUPID_BASELINE_CASE: ctsm5.4.002_clm6_BGCcrop_crujra_4x5_HIST
        CUPID_BASELINE_ROOT: /glade/campaign/cesm/development/cross-wg/diagnostic_framework/CESM_output_for_testing
        CUPID_BASE_CLIMO_END_YEAR: 1995
        CUPID_BASE_CLIMO_N_YEAR: 10
        CUPID_BASE_STARTDATE: 1995-01-01
        CUPID_BASE_STOP_N: 20
        CUPID_BASE_STOP_OPTION: nyears
        CUPID_CLIMO_END_YEAR: 1995
        CUPID_CLIMO_N_YEAR: 5
        CUPID_EXAMPLE: land_only

[NOTE]: It's important to make sure you have a softlink to either the full or subset config file in land_only. This was done in the earlier page, but in case you didn't do it here are the instructions for it. The link to that page is here.

cd $CUPID_ROOT/examples/land_only
ln -s config_subset.yml config.yml
cd ${CASES_ROOT}/${CASE}

Will show all the settings for CUPID_* variables in the env*.xml files in your case.

  • Look: What is the base case we're comparing to?
  • Check: What is CUPiD actually going to run in this example? Do you want to turn anything else on?
  • Check: What CUPiD example is going to be run? What is the base case that will be compared?
  • Hint: setting ./xmlchange CUPID_RUN_ILAMB=TRUE could make things more interesting

You can see this is still a work in progress. We'll work to improve our documentation on how to set this up for the CLM6 release, but hopefully this information is helpful if you want to try things out.

Finally, it's a good idea to make sure the time series and climatology files that were already created for the ILAMB and LDF part of this tutorial (page 3) are gone. This will prevent LDF from getting confused when you do the current analysis. BUT, if you're careful about the time periods you're analyzing this can be a time saver.

rm -rf /glade/derecho/scratch/$USER/cupid_output/ctsm5.4.002_clm6_BGCcrop_crujra_4x5_HIST

2d) Run in our special tutorial queue with the LMWG account

export PROJECT=P93300041
./xmlchange PROJECT=$PROJECT

NOTE: You do need to use the force here! Since this queue isn't one we normally recognize you have to force the change, as otherwise it'll sayERROR: Unknown Job Queue specified use --force to set.

2e) Setup and create the namelists for your case

./case.setup
./xmlchange --force JOB_QUEUE=tutorial
./preview_namelists

At this point make sure the namelist creation is successful. It will fail if there is a problem with one of the XML settings above. And you can take a moment to look over the namelists in the CaseDocs subroutine and make sure they make sense. For example check that the finidat file you set above was actually used in the CaseDocs/lnd_in file it created.

2f) Build the case

qcmd -A $PROJECT -- ./case.build
  • Remember: building a case takes some time, typically about 5 minutes
  • Check: does this case look like you'd expect?
  • HINT: You can compare lnd_in files with the baseline case.

3) Submit your case

./case.submit

Now you can spend some time getting to know your neighbors, looking at other tutorial materials, or exploring web pages you made earlier in the tutorial

You can also check progress on your case:

qstat -u $USER
  • Look! What jobs do you see running or pending in the queue? How is this different from what you're used to seeing?

[NOTE]: You may need to submit your cupid case more than once in order to get everything setup correctly. Or you might want to do so in order to experiment with some settings. If you JUST want to run CUPID do this...

./case.submit --only-job case.cupid

4) Examine the output files and logs created

Checkpoint #1

Congratulations! At this point you have created a case running CTSM, created input namelists, built it, and submitted it to the batch queue. Look around in the files in your case directory. After the case runs, see what appears in the Case run directory, and when the short term archiver runs, see where files get moved to. Once, CUPiD has run see what input and output files it created and where they were put.

Suggested paths to capture

export DOUT_S_ROOT=`./xmlquery --value DOUT_S_ROOT`
export CASEROOT=`./xmlquery --value CASEROOT`
export RUNDIR=`./xmlquery --value RUNDIR`
export CIME_OUTPUT_ROOT=`./xmlquery --value CIME_OUTPUT_ROOT`
export CASE=`./xmlquery --value CASE`
export CUPID_OUTPUT="$CIME_OUTPUT_ROOT/cupid_output/$CASE"

CTSM Case Files

After the CTSM case has run output files will be placed in the '$RUNDIR' directory. Input namelist files will be there (identical to the ones under your $CASEROOT/CaseDocs directory. Output log files and history files will also be there. Log files are per component so look for 'cesm.log.', 'lnd.log.', 'atm.log.', 'cpl.log.', and 'drv.log.*'.

However, note that most of the the history files as well as all of the log files will get moved at the next step.

Short Term Archive History Files

After the st_archive script runs, it will copy files into the $DOUT_S_ROOT location. CTSM output history files go under '$DOUT_S_ROOT/lnd/hist/' for example.

CUPID Output Files

cupid-postprocessing directory under your case directory

After CUPiD runs for your case it will create a 'cupid-postprocessing' directory under your case directory ($CASEROOT). The top level has the config files used to configure and run CUPiD. Sub-directories have the notebooks, images and web-pages.

CUPiD Postprocessing Directory

$CUPID_OUTPUT has the time-series and climatology files created underneath it. These are the data files to make the plots.

Batch output

In the case directory each submission of a job will create batch log files like 'run.$CASE.oNNNNNNN' where 'NNNNNNN' is the batch submission number. 'run.$CASE.o*' is the CTSM case batch output, 'st_archive.$CASE.o*' is the short term archiver batch output, and 'cupid.$CASE.o*' is the cupid batch output.

  • Remember: after the short term archive script runs the log files and most history files will be moved
  • Look: what are the files and directories in the cupid-postprocessing directory?
  • Question: if the CTSM case fails, what files might be useful to look at?
  • Question: if the short term archiver step fails, what files might be useful to look at?
  • Question: if CUPiD fails, what files might be useful to look at?

5) Look at diagnostic plots generated by CUPID

After you case completes successfully, it will run the short term achiever and then the CUPID diagnostics. They were already in the queue, but Held until the case is successful.

As in the previous page, you can copy webpages directly to your local machine

  • Open a terminal locally
  • Move to your Desktop
  • Set your username on derecho in the export command, below.

[NOTE]: This last bit is run on your Desktop in the bash shell

export DUSER=<derecho user name>
scp -r [email protected]:/glade/work/$DUSER/ctsm_cupid_2026/cases/ctsm5.4.021_clm6_BGCcrop_crujra_4x5_2000/cupid-postprocessing/computed_notebooks/_build/html/LDF .

To view the webpage, open the LDF/ctsm5.4.021_clm6_BGCcrop_crujra_4x5_2000_2000_2004_vs_ctsm5.4.002_clm6_BGCcrop_crujra_4x5_HIST_1995_2014/website directory on your desktop and then open index.html.

Next conclusions and next steps