CTSM development workflow - ESCOMP/CTSM GitHub Wiki

This is a work in progress

This is a work in progress, and should be viewed as just a place-holder for now.

Testing Protocols for Developers and Scientists

The following test suites must be run prior to submitting a pull request to have code accepted into the ctsm master branch. These tests provide reproducible evidence that the code is working the way you expect it to be working.

Full test suite

Running Tests

Running the test suite:

    cd cime/scripts
    ./create_test --xml-category aux_ctsm \
        --machine ${MACHINE} --compiler ${COMPILER} \
        --xml-machine aux_ctsm --xml-compiler aux_ctsm \
        --compare ${BASELINE_CTSM} \
        --test-root ${TEST-ROOT} \
        --test-id ${TEST-ID}e

    ./create_test --xml-category clm_short
        --machine ${MACHINE} --compiler ${COMPILER} \
        --xml-machine cheyenne --xml-compiler intel \
        --compare ${BASELINE_CLM} \
        --test-root ${TEST-ROOT} \
        --test-id ${TEST-ID}5

where ${ALLCAPSWORD} should be replaced with the appropriate values for the machine and user. These can either be set on the command line or in a script using 'set', e.g.

set TEST-ROOT=testrootexample

  • MACHINE is the computer on which you are working (cheyenne, yellowstone, hobart etc.)

  • COMPILER is, well, the compiler you want to use. gnu, intel, nag or pgi.

  • TEST-ROOT is the path to a directory where all of the test cases will be created. (user defined)

  • TEST-ID is a specific name for this set of tests (user defined - there is no wrong answer. example: MMDDfire or 0515fire). But NOTE:

    • If you are putting all the tests into the same test root directory, then the testid's must be unique to ensure proper results. The simplest way to do this is add an 't', or '5' the the end of the id, indicating ctsm or clm50.
    • The test suite occasionally has file name length problems. To avoid these issues, keep the testid to about 10-12 characters.

  • BASELINE_X is a path to a set of baseline simulations which are defined for each answer-changing commit to the main repository master. Numerous tests determine whether the code modifications have changed the answers relative to the most-recent baseline. ED and CLM will generally have different baselines. See Determining your baseline below.

NOTE: the clm_short test suites are not setup the same way as the ed test suite. You must specify xml_mach=cheyenne and xml_mach=intel regardless of what machine and compiler you are using.

Test Results

from tests output directory ${TEST_ROOT}:

 /${PATH_TO_GIT_CTSM}/ctsm/parse_cime.cs.status -s cs.status*

Single tests

The full test suite takes several hours to run and isn't usable for incremental, day to day development where quick, i.e. minutes, turn around is needed. For quick testing, you can run or two single tests that cover the code you are working in or a general case to ensure you aren't breaking things.

    cd cime/scripts
    ./create_test -testname SMS_D_Ld5.f45_f45.${COMPSET}.${MACHINE}_${COMPILER}.clm-default -testid junk-dev -compare ${BASELINE_X}
    cd SMS_D_Ld5.f45_f45.${}COMPSET.${MACHINE}_${COMPILER}.clm-default.C.junk-dev
    ./case.test_build
    ./case.submit
    cat TestStatus
    # did everything pass?

You can run other tests manually, for example to test the fire code, change the above test name to: ERS_D_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edFire.

Determining your baselines

The version of master your branch is up-to-date with determines your baselines for ctsm and clm. You need to analyze your branch history to determine which version of master you are based on. To see the history of your clone:

git log --graph

Look through the history for merge commits. One of the first two commit comment lines will generally contain a description of the merge, something like:

  • Merge branch 'ctsm-some-branch' into master the commit id of this commit is your CTSM baseline.
  • update ed master to clm4_5_12_r195 the clm trunk is your clm baseline.

NOTE: You do provide good commit messages when you commit code, right? The ability to provide good commit messages is why you should not use the '-m' option to commit.

Machine specific baselines

  • On yellowstone:

    • baselines are stored in /glade/p/cesm/cseg/ccsm_baselines/
    • ctsm baselines are in directories following the convention ctsm-HASHID where the hash id corresponds to the master commit in question.
    • clm baselines are stored according to the clm trunk tag naming convention, e.g. clm4_5_12_r196.

  • On cheyenne

    • baselines are stored in /glade/p/cesm/cseg/cesm_baselines

Testing Protocols for SE's

Same as for scientists. While scientists only need to run tests on their production machine with one compiler, SE's should run the test suite with all supported compilers, currently gnu, intel, nag and pgi. The tests should be run on the final commit to master to ensure that there are no semantic conflicts in the final merge. Semantic conflicts are changes in meaning between the master and branch that did not result in a text conflict but change behavior. The final merge to master should be include the detailed commit message that is self contained explaining the changes. The required information includes: the test status, answer changes, user interface changes. The simplest, most efficient, and least error prone way to do this testing is use leverage git.

  • Create a small script to process a pull request, ctsm-pr.sh:
#!/usr/bin/env bash

PR_NUM=$1
PR_REPO=$2
PR_BRANCH=$3

[email protected]/CTSM.git

PR_DIR=pr${PR_NUM}
mkdir -p ${PR_DIR}

cd ${PR_DIR}
git clone ${REF_REPO}

cd ctsm
git remote add pr ${PR_REPO}
git fetch pr
git checkout ${PR_BRANCH}
git checkout master
git merge --no-ff ${PR_BRANCH}
  • Initiate the merge:
ctsm-pr.sh PR_NUM [email protected]:SOME_USER/ctsm.git some_branch
# The PR should merge cleanly. Enter a minimal commit message.
  • Clone the merge repo to hobart:
ssh hobart
git clone ssh://[email protected]/PATH/TO/SOMEWHERE/prXYZ/clm-cstm prXYZ

  • Kick off testing on hobart using the 'ctsm' suite and 'nag' compiler. Generate new baselines in the system default location as clm-cstm-prXYZ

  • Once nag tests have been verified to compile, kick off testing on cheyenne:

    • cstm test suite with gnu, intel and pgi compilers. Generate new baselines in the system default location as clm-cstm-prXYZ

    • aux_clm or clm_short test suites with gnu, intel, pgi.

  • Verify that results are as expected:

    • All functionality tests must pass.

    • Are ctsm baselines expected to be bit for bit? Expect changes limited to a limited subset of fields? All fields?

    • Are all CLM baselines bit for bit?

    • If there are any unexpected results:

      • Update the pull request on git hub with failure information.

      • Delete or rename the merge clone

      • Remove the baselines generated for the PR.

  • If all tests are ok, verify there are no source changes in the PR directory. Amend the commit message by copying the info from the pull request message into the message. Add final testing status.

  • Push to central repo master.

  • Determine the changeset id of the new master. Rename the baselines on cheyenne and hobart: mv ctsm-prXYZ ctsm-CHANGESET_ID.