Running operational requirements tests - ufs-community/ufs-weather-model GitHub Wiki

What are the Operational Requirements Tests (ORT)?

  • A suite of tests
  • Each test addresses a single requirement that a new implementation should satisfy:
    • Thread (THR) reproducibility: varying the number of threads produces the same results
    • MPI process (MPI) reproducibility: varying the number of MPI tasks produces the same results
    • Domain decomposition (DCP) reproducibility: varying the tile layout of FV3 produces the same results
    • Restart (RST) reproducibility: restarting produces the same results
    • 64/32 (BIT): changing to double/single precision can be compiled and simulation can be run to completion
    • Debug (DBG): can be compiled and simulation can be run to completion in debug mode
    • Diagnostic reproducibility test (FHZ)

Why ORTs?

  • The purpose of running ORTs is to satisfy the operational implementation requirement.
  • Currently, regression tests conduct unit tests only for 'control' test
  • Bugs that break the unit tests and are found later during the model implementation stage could be very difficult to debug
    • Identify problems earlier in the development process
  • Developers ensure their implementations pass unit tests before requesting a Pull Reqeust

Procedure to Run ORTs

For the most up-to-date information on running ORTs, see the WM User's Guide ORT documentation.

  • Developers finish code changes and do their own tests
  • Pick a test case under ufs-weather-model/tests/tests that is relevant to their code changes
  • Modify or add a test case if no test exists that can test their code changes
  • $ cd ufs-weather-model/tests
  • $ ./opnReqTest -n <test-name>
  • Check and commit the unit test report/logfile to the developer's branch
  • Make a pull request

Examples (using fv3_ccpp_control as <test-name>)

  • To run all tests: $ ./opnReqTest -n fv3_ccpp_control
  • To run THR test only: $ ./opnReqTest -n fv3_ccpp_control -c thr
  • To run MPI and DCP tests only: $ ./opnReqTest -n fv3_ccpp_control -c mpi,dcp
  • To run DBG and RST tests only: $ ./opnReqTest -n fv3_ccpp_control -c dbg,rst

Command Line Interface

  • $ ./opnReqTest -h: displays usage of shell script
  • $ ./opnReqTest -n <test-name>: runs all unit tests for test-name
  • $ ./opnReqTest -n <test-name> -c <test-case>: runs a specific unit test case (thr, mpi, dcp, rst, bit, dbg, fhz)
  • The full set of option flags is:
    • -a: specify HPC to use for batch job
    • -n: specify
    • -c: specify ; defaults to all test-cases: thr,mpi,dcp,rst,bit,dbg,fhz; comma-separated list of any combination of std,thr,mpi,dcp,rst,bit,dbg,fhz
    • -b: test bit reproducibility; compare against baseline
    • -d: test debug reproducibility; compare against baseline
    • -s: test reproducibility for std; compare against baseline
    • -x: skip compilation
    • -z: skip running (i.e., only compiles)
    • -k: keep run directory
    • -e: use ecFlow workflow manager

Input Files

  • utest.bld
    • Specify required build options in this format: <test-name> | <build_opt>
    • <build_opt> can be WW3=Y, CCPP=Y, etc.
  • <test-name>, <INPUT_NML>, <FV3_RUN>: all similar to those in regression tests

Output Files

  • UnitTests_<machine>.<compiler>.log: unit test logfile (commit)
  • Compile_ut_<machine>.<compiler>.log: compile logfile (do not commit)
  • ufs-weather-model/log_ut_<machine>.<compiler> directory (do not commit)

File Locations Summary

  • utest: ufs-weather-model/tests
  • utest.bld: ufs-weather-model/tests
  • <test-name>: ufs-weather-model/tests/tests
  • UnitTests_<machine>.<compiler>.log: ufs-weather-model/tests
  • Compile_ut_<machine>.<compiler>.log: ufs-weather-model/tests
  • log_ut_<machine>.<compiler> directory: ufs-weather-model/tests
  • Run directory
    • On Hera: /scratch1/NCEPDEV/stmp2/$USER/FV3_UT/ut_*
    • On Orion: /work/noaa/stmp/$USER/stmp/$USER/FV3_UT/ut_*
    • On Dell P3: /gpfs/dell2/ptmp/$USER/FV3_UT/ut_*

Details on Unit Test Configuration

  • THR: Number of threads = 2, Tasks per node /= 2
  • MPI: JNPES /= 2, WRITE_GROUP = 2, WRTTASK_PER_GROUP = 12
  • DCP: Swap INPES and JNPES
  • RST: use restart files at middle of the run (SHOUR+FHMAX/2) and compare results at end of the run (SHOUR+FHMAX)
  • BIT: able to compile and run to completion with different precision
  • DBG: able to compile and run to completion in debug mode
⚠️ **GitHub.com Fallback** ⚠️