Expectations for Developers - ufs-community/ufs-weather-model GitHub Wiki

Testing Expectations

  • Run the full RT suite, preferably on Ursa, Hercules, or Derecho (unless changes are specific to a different machine). Developers often choose to add the -e or -r options, too, to expedite testing: 
    ./rt.sh -a <account> -e
  • Commit and push the updated test_changes.list file.
  • Commit and push the log file (RegressionTests_<machine>.log -- under ufs-weather-model/tests/logs). In the case of baseline changes, this log file will show failures, but this is fine as long as the failures are reasonable (e.g., a comparison failure because a test has baseline updates). It is the developer's responsibility to assess whether the failures are reasonable and whether code is scientifically valid.
  • Exceptions: Regression testing is not required for WM-level text-only changes, such as documentation updates, or for updates to the WM CI, which do not touch the model code.

Important

Do NOT run with the -c option when producing the log and test_changes.list file that you plan to upload. Code managers need to see which tests (if any) will fail using the current baselines. Using the -c option will produce passes for baseline-changing tests, and the tests will not be added to test_changes.list.

Code managers will recreate baselines when necessary as part of the PR merge process. Developers are welcome to test with -c and run against their newly created baselines using -m as part of their own testing process to ensure reproducibility, but the logs produced from these steps should not be the ones uploaded to the PR.

Review Expectations

Before we can schedule a PR for testing:

  • All subcomponent PRs must be approved with the correct number of reviewers on each.
  • Pretesting must be complete:
    • Push a log (RegressionTests_<machine>.log) with results from a run of the full RT suite on the most recent code.
      • No increased warnings or remarks
    • Push test_changes.list (even if empty)
  • Fill out PR template. The PR should include:
    • Description
    • Commit message
    • Priority
    • Links to issues that the PR will close
    • Links to sub-PRs
    • Information on baseline changes, input data changes, library updates
⚠️ **GitHub.com Fallback** ⚠️