Test configuration - cma-open/cmatools GitHub Wiki

The testing of a python package can be setup, undertaken and configured in many ways.

The selected method should:

  • Be repeatable and give consistent results
  • Test the code from a stated perspective or use case, e.g. installed code, vs local source code

Testing package and frameworks

  • This training package uses pytest - a popular python testing package allowing testing via the command line, scripts, and with a wide variety of configuration options
  • Testing frameworks such as Tox and Nox further extend the range of testing options and can be combined with pytest

Test runs and configuration options that affect testing

  • Current location from which pytest is called
  • Installation status of the system under development
  • Presence / absence of isolated virtual environments / conda envronments
  • Optional testing flags to return additional useful information from test runs, such a ssummary reports, terminal ouputs per test, etc

Testing methods and configuration options include:

  • Documentation to record the use of command line options. The docs must state the directory locations from which to run commands
  • Test shell or python scripts to capture the calling of the test package, with selected options and for selected tests / directories
  • Customised test configuration files to store or automate selected or project-specific test options and to ease the calling of the test package from either manaul command line calls or via scripts
  • Customised warning message of prompts to report situation where tests are being called incorrectly or inapproripately
  • Structuring the python package to facilitate effective test development and use

Pytest notes:

  • "If no arguments are specified then collection starts from testpaths (if configured) or the current directory. Alternatively, command line arguments can be used in any combination of directories, file names or node ids."
  • "pytest determines a rootdir for each test run which depends on the command line arguments (specified test files, paths) and on the existence of configuration files. The determined rootdir and configfile are printed as part of the pytest header during startup."
  • "rootdir is NOT used to modify sys.path/PYTHONPATH or influence how modules are imported."
  • "If no args are given, pytest collects test below the current working directory and also starts determining the rootdir from there."

Some common recommendations, tips and configuration settings are:

  • Capture common config patterns in a pytest config file https://docs.pytest.org/en/stable/customize.html
  • Use a src based package structure. Pytest Good Practises recomend that if tests are included as a package (e.g. to avoid test name conflicts and available to users after install) then "it is strongly suggested to use a src layout where application root package resides in a sub-directory of your root". "This layout prevents a lot of common pitfalls and has many benefits.". See
  • Test installed code to ensure the system will work for end users
  • Automate testing where possible (GitHub actions, CI pipleines etc)
  • Use testing frameworks to drive tests , e.g. Tox, Nox and test against a range of environment to prepare eg for python or dependency version upgrades
  • Run the tests via a method that is suited to the chosen package structure and test framework. e.g. for packages where the tests are include din line with the code and installed as a package use the --pyargs option, pytest --pyargs mypkg, here pytest will discover where mypkg is installed and collect tests from there. For project structured with a src/lib directory and tests placed at root, then tests can be called via pytest tests. See this page for summary notes on the impact of package structure on testing https://github.com/cma-open/cmatools/wiki/Package-structure-and-testing
  • Set any project specific test options in a config file, so they are automatically set each time tests are called (https://docs.pytest.org/en/stable/example/simple.html)

Alternative recommendations, not used in this project

Supporting notes