Package structure and testing - cma-open/cmatools GitHub Wiki

This page holds a brief summary of issues relating to testing python packages and how the structure of the package influences the calling and bahaviour of tests.

There are two main recommended package structures and methods for calling tests, and a third with similarities to 2.

These are based on the following:

  • It is better to use the features and functions of established packages rather than creating unneccessary custom code (e.g. init files or setup.py)
  • It should not be neccessary to modify the python path directly via custom code or via steps in documentation

1). Package is within a src directory and the test package at root

src
  package
tests
setup.py
  • Run the tests from project root via pytest or pytest tests
  • This method minimises the risk of accidentally running the tests against local source code, rather than the installed code
  • The tests are prominently located at root and this facilitates a test directory structure organised by test type

Example packages with this structure include: pip, pytest, black, flask

2). Package is at root and tests is a subpackage within the main package

package
   tests
setup.py
  • Run the tests via pytest --pyargs package
  • This takes advantage of the fact that tests are part of the package and pytest automatically locates and tests the installed code
  • This uses the recommended best practice stated by Pytest documentation (for situations where tests are a package inline with code)
  • Do not call tests from the project root via pytest package/tests or pytest this is because the expected behaviour of the tests differs between full and development (editable) installs. For full installs, this method does not test the installed code.

Example packages with this structure include: datashader, setuptools, xarray, scikit-learn, numpy, pandas,

3). Package is within a lib directory and the test package is a subpackage of the main package

This structure variation is used by several well known packages. I'm currently unclear of the benefits of this structure for testing compared to structures 1 or 2.

lib
   /package
      /tests
setup.py
  • Run the tests via pytest --pyargs package
  • This takes advantage of the fact that tests are part of the package and pytest automatically locates and tests the installed code
  • This uses the recommended best practice stated by Pytest documentation (for situations where tests are a package inline with code)
  • Do not call tests from the project root via pytest package/tests or pytest this is because the expected behaviour of the tests differs between full and development (editable) installs. For full installs, this method does not test the installed code.

Example packages with this structure include: iris, cartopy, matplotlib,