Package structure examples - cma-open/cmatools GitHub Wiki

Examples of python package structure

GitHub source repos viewed in Jan 2021

Python software systems differ greatly in complexity and structure. As projects increase in maturity and support, from a simple module to an advanced framework, a user would expect there to have been an increase in development effort and testing and a typical user may also be expected to be more of a passive end-user e.g. without the need to access and run the tests, compared to a more active or collaborative user of a module or library who may want to access and run tests and interogate the code more closely.

The way that a variety of python systems have been structured is noted below, along with notes on the use of version.py files and highlighting the use of the "tests" directory at root, or as a subpackage, and if this is installed for use by users or not.

These notes are mainly for reference and were used when deciding how to structure the training python package project.

---

Possible categories include:

• Simple module
• Command-line tool / Command-line application
• Application / "App"
• Package / Package with subpackages
• Libaries
• Frameworks (e.g. dash)

---

The main repos checked to inform package structure for shareable packages, libraries, command line tools and applications were:

• iris, cartopy, matplotlib
• pip, pytest, black
• dash, dvc, poetry

The summary python strutures chosen for this cmatools project, following assessment of these repos, were:

• Tests directory as a package (with init)
• Tests directory at root of the main package
• Main package nested within a src directory

The advantages of this are:

• Developers can run the tests after cloning the repo.
• Testing will be focussed on installed code, rather than accidental testing of local source code

Advice based on these main sources

• https://docs.pytest.org/en/stable/goodpractices.html
  • For "Tests outside application code layout" the tests dir can be a simple directory or a package.
  • If designed as a package this allows tests with the same name not to cause conflicts but can potentially cause problems when installing code via test frameworks such as Tox.
    • "In this situation, it is strongly suggested to use a src layout where application root package resides in a sub-directory of your root"
• The "Tests as part of application code layout" also works in conjunction with the "src layout".
• https://blog.ionelmc.ro/2014/05/25/python-packaging/
• https://blog.ionelmc.ro/2014/06/25/python-packaging-pitfalls/

Conflicting advice

• https://docs.python-guide.org/writing/structure/
  • Your library does not belong in an ambiguous src or python subdirectory.
  • No reasoning or examples given

---

There are various pros and cons of the alternative structures for setting up a python package:

• Tests directory at root, not a package
  • Useful when end users do not need to access the test suite and therefore installation code downloads are minimised
  • As the tests directory is not a package it is not installed during setup
• Tests directory at root, as a package
  • Useful when end users do not need to access the test suite and therefore installations code downloads are minimised
  • This structure typically involves the test package being actively excluded from the installation by setup.py
  • If installed in site-packages during setup any package named "tests" would cause namespace conflicts

---

Dash (framework) https://github.com/plotly/dash

dash
  subpackage
  modules
  version.py
tests
  assets
  integration
  unit
setup.py

• The tests directory is a package (with init)
• The tests directory is at root
• The main package is at root
• The test package is not installed by setup.py ( packages=find_packages(exclude=["tests*"]), )
• Users cannot run the tests after installation

DVC (command-line tool / application) https://github.com/iterative/dvc

dvc
  subpackages
  modules
  version.py
tests
  func
  remotes
  unit
setup.py

• The tests directory is a package (with init)
• The tests directory is at root
• The main package is at root
• The test package is not installed by setup.py ( packages=find_packages(exclude=["tests*"]), )
• Users cannot run the tests after installation

setuptools (command line tool / application) https://github.com/pypa/setuptools

docs
setuptools
  modules
  subpackages
  tests
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is at root

poetry (tool / application) https://github.com/python-poetry/poetry

docs
poetry
  subpackages
  modules
  __version.py__
tests
  subdirectories
pyproject.toml

• The tests directory is a subpackage (with init)
• The tests directory is at root
• The main package is at root

---

pip (application) command line tool / application) https://github.com/pypa/pip

docs
src
   pip
     subpackages
     modules
tests
setup.py

• The tests directory is a package (with init)
• The tests directory is at root
• The main package is nested within a src directory

sampleproject https://github.com/pypa/sampleproject

data
src
  sample
tests
setup.py

• The tests directory is a package (with init)
• The tests directory is at root
• The main package is nested within a src directory

datashader

datashader
  subpackages
  tests
  modules
docs
examples
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is at root

xarray

doc
xarray
  subpackages
  modules
  tests
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is at root

panderas

docs
pandera
  modules
scripts
tests
setup.py

• The tests directory is a package (with init)
• The tests directory is at root
• The main package is at root

scikit learn

doc
examples
sklearn
  subpackages
  modules
  tests
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is at root

cartopy

docs
examples
lib
  cartopy
    modules
    subpackages
    tests
      modules
      subpackages
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is nested within a lib directory

numpy

doc
numpy
  subpackages(multiple)
  tests
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is at root

iris (scitools)

docs
  iris
lib
  iris
    subpackages(multiple)
    tests
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is nested within a lib directory

sphinx

doc
sphinx
tests
setup.py

• The tests directory is a package (with init)
• The main package is at root
• The tests directory is at root

rasterio

docs
rasterio
  modules(multiple)
  cli-tool subpackage
    modules
scripts
tests
setup.py

• The tests directory is a package (with init)
• The main package is at root
• The tests directory is at root

pytest

doc
  en
scripts
src
  pytest
    modules
  non-public pytest
    subpackage
    modules
testing
setup.py

• The testing directory is not a package (no init)
• The main package is nested within a src directory
• The testing directory is at root

pandas

doc
pandas
  subpackages
  tests
scripts
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is at root

pytorch

docs
scripts
test
  modules
  subdirectories
torch (main package)
  subpackages
  modules
setup.py

• The tests directory is not a package (no init)
• The tests directory is at root
• The main package is at root

matplotlib

doc
lib
  matplotlib
    tests
  mpl_toolkits
    subpackages
    tests
src
setup.py

• The tests directory is a subpackage (with init)
• The tests directory is nested within the main package
• The main package is nested within a lib directory