Test plan - cma-open/cmatools GitHub Wiki

The test plan aims to ensure that at all times the main branch holds working, tested, high quality code. This will be achieved by:

  • all new development will be accompanied by new tests
  • regular development will run the helper shell scripts to check that tests pass during development
  • tests will cover: unit, integration, user-interface, and end-to-end tests
  • new code will be accepted into main branch via Pull Requests where automted tests and Codacy test results must pass, before a merge is acccepted
  • the enture system will be regularly re-built via system documentation as a fresh build from scratch (e.g. every month) to test documentation and all system tests

Refer to the "Python package development" notes on testing

Testing checklist

Using the example command line tool "cli-simple" the following test coverage is developed:

  • Unit tests fine grained tests to cover each function, class method or class object where a function includes a call to another function this can be "mocked" to ensure only the main function of interest is tested simple, trivial units need not always be tested

  • Integration tests testing where the system software integrates with any other API, system, network, database this includes testing of network access, disk write/access, http access, data downloads this page uses integration tests in the scope of "narrow integration tests" see

  • End-to-end tests test the full scope of the software system from inut to user interaction and output does running the software give the correct outputs? these tests may run slower and will be fewer in number that other types of test

  • User interface tests testing how the user interacts with the software system includes command line tools, API calls, GUIs these tests may, or may not, require the full system to be run where the user interface is coded as a microservice, class or other modular unit, then linked modlues may be able to be mocked out

  • Installation tests confirm that the system installs and is accessible to users

  • Test coverage confirm test coverage is at a high level

  • Documentation tests confirm the new functionlity has been documented and the documentation builds as expected