Testing - cma-open/cmatools GitHub Wiki

Current context = cmatools holds several command line interface tools that allow analysis to be called from simple command line commands

Testing of such software occurs at different levels. The naming of each level is not standardised, but it is widely agreed that testing should involve very many unit tests that cover the code base at a fine grained level, such as for each class method and function. These should be small and quick to run. Other tests should cover the way the developed system interacts with other systems (integration tests). Tests should ensure that the system functions as expected, and runs correctly accross a whole analysis flow or pipeline (e.g from inputs to outputs). A limited number of tests should also cover how the software is used and interacted with by the end user.

Tests directory

  • impact and options for tests dir
  • behaviour via pip install -e . vs pip install .

Tests structure

  • Setup the test (inputs, required data, input commands)
  • Call the test function
  • Assert the results are as intended

Test types

  • Unit

    • fine grained tests to cover each function, class method or class object
    • where a function includes a call to another function this can be "mocked" to ensure only the main function of interest is tested
    • simple, trivial units need not always be tested

  • Integration

    • testing where the system software integrates with any other API, system, network, database
    • this includes testing of network access, disk write/access, http access, data downloads
    • this page uses integration tests in the scope of "narrow integration tests" see

  • User interface

    • testing how the user interacts with the software system
    • includes command line tools, API calls, GUIs
    • these tests may, or may not, require the full system to be run
    • where the user interface is coded as a microservice, class or other modular unit, then linked modlues may be able to be mocked out

  • End-to-end

    • test the full scope of the software system from inut to user interaction and output
    • does running the software give the correct outputs?
    • these tests may run slower and will be fewer in number that other types of test

Other types of testing are currently out of scope for this example system repo (to be added at a later date)

  • Acceptance tests

    • A correctly specified Agile development User Story will include defined acceptance tests.
    • These allow a tester to determine when the story is complete and passes

  • Non-functional tests

    • https://en.wikipedia.org/wiki/Non-functional_testing
    • Many aspects of the software system may be specified by the product owner, or tester that do not completely rely on the functionality of the system
    • These include documentation, security, performance, resilience, stress, compliance
    • Relevant tests ares include:
      • system runs in less than x sec / mins
      • system documentation builds without errors
      • system runs with x active users
    • The scope of these tests may overlap with user intrface and end-to-end testing, but viewing test creation from these narrowly defined perspectives may help to create these tests

Sources