Test Generation and Execution - DevOps-MBSE/AaC GitHub Wiki

Generation

AaC unit testing focused on an ‘as utilized’ effort using happy path testing via the standard user usage mechanisms. This carried forward the IRAD basis of AaC to allow users to utilize early versions of the product while allowing AaC development to focus on rapid feature additions and modifications rather than bullet proofing software. Some key sad path testing was also included. Once AaC matured to a sufficient point the test coverage percentage was kept at a high value.

As AaC development continued, additional sad path was added which was focused on common exceptions and exception handling. This enhanced the testing of normal usage mechanisms by testing the most common sad paths encountered by users passing in bad or unexpected inputs.

To better support and enforce this team goal, the unit test template for gen-plugin was modified to stub out method reminders for happy path and sad path tests to promote better test development and test coverage as new functionality gets added to the tool.

In some cases there is additional detailed testing of a more typical ‘unit’ test variety. Furthermore, certain key code blocks within AaC are tested from different entry points. For instance, the parse_and_load and parse methods are tested from a more than one user entry point. This allows additional testing of those key code blocks to be exercised.

In conclusion, AaC has good test code coverage. This initially started as happy path testing during early development. Code test coverage was augmented with sad path testing to establish baseline high coverage numbers. Enhancements of the unit testing strategy focused on common sad paths, exceptions, exceptions handling, and more robust testing of certain key common code blocks.

Execution

AaC's test suite is "managed" via Tox: User Guide - tox. This allows for both local execution of the test suite through the IDE of choice or within the pipelines for automated runs. Tox will run all of the found test_*.py files which house any internally generated and modified Pythonic unit tests. Tox will run all of the found *.feature files which house any internally generated and modified Behavior tests. Thus allowing a wider range of test cases and testing methods to be utilized for establishing a reliable and robust tool. 

Within a repository's tox.ini configuration file we incorporate the two testing commands:

python -m unittest
behave

Doing so results in executing all of the found corresponding test files when the tox command is entered on the command line for local runs or within the script of the testing pipeline stage for automated runs. 

Tox is also able to be configured to test against multiple Python versions, allowing for compatibility testing. Currently Python 3.9.x, 3.10.x, and 3.11.x are included for running both the unit tests and the behavior tests against to confirm integration compatibility with various Python versions.