Work in Progress

The main development work of Poplog predates the popularity of unit-testing frameworks. One of the challenges in putting unit testing into Poplog is the absence of any distinct 'compilation' phase in Poplog's workflow and hence no natural place to add testing hooks. Also it should always be possible to compile code that has unit tests without any overhead. This note describes a design that overcomes these challenges.

The key idea is to separate unit-tests for a file SOURCE.p into another file (say) TESTS.p. When we compile SOURCE.p the only action taken is to register the link between SOURCE.p and TESTS.p. This link can be allowed to be a many-to-many relationship; no complications arise from this. We can pick up the filename using popfilename - if it is false then uses-unittests will have no effect (maybe a warning?). We also need to capture the popsection so we can reinstate that when we run the TESTS.p file.

The TESTS.p files will be compiled from source each time the tests are run. This means that top-level constants get recalculated on each test and subtle compile-time issues are avoided.

The test definition syntax needs to allow for an optional variable name, an optional descriptive name, an optional identification of the procedure that is under test. The minimal unit-test looks like:

define :unittest;
    ;;; Arrange
    lvars test_data = [ 1 2 3 ];
    ;;; Act
    lvars result = reverse( test_data );
    ;;; Assert
    assert test_data = result;
enddefine;

Unit-tests may be anonymous i.e. have no variable associated with them. This relieves the programmer of the burden of finding complicated naming conventions. Because this unit-test is not bound to a variable, the only effect is that the test is registered using register_unittest( unittest ).

And a fully instantiated unit test looks like:

with
    subject = reverse,
    expects = 'the reverse of a short list is equal to a reversed copy' 
define :unittest test_reverse();
    ;;; Arrange
    lvars test_data = [ 1 2 3 ];
    ;;; Act
    lvars result = reverse( test_data );
    ;;; Assert
    assert [ 3 2 1 ] = result;
enddefine

This declaration binds to the variable test_reverse, which is useful for testing during development. Once the test is developed the variable should be declared as lconstant. The test is also registered.

Registering a unit test only has an effect inside the unittest runner (run_unittests) where the test is added to the list of tests to run. Outside of the unittest runner it will simply run the test immediately as if the test runner was invoked on just that one test.

To run unit tests inside VED we will use the ENTER test RETURN command (i.e. ved_test()). This will run the unit tests associated with this source file. To run all tests associated with files in the same parent directory we write ENTER test . or all tests that are associated with the grandparent directory we use ENTER test ...

This will report a green status on the command line. A red status is reported by opening a new write-protected buffer and adding the results there (using "text" format).

To run unit tests at the top-level the procedure run_unittests(path: string, format?: word) can be used. The allowed formats will be "markdown", "xml" and "text". The default will be "text". The results are sent to cucharout.

The define :unittest foo; syntax will declare foo and bind it to the unit-test object. It can then be called from inside another unit test like this:

foo()

It will be reported as a subsidiary test of the main test.

We often need to run the same test function over a lot of data. It is not enough to merely run a loop over test-data because then the final test-report will show only the first failure and will lack detail. Hence is useful to be able to dynamically generate unit tests. We accomplish that with the newunittest(test_name: word, subject: procedure, test_description: string, test_body: item* -> (), test_data: iterable<item>) procedure:

lvars mytest = newunittest(
    "mytest", 
    reverse, 
    'the reverse of a short list is equal to a reversed copy', 
    procedure();
    endprocedure,
    []
)

Having built a unit test, it can be called in order to execute the test & add to the test-results.

mytest()

When SOURCE.p is recompiled, we want to clear away old registrations associated with SOURCE.p. This can only be reliably achieved if we can identify a recompilation. To do this, we will modify sysCOMPILE so that it has a unique value (an integer) associated with a new dlocalised variable pop_sysCOMPILE_ID. That ID is only guaranteed to be unique within a single Poplog process (it will count up from 0).

sysCOMPILE is a protected variable, so it is straightforward to redefine it.

When pop_debugging is false, we are very likely to be compiling the code without any intention to use unit tests. If that is the case we do not want to be doing any registrations at all. On the other hand, we might be wanting to unit test the case when pop_debugging is false.

To accommodate that less common case, I propose that registration is controlled an the active variable pop_unittesting.

;;; undef means default to pop_debugging.
lvars pop_unittesting_backingstore = undef;

define active pop_unittesting;
    if pop_unittesting_backingstore == undef then pop_debugging
    else pop_unittesting_backingstore
    endif
enddefine;

define updaterof active pop_unittesting( v );
    v -> pop_unittesting_backingstore
enddefine;