Best Practices for Testsuite Development

Testsuite Development

This wiki describes the various aspects of testcase development and testing to produce a testsuite for IVCT.

Inheritance

An IVCT testcase extends the AbstractTestCase class which provides the framework for the testcase. The AbstractTestCase class forces the testcase writer to decide which testcase code belongs in the preamble, testbody or postamble sections. The testcase writer should try to only have the code in the testbody which is applicable to the test purpose: in this way a FAILED verdict applies to the testpurpose and not to any other problem. If an error occurs in the preamble or postamble, the verdict shall be INCONCLUSIVE. The AbstractTestCase class also provides the method to update the percentage completed, which lets the testsystem operator see how far the testcase has advanced and judge how long it may still run.

Testsuite Libaray

A testcase generally needs a specific set of support methods to perform either complex sequences of teststeps, or complex test logic or other code which is used in more than one testcase. The parsing the the TcParam.json value is also done in this library.

Logging

The Logback logging framework is used to perform testcase logging. The testsystem provides the functionality of dynamically setting the logging level. The testcase writer shall provide the test purpose and operating instructions for the test system operator. These texts will be logged in the testcase log and form part of the testcase report. The IVCT IVCT_LoggingFederateAmbassador and IVCT_RTIambassador classes are be used for any RTI calls. These classes encapsulate the RTI calls and perform logging of the parameters used in each call. The testcase should display its progess as an estimated percentage of expected running time.

Compilation

The testsuite gradle files should use https://oss.sonatype.org/content/repositories/snapshots to get the necessary IEEE1516e, MessagingHelpers and tc_lib jar files. This makes it possible to develop testcase without installing the test engine. Only the jar files containing the testcases, the specific testsuite library and any libraries not provided by the test engine, are needed in the lib directory created by gradlew install: this is achieved by using gradle compileOnly for all other libraries. The runtime environment is provided by the test engine: a testcase is not expected to run standalone.

Configuration

The SUT Conformance Statement The IVCT Testsystem selects the testcases applicable to an SUT by reading the Badges stated in the SUT conformance statement. There is a certain standard logic used by gradle for copying files into an install directory: whatever is found in src/main/resources is included in the installed jar file. This makes it somewhat difficult for a user to edit: which may be alright for FOM/SOM files, but not for other configuration files. In this case, there should be sample files with some sample parameters made available outside of the jar files. A testcase needs

1. The FOM/SOM files needed for the HLA Federation
2. The TcParam.json

Runtime

There are several ways to configure the IVCT Test System:

1. Defaults are used
2. The file $IVCT_CONF/IVCT.properties is available
3. Environment variables are used If any environment variables are found they are used, else if any other variables exist in IVCT.properties these values are used, else if any other variables are not found the defaults are used.

The test engine provides the environment to start and monitor a testcase. The test engine finds the necessary components required through the configuration above:

1. SUTs are found via the IVCT_SUT_HOME_ID environment variable
2. Badges are found via the IVCT_BADGE_HOME_ID environment variable
3. The testsuite libraries and configuration files are found via the IVCT_TS_HOME_ID environment variable

Testing the Testcase

It is recommended that a test federate be produced to reflect the behaviour expected or deliberately false behaviour by a testcase. This will help the testcase writer to verify if the testcase is complete by checking the various possible behaviours of an SUT.

Junit

For each method in the testsuite library, check if making a junit test is appropriate and feasible.

Docker

The IVCT test system is configured to also run in a Docker environment. This means that a Docker docker-compose.yml file shall be produced to make the testsuite as a Docker image.