Combinatorial testing [Optional]

We will try out combinatorial testing by creating a test input set satisfying n-wise coverage.

You can use one of the following tools:

• ACTS: Java program, has a GUI; free, but registration required,
• PICT: C++ program (can be built on Windows/Linux/Mac), only CLI; open source.

See the materials on the course website (only visible after logging in with your BME eduID).

Both tools have detailed user guides.

The method to test is the following:

int calculateEngineInput( GearMode g, bool economyMode, int acceleration )

• the domain of GearMode is {Backward, None, Parking, Drive},
• acceleration can have values between 0-3.

We want to test the combinations of the parameters, but do not want to try all the 32 possible combinations. We first check only the combinations of any two parameters (2-wise or pairwise coverage).

Exercises

1. How many tests are needed for pairwise coverage? Try to create such a test input set!

2. Create a new model in the tool, and define the above parameters.

3. Generate a pairwise test set! How many combinations did the tool select?

4. After refining the requirements of the system it turns out that not all combinations are possible: economyMode could not be true in case of Backward. Add this constraint to the tool, and generate a new test set! What changed?

   1. ACTS uses similar syntax for constraints (see user guide for details):

      (OS = "Windows") => (Browser = "IE" || Browser = "FireFox")

   2. PICT uses similar syntax for constraints (see user guide for details):

      IF [File system] = "FAT32" THEN [Size] <= 32000;

5. In a new version of the system a new parameter was introduced: BrakeStatus with values {Pressing, Releasing, None}. Modify the model and create a new test set!

6. Modify the configuration of the tool to create a test set with 3-wise coverage! How many combinations were selected?

CHECK Create a screenshot about the last test suite.

This simple example showed how combinatorial testing can be used in practice. The tools and techniques are even more useful when the system under test has many more parameters. Each tool comes with larger sample models, try to load them and generate test sets for them.