3 Model based test generators - ftsrg-edu/swsv-labs GitHub Wiki

The goal of this lab is to try out model-based testing (MBT) and a model-based test generator tool.

In the lab we will use the open-source GraphWalker tool. (There are many more MBT tools, see e.g. this overview.)

Overview of GraphWalker

GraphWalker in a nutshell: GraphWalker

  • uses an Extended Finite State Machine (EFSM) model for test design,
  • can generate paths from the graph model using different algorithms and coverage criteria, and
  • uses Java code for the adaptation layer to connect the model to the implementation to support test execution.

Modeling

  • Models are stored in JSON format and can be created in GraphWalker Studio.
  • The model is a combination of a Usage and System model (according to the terminology used in the lecture). The SUT is modeled from a testing point of view:
    • Edges represent the input events coming from the users or the environment of the SUT. Input events do not have parameters.
    • Output actions of the SUT are not modeled, but the nodes of the model represent the states, where the SUT should move after the given input events.
  • The model can contain variables that can be initialized in nodes and manipulated in actions on edges. The variables are to be used in guards on edges.
  • GraphWalker can work with several models to create hierarchical models using the SHARED keyword in nodes.
  • Many GraphWalker examples use the v_ prefix for nodes (vertex) and e_ for edges, but it is not mandatory.

Paths generated from the model are the test sequences:

  • edges represent receiving inputs and performing some action in the SUT, and
  • nodes represent performing some validation that the SUT indeed behaved as expected and responded and/or moved to the expected state.

The recommendation of the authors of GraphWalker is to try to model first with simple FSMs, and use only EFSMs (variables) when necessary, as it greatly increases complexity.

Test generation

GraphWalker supports the following generators:

  • random and weighted random walk,
  • A* (shortest path to a specific vertex or edge),
  • quick random.

GraphWalker supports the following stopping conditions:

  • edge or vertex coverage [%],
  • requirement coverage [%],
  • reach a specific edge or vertex,
  • time duration.

Tests can be generated offline or online:

  • offline: GraphWalker only prints the generated paths. These can be used to validate the model or store them to execute later in some way.
  • online: GraphWalker is connected to the SUT and executes each step on the SUT before generating the next step. Test adaptation code is required to map events in the model to the actual implementation.

SEE: Generators and stop conditions

Test adaptation

An adaptation layer is required to concretize the abstract model elements and map them to the elements of the implementation.

GraphWalker can use Java code for adaptation. From the adaptation code any test execution framework or tool can be called (e.g. simple Java code if the SUT is a Java library, Selenium if the SUT is a web application or Appium if the SUT is a mobile app).

The structure of the adaptation code is the following:

  • GraphWalker generates an interface from the model, where for each node and edge there is a method without parameters.
  • Tests should implement this interface and call the SUT in the methods for edges, and verify the SUT's response or state using assert or verify calls in the methods for nodes.

Usage of GraphWalker

GraphWalker is quite flexible and can be used and configured in several ways.

Command line interface (CLI)

GraphWalker binary (graphwalker-cli-X.Y.Z.jar) needs to be downloaded from the project's website, and called from the command line.

The location of the model, the generator and stopping condition to use can be specified as parameters.

Maven

GraphWalker can be used directly in a Maven build just by adding GraphWalker as a dependency in the pom.xml. We will use this method in the current lab.

The models should be placed inside the src/main/resources or test/main/resources folder in a folder structure similar to the application's package hierarchy.

There are several goals for GraphWalker, e.g.:

  • graphwalker:generate-test-sources: generates the interface from the model. The interface is placed inside the target/generated-test-sources/graphwalker/ folder.
  • graphwalker:test: will call GraphWalker for all classes annotated with the @GraphWalker annotation.

The generator and stopping conditions can be specified as parameters in the @GraphWalker annotation. In this case no @Test methods are needed. For example:

@GraphWalker(start = "e_Init", pathGenerator = RandomPath.class, stopCondition = VertexCoverage.class, stopConditionValue = "100")

(Note: An other option is to specify generators and stopping conditions imperatively in Java code in JUnit @Test methods. In this case the usual test goal should be called instead of graphwalker:test. However, this is not the recommended setup.)

SEE: GraphWalker Maven (see the various goals in wiki pages on the right)

Exercises

IMPORTANT: Use the latest version of GraphWalker, as previous versions were not working on newer JDKs.

Try out GraphWalker Studio

Check out the home page of GraphWalker, and watch the short animations and read the text to understand:

  • how models can be created in the Studio web app,
  • what is the basic usage process of the tool.

Try out GraphWalker Studio:

  • Download the latest GraphWalker studio.
  • Start it using the following command: 
    java -jar graphwalker-studio-4.3.1.jar
  • Open the following URL in your browser: 
    http://localhost:9090/studio.html
  • Create a vertex, set it as start element. Create an other vertex, and connect them with and edge.
    • Remember: v should be pressed while clicking for creating a vertex. e should be pressed while holding down the left button and dragging the cursor from one vertex to another for creating and edge. (Yes, really.)
    • Some laptops disable the touchpad when a key is pressed. If you are using a laptop, and creating a vertex or an edge does not work, try to use a mouse or check the configuration of your touchpad.
  • Simulate the model with the Play button.
  • Save the model. It will download it as test.json that can be later renamed.

Windshield wiper exercise

In this exercise we will create the model from a specification, and write the adaptation code interacting with a Java implementation. (The system to model is greatly simplified compared to reality to fit the timeframe of the lab.)

Specification

The system under test is an automatic, rain-sensing windshield wiper controller. The controller receives the position the user sets for the wiper switch with possible values off, speed one, speed two or auto. The controller in response sets the wiping interval for the wiper motor: for speed one it is 1 second and for speed two it is 0.5 seconds. In the auto mode the controller acts accordingly to the data received from the rain sensor: if the rain sensor reports light rain then it sets the interval to the same as used for speed one, while for heavy rain it sets the interval of speed two. If there is no rain, then it turns off the wiper (interval value 0).

Your task is to test that the controller sets the correct interval values.

Project setup

Use the WindshieldWiper skeleton project from the model-based-testing folder in this repository. Clone the swsv-labs repository and navigate to the appropriate folder.

  • The project consists of three sub-modules:

    • wiper-api: interface of the SUT,
    • wiper-impl: implementation of the SUT only available as a JAR file,
    • wiper-test: project for the MBT model and test code.

  • The actual SUT is available in a jar file. In the wiper-impl folder execute the following command to install it to the local Maven repository:

    mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file   -Dfile=wiper-impl-1.0-SNAPSHOT.jar

    (If you are using Windows, use a Command Prompt for these commands and not a PowerShell window as PowerShell will not interpret the parameters correctly.)

  • Change back to the WindshieldWiper folder and install the other projects in the local repository (in this way the test module will always find the api module).

    mvn install

Tasks

  • Create a model representing the behavior of the SUT.

    • Open the WiperModel.json file from wiper-test/test/resources in Studio as a starter.
    • Pay attention that the model should be abstract and not necessarily contain every detail. The model should use keywords (events, states) from the specification and the domain, and not from the implementation.
    • Try to work in an iterative way: create first a model without the auto mode.
    • Did you find anything ambiguous or missing in the specification during model development?

  • Save the model in the same json file.

  • Change to the wiper-test folder and check the syntax of your model with the command

    mvn graphwalker:validate-test-models

  • Take a look at the wiper-api project. It consists of the following main parts:

    • Interface of the SUT: WiperController
    • Two enums used in the interface: WiperMode, RainMode
    • The SUT calls the wiper motor, which has the following interface: WiperMotor

  • Take a look at the wiper-test project. From now on work in this project.

    • A sample method in WiperControllerTest shows how the SUT can be instantiated (the implementation for the WiperMotor is not available, therefore some kind of test double needs to be used).

  • Generate an interface from the model with

    mvn graphwalker:generate-test-sources

    • It will create an interface in the target folder inside generated-test-sources.
    • Run the mvn graphwalker:test command. It will result in an error eventually ("No start context found"), but will compile the WiperModel interface that can be later used.
    • Take a look at the generated interface: it has the @Model annotation for the class, and @Edge and @Vertex annotations for the model elements.
    • Note: GraphWalker will not generate a method for the start vertex as this is more like an initial pseudo-state. Place your initialization code in the method generated for the edge going out from start.

  • The task is now to implement this interface in a test adaptation code.

    • Create a class called WiperModelTest under src/test/java in the proper package hierarchy.

    • Add the following annotation, extends and implements directives to the class:

      @GraphWalker(start = "init")
public class WiperModelTest extends ExecutionContext implements WiperModel{}

    • Implement the methods defined in the interface:

      • Methods for edges need to call the SUT and perform the appropriate actions.
      • Methods for nodes need to check the state or interactions of the SUT.

    • As WiperMotor is not implemented yet, use Mockito to mock it.

  • Add a suitable generator and stopping condition to the @GraphWalker annotation.

  • You can start generating tests with

    mvn graphwalker:test

  • Did you find any problems in the implementation?

CHECK: create a screenshot of your final model.

CHECK: Create a screenshot of your final test code and create a screenshot from the output of GraphWalker.