4a Code based test generators java - ftsrg-edu/swsv-labs GitHub Wiki

The goal of this lab is to try out tools that can generate tests from source or binary code (code-based or white-box test generators).

These tools select relevant test inputs achieving high coverage or triggering exceptions. Moreover, they record the observed behavior and return values and encode them in assertions (in this way these tests could be used in regression testing).

We will try out two tools working on Java bytecode:

• Randoop uses feedback-directed random test generation.
• EvoSuite uses search-based techniques (genetic algorithms),

WARNING: These tools will call the methods in the classes under test with random parameters. Be careful and do not run these tools on code that writes/deletes files, as during test generation it could delete your data!

• Read the introduction and stages of test generation sections of Randoop's manual to get a quick overview about the tool.

• Download the latest version of randoop-all jar from the releases. As of now it is randoop-all-4.3.1.jar.

• Set an environment variable pointing to the downloaded jar file.

  • On Linux adjust the following command to point to the correct location:

  export RANDOOP_JAR=/home/.../randoop-all-4.3.1.jar
  

  • On Windows, use the following syntax and adjust the file location:

  set RANDOOP_JAR=c:\tools\randoop-all-4.3.1.jar
  

• Run the following command to print the help message for generating tests in order to check that everything is working so far.

  • Linux:

  java -cp $RANDOOP_JAR randoop.main.Main help
  

  • Windows:

  java -cp %RANDOOP_JAR% randoop.main.Main help
  

• The rest of the text uses syntax for Linux, adjust it if you are using Windows.

  • Use cmd.exe instead of a PowerShell shell, as PowerShall handles environment variables differently
  • Environment variables can be referred as %RANDOOP_JAR%
  • Pay attention that in Windows the ; character is the separator in the class path.
  • If you encounter the error "Cannot find the Java compiler. Check that classpath includes tools.jar", see this or try to directly call java.exe from the bin folder of a JDK and not a JRE.
  • There is no need to put ./ in the beginning of commands, as the current directory is in the path on Windows.

• Clone the Randoop tutorial.

  • Note: the documentation of the tutorial is a bit out of date, the syntax of some commands might have changed. Follow this guide, or check Randoop's manual about the actual syntax if you encounter an error.

• The tutorial contains the implementation of MyInteger.java, which has an add, equals and multiply methods among others. The tutorial has several steps, where different versions of MyInteger is copied to the source folder.

• Move to the first part of the tutorial.

./gradlew first

• Open the project in your favourite IDE, and examine MyInteger and the current tests in MyIntegerTest. The implementation is simple, but it seems to be fine.
• Run the existing manually created tests:

 ./gradlew test

• Generate tests for the MyInteger class. Randoop needs the compiled class files, and you need to set the Java class path correctly, otherwise Randoop will not found it.
• Navigate to the root folder of the tutorial. The MyInteger.class is located inside the folder build/classes/java/main/math, and its fully qualified name is math.MyInteger.
• Call Randoop to generate tests for Stack:
  • The : character is the separator in the class path (-cp). Note that the testclass is given with its fully qualified name, and the .class extension is not needed.
  • The --testclass specifies the class to generate tests for.
  • The --junit-output-dir sets the folder where the generated tests are placed.
  • The --output-limit parameter sets the limit for the number of generated tests.

  java -cp build/classes/java/main:$RANDOOP_JAR randoop.main.Main gentests --testclass=math.MyInteger --junit-output-dir=src/test/java --output-limit=20

• Execute the generated tests (./gradlew test)

• Examine the generated tests! Randoop generated error-revealing tests (test violating some general contract or some implicit test oracles) and regression tests (tests capturing the current behavior for some generated inputs).

• Investigate the error-revealing tests!

  • What are the test checking?
  • Find the problem in the implementation!

• Use the following command to load a fixed version of the implementation.

  ./gradlew second

• See the fix in the source of the implementation and run again the tests to validate the fixes.

• See "3.3 Discovering a regression error" in the original tutorial for using Randoop to detect regression errors.

• Perform the steps in the tutorial to understand how generated regression tests can help to catch changes in the behavior of the code.

• On recent versions of Randoop, you might need to adjust test generation settings. See this issue for details: https://github.com/randoop/tutorial-examples/issues/2

CHECK Create a screenshot about the generated tests.

Explore the functionality of Randoop by changing the parameters of the test generation. More information can be found in the detailed manual.

Some initial ideas:

• Change the limits (time, output). Is Randoop able to increase the coverage?
• Change the values used in tests (nulls, literals...).
• Change classification of tests (e.g. whether exceptions are considered errors).
• Try to specify expected code behavior with pre- and post-conditions.

CHECK Create a screenshot of the revised test generation commands.

The 1.1.0 version that supports JDK 9+ is not yet available on Maven Central and has some other bugs, therefore we do not use EvoSuite in 2020.

Read the short introduction about EvoSuite summarizing its main features.

EvoSuite can be used from the command line, from Maven, and it has a simple IntelliJ and Eclipse plugin. In this lab we will use the Maven plugin from the command line.

(The rest of this section is based on the EvoSuite Maven tutorial, but it adds some other exercises.)

• Download and unzip the tutorial project:

  wget http://evosuite.org/files/tutorial/Tutorial_Maven.zip
  unzip Tutorial_Maven.zip
  cd Tutorial_Maven

• Take a look at the source code of the project. It has four classes for simple data structures.

• Look at the existing pom.xml.

• Add EvoSuite to the pom.xml as a plugin to the Maven build to be able to call its tasks:

  <build>
    <plugins>
      <plugin>
        <groupId>org.evosuite.plugins</groupId>
        <artifactId>evosuite-maven-plugin</artifactId>
        <version>1.1.0</version>
      </plugin>
    </plugins>
  </build>

• and set the compilation level to Java 11

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>11</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>
  </properties>

• EvoSuite works on Java bytecode, thus compile it first:

  mvn compile

• See what goals the plugin offers:

  mvn evosuite:help

• The generate goal can be used to start test generation. Check its parameters:

  mvn evosuite:help -Ddetail=true -Dgoal=generate

• EvoSuite runs its genetic algorithm and searches for new tests until a time limit is reached (timeInMinutesPerClass). By default this time limit is 2 minutes, but we will first try to generate tests with a bit lower time limit:

  mvn evosuite:generate -DtimeInMinutesPerClass=1

• Observe the output of generate (starting a new job for each class).

• Print the high-level information for the generated tests:

  mvn evosuite:info

• We can see that the generated tests reached quite high code coverage.
• The generated tests are located in the hidden folder .evosuite right now. That folder contains logs and the generated tests (in the best-tests folder) among other. Currently these tests cannot be executed, thus let's export them to the src/test/java folder (note: EvoSuite can keep the generated tests separated from the manual tests, see the original tutorial on how to do this if you are interested):

  mvn evosuite:export

• Observe the generated tests. Every test has two files. The one ending in ESTest_scaffolding.java helps to run the tests in a sandbox (this is important if the code under test uses file system or network calls). The one ending in ESTest.java contains the generated tests in JUnit format.

• Investigate Stack_ESTest.java.

  • What kind of test were generated?
  • What do these tests do?
  • What do the tests with exceptions do?
  • Are there any errors in the classes under test?

• Before we can run the tests, the EvoSuite runtime has to be added as a dependency as it is used by the generated tests. Add the following to the pom.xml:

<dependency>
  <groupId>org.evosuite</groupId>
  <artifactId>evosuite-standalone-runtime</artifactId>
  <version>1.0.6</version>
  <scope>test</scope>
</dependency>

• Run the tests in the usual way:

 mvn test

• Inject an error into the code of the Stack class: change the implementation of Stack.java.

  1. Decrease the capacity to 8.
  2. Return null instead of throwing IllegalArgumentException if pop() is called on an empty stack.

• Run the tests again. Are the tests able to detect the fault?

CHECK: Create a screenshot about the output of the tests.