CyBench T2B agent (cybench-t2b-agent) allows you to run your project unit test as JMH benchmarks annotated with:

• JUnit4 - @org.junit.Test
• JUnit5 - @org.junit.jupiter.api.Test
• TestNG - @org.testng.annotations.Test

See cybench-t2b-agent readme.

• Step 1: to run T2B agent from Maven, edit POM of your project first by adding these properties and profiles:

  <project>
      <...>
      <properties>
          <...>
          <!-- ### Java 9+ OPTIONS ### -->
          <t2b.module.prop></t2b.module.prop> 
          <!-- ### Config for JUnit5 launcher - default one, and it is also capable to run JUnit4 tests ### -->
          <t2b.test.runner.class>org.junit.platform.console.ConsoleLauncher</t2b.test.runner.class>
          <t2b.test.runner.class.args>
              <YOUR_TESTS_LAUNCHER_ARGUMENTS>
          </t2b.test.runner.class.args>
          <...>
      </properties>
      <...>
      <profiles>
          <profile>
              <id>java9-plus</id>
              <activation>
                  <jdk>[9.0,)</jdk>
              </activation>
              <properties>
                  <!-- ### Java 9+ OPTIONS ### -->
                  <t2b.module.prop>--add-exports=java.base/jdk.internal.loader=ALL-UNNAMED
                      --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED
                  </t2b.module.prop>
              </properties>
          </profile>
          <!-- Profile to use JUnit4 tests launcher --> 
          <profile>
              <id>run-junit4</id>
              <properties>
                  <!-- ### Config for JUnit4 launcher ### -->
                  <t2b.test.runner.class>org.junit.runner.JUnitCore</t2b.test.runner.class>
                  <t2b.test.runner.class.args>
                      <YOUR_TESTS_LAUNCHER_ARGUMENTS>
                  </t2b.test.runner.class.args>
              </properties>
          </profile>
          <!-- Profile to use TestNG tests launcher -->
          <profile>
              <id>run-testng</id>
              <properties>
                  <!-- ### Config for TestNG launcher ### -->
                  <t2b.test.runner.class>org.testng.TestNG</t2b.test.runner.class>
                  <t2b.test.runner.class.args>
                      <YOUR_TESTS_LAUNCHER_ARGUMENTS>
                  </t2b.test.runner.class.args>
              </properties>
          </profile>
          <profile>
              <id>test-2-bench</id>
              <!-- @@@ Maven central snapshots repository to get dependency artifacts snapshot releases @@@ -->
              <repositories>
                  <repository>
                      <id>oss.sonatype.org</id>
                      <url>https://s01.oss.sonatype.org/content/repositories/snapshots</url>
                      <releases>
                          <enabled>false</enabled>
                      </releases>
                      <snapshots>
                          <enabled>true</enabled>
                      </snapshots>
                  </repository>
              </repositories>
              <dependencies>
                  <!-- @@@ T2B agent app dependency @@@ -->
                  <dependency>
                      <groupId>com.gocypher.cybench</groupId>
                      <artifactId>cybench-t2b-agent</artifactId>
                      <version>1.0.8-SNAPSHOT</version>
                      <scope>test</scope>
                  </dependency>
                  <dependency>
                      <groupId>org.junit.platform</groupId>
                      <artifactId>junit-platform-console-standalone</artifactId>
                      <version>1.8.1</version>
                      <scope>test</scope>
                  </dependency>
              </dependencies>
              <properties>
                  <!-- ### Java executable to use ### -->
                  <t2b.java.home>${java.home}</t2b.java.home>
                  <t2b.java.exec>"${t2b.java.home}/bin/java"</t2b.java.exec>
                  <!-- ### Java system properties used by T2B ###-->
                  <t2b.sys.props>
                      -Dt2b.aop.cfg.path="${project.basedir}"/t2b/t2b.properties
                      -Dt2b.metadata.cfg.path="${project.basedir}"/t2b/metadata.properties
                      <!-- ### To use custom LOG4J configuration -->
                      <!-- -Dlog4j2.configurationFile=file:"${project.basedir}"/t2b/log4j2.xml-->
                  </t2b.sys.props>
                  <!--  ### Class path used to run tests: libs;classes;test-classes -->
                  <t2b.run.class.path>
                      ${t2b.compile.classpath}${path.separator}${project.build.outputDirectory}${path.separator}${project.build.testOutputDirectory}
                  </t2b.run.class.path>
                  <!-- ### Skip running unit tests as benchmarks ### -->
                  <t2b.bench.runner.skip>false</t2b.bench.runner.skip>
              </properties>
              <build>
                  <plugins>
                      <!-- @@@ Make classpath entries as properties to ease access @@@ -->
                      <plugin>
                          <groupId>org.apache.maven.plugins</groupId>
                          <artifactId>maven-dependency-plugin</artifactId>
                          <version>3.1.2</version>
                          <executions>
                              <execution>
                                  <id>get-classpath-filenames</id>
                                  <goals>
                                      <goal>properties</goal>
                                  </goals>
                              </execution>
                              <execution>
                                  <phase>generate-sources</phase>
                                  <goals>
                                      <goal>build-classpath</goal>
                                  </goals>
                                  <configuration>
                                      <outputProperty>t2b.compile.classpath</outputProperty>
                                      <pathSeparator>${path.separator}</pathSeparator>
                                  </configuration>
                              </execution>
                          </executions>
                      </plugin>
                      <plugin>
                          <groupId>org.codehaus.mojo</groupId>
                          <artifactId>exec-maven-plugin</artifactId>
                          <version>3.0.0</version>
                          <executions>
                              <!-- @@@ Run unit tests as benchmarks @@@ -->
                              <execution>
                                  <id>run-benchmarks</id>
                                  <goals>
                                      <goal>exec</goal>
                                  </goals>
                                  <!-- ### Maven phase when to run unit tests as benchmarks ### -->
                                  <phase>integration-test</phase>
                                  <configuration>
                                      <skip>${t2b.bench.runner.skip}</skip>
                                      <executable>${t2b.java.exec}</executable>
                                      <classpathScope>test</classpathScope>
                                      <commandlineArgs>
                                          ${t2b.module.prop}
                                          -javaagent:${com.gocypher.cybench:cybench-t2b-agent:jar}
                                          ${t2b.sys.props}
                                          -cp ${t2b.run.class.path}
                                          ${t2b.test.runner.class}
                                          ${t2b.test.runner.class.args}
                                      </commandlineArgs>
                                  </configuration>
                              </execution>
                          </executions>
                      </plugin>
                  </plugins>
              </build>
          </profile>
          <...>
      </profiles>
      <...>
  </project>

  Note: configurable sections are marked with comments starting <!-- ###.

  Note: since cybench-t2b-agent now is in pre-release state, you have to add maven central snapshots repo https://s01.oss.sonatype.org/content/repositories/snapshots to your project repositories list.

  Note: replace <YOUR_TESTS_LAUNCHER_ARGUMENTS> placeholder with your project unit testing framework configuration, e.g. --scan-class-path -E=junit-vintage for JUnit5.

  Note: change system properties defined path values to match your project layout.

  Note: to run CyBench Launcher runner you'll need configuration file cybench-launcher.properties. Put it somewhere in your project scope and bind it to CyBench Launcher wrapper com.gocypher.cybench.t2b.aop.benchmark.runner.CybenchRunnerWrapper class argument cfg=<YOUR_PROJECT_PATH>/cybench-launcher.properties in T2B benchmarks runner configuration file t2b.properties.

• Step 2: run your Maven script with test-2-bench profile enabled:

  mvn clean validate -f pom.xml -P test-2-bench 

  Note:

  • clean - this goal is optional, but in most cases we want to have clean build
  • validate - this goal is used to cover full build process lifecycle, since our predefined phase to run tests as benchmarks is integration-test. But you may change accordingly to adopt your project build lifecycle, just note those phases must go after test-compile phase, since we are dealing with the product of this phase.
  • -f pom.xml - you can replace it with any path and file name to match your environment

• Step 1: to run T2B agent from Gradle, edit build.gradle of your project first by adding these repository, configurations, dependnecies and task definitions:

  • Groovy

    repositories {
        mavenCentral()
        maven { url 'https://s01.oss.sonatype.org/content/repositories/snapshots' }
    }
    // ...
    // System properties to choose unit testing framework. JUnit5 is default
    ext {
        // JUnit4
        isJU4 = System.properties['unitTests'] == 'junit4'
        // TestNG
        isTNG = System.properties['unitTests'] == 'testng'
    } 
    // ...
    configurations {
        t2b
    }
    // ...
    dependencies {
        // ...
        // Needed to run JUnit5 tests
        testRuntimeOnly 'org.junit.platform:junit-platform-console-standalone:1.8.1'
        // T2B runtime dependency
        t2b 'com.gocypher.cybench:cybench-t2b-agent:1.0.8-SNAPSHOT'
    }
    // ...
    task runBenchmarksFromUnitTests(type: JavaExec, dependsOn: testClasses) {
       group = 'cybench-t2b'
       description = 'Run unit tests as JMH benchmarks'
       classpath = files(
           project.sourceSets.main.runtimeClasspath,
           project.sourceSets.test.runtimeClasspath,
           configurations.t2b
       )
    
       if (JavaVersion.current().isJava9Compatible()) {
            jvmArgs = [
                "-javaagent:\"${configurations.t2b.iterator().next()}\"",
                '--add-exports=java.base/jdk.internal.loader=ALL-UNNAMED',
                '--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED'
           ]
       } else {
            jvmArgs = [
                "-javaagent:\"${configurations.t2b.iterator().next()}\""
            ]
       }
       systemProperties = [
           't2b.aop.cfg.path'     : "$project.projectDir/t2b/t2b.properties",
           't2b.metadata.cfg.path': "$project.projectDir/t2b/metadata.properties",
           // ### To use custom LOG4J configuration
           //'log4j2.configurationFile'  : "file:$project.projectDir/t2b/log4j2.xml"
       ]
    
       if (isJU4) {
           //    ### Config for JUnit4 runner ###
           main = 'org.junit.runner.JUnitCore'
           args = [
               <YOUR_TESTS_LAUNCHER_ARGUMENTS>
           ]
       } else if (isTNG) {
           //    ### Config for TestNG runner ###
           main = 'org.testng.TestNG'
           args = [
               <YOUR_TESTS_LAUNCHER_ARGUMENTS>
           ]
       } else {
           //    ### Config for JUnit5/JUnit4 runner ###
           main = 'org.junit.platform.console.ConsoleLauncher'
           args = [
               <YOUR_TESTS_LAUNCHER_ARGUMENTS>
           ]
       }
       
      ant.mkdir(dir: "${projectDir}/config/")
      ant.propertyfile(file: "${projectDir}/config/project.properties") {
          entry(key: "PROJECT_ARTIFACT", value: project.name)
          entry(key: "PROJECT_ROOT", value: project.rootProject)
          entry(key: "PROJECT_VERSION", value: project.version)
          entry(key: "PROJECT_PARENT", value: project.parent)
          entry(key: "PROJECT_BUILD_DATE", value: new Date())
          entry(key: "PROJECT_GROUP", value: project.group)
      }
    }

  • Kotlin

    // ...
    repositories {
      mavenLocal()
      mavenCentral()
      maven {
        setUrl("https://s01.oss.sonatype.org/content/repositories/snapshots")
        mavenContent {
          snapshotsOnly()
        }
      }
    }
    // ...
    val t2b by configurations.creating {
      isCanBeResolved = true
      isCanBeConsumed = false
    }
    // ...
    dependencies {
      // ...
      // Needed to run JUnit5 tests
      testRuntimeOnly ("org.junit.platform:junit-platform-console-standalone:1.8.1")
      // T2B runtime dependency
      t2b ("com.gocypher.cybench:cybench-t2b-agent:1.0.8-SNAPSHOT")
    }
    // ...
    val launcher = javaToolchains.launcherFor {
      languageVersion.set(JavaLanguageVersion.of(11))
    }
    
    tasks {
      val runBenchmarksFromUnitTests by registering(JavaExec::class) {
        group = "cybench-t2b"
        description = "Run unit tests as JMH benchmarks"
        dependsOn(testClasses)
    
        if (JavaVersion.current().isJava9Compatible) {
          jvmArgs("-javaagent:\"${configurations.getByName("t2b").iterator().next()}\"")
          jvmArgs("--add-exports=java.base/jdk.internal.loader=ALL-UNNAMED")
          jvmArgs("--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED")
        } else {
          jvmArgs("-javaagent:\"${configurations.getByName("t2b").iterator().next()}\"")
        }
    
        systemProperty("t2b.aop.cfg.path", "${project.rootDir}/t2b/t2b.properties")
        systemProperty("t2b.metadata.cfg.path", "${project.rootDir}/t2b/metadata.properties")
        // ### To use custom LOG4J configuration
        //systemProperty("log4j2.configurationFile", "file:${project.rootDir}/t2b/log4j2.xml")
    
        classpath(
          sourceSets["main"].runtimeClasspath,
          sourceSets["test"].runtimeClasspath,
          configurations.getByName("t2b")
        )
    
        mainClass.set("org.junit.platform.console.ConsoleLauncher")
        args("<YOUR_TESTS_LAUNCHER_ARGUMENTS>")
        
        ant.withGroovyBuilder {
           "mkdir"("dir" to "${projectDir}/config/")
           "propertyfile"("file" to "$projectDir/config/project.properties") {
            "entry"("key" to "PROJECT_ARTIFACT", "value" to project.name)
            "entry"("key" to "PROJECT_ROOT", "value" to project.rootProject)
            "entry"("key" to "PROJECT_VERSION", "value" to project.version)
            "entry"("key" to "PROJECT_PARENT", "value" to project.parent)
            "entry"("key" to "PROJECT_GROUP", "value" to project.group)
           }
        }
      }
    }

  Note: configurations section defines custom ones to make it easier to access particular cybench dependencies.

  Note: since cybench-t2b-agent now is in pre-release state, you have to add maven central snapshots repo https://s01.oss.sonatype.org/content/repositories/snapshots to your project repositories list.

  Note: replace <YOUR_TESTS_LAUNCHER_ARGUMENTS> placeholder with your project unit testing framework configuration, e.g. --scan-class-path -E=junit-vintage for JUnit5.

  Note: change system properties defined path values to match your project layout.

  Note: to run CyBench Launcher runner you'll need configuration file cybench-launcher.properties. Put it somewhere in your project scope and bind it to CyBench Launcher wrapper com.gocypher.cybench.t2b.aop.benchmark.runner.CybenchRunnerWrapper class argument cfg=<YOUR_PROJECT_PATH>/cybench-launcher.properties in T2B benchmarks runner configuration file t2b.properties.

• Step 2: run your Gradle script:

  • To run your project unit tests as benchmarks

    gradle :runBenchmarksFromUnitTests 

CyBench T2B agent release package contains shell scripts you can use to run it. Scripts are located in bin directory of the release package.

runit.bat is for Windows, while runit.sh for *nix family OS.

Bash scripts are kind of run wizards: it will ask you for you about your environment configuration and having required variables set, it will run your project tests as benchmarks.

In case you want to set your environment variables once and only time without prompts for every run, edit shell script file by setting your environment variables values and comment out prompts.

• If test method is annotated as test using annotations of multiple unit test frameworks ( e.g. @org.junit.jupiter.api.Test and @org.junit.Test), AspectJ can apply incorrect aspect (JUnit4 while it is run using JUnit5) and test execution as benchmark will fail.

• Found a bug, need a support or want a new feature? - register a ticket at K2NIO/gocypher-cybench-junit issue tracker.
• Need more details on how to use cybench-t2b-agent? - see cybench-t2b-agent readme file for more details.