Getting Started with Maven plugin - Nastel/gocypher-cybench-java GitHub Wiki

Code benchmarks with Eclipse and Maven

  • In this quick tutorial, we will be creating a Maven project inside Eclipse. We will then use the CyBench Eclipse plugin to add the required dependencies and generate benchmark stubs. Once prepared, we will then use the CyBench Maven plugin to actually execute our benchmarks. Please note, while the Eclipse CyBench plugin is used in this tutorial, it is not strictly necessary, but is instead used to aid the set-up process (dependencies, file/folder structure, generating method stubs). By using the Maven plugin for execution, you're able to have benchmarks become a step in your project's build process, useful for CI/CD environments.

Video Tutorial

This page also has a video tutorial, which can be found here.

Prerequisites

Must download and install:

  • JDK (version 1.8 or above).

  • Eclipse (for Java, JEE developers) version 2020-09 or above.

    • This tutorial uses Eclipse version 2021-09.

  • CyBench Eclipse plugin via https://github.com/K2NIO/gocypher-cybench-eclipse/releases (must restart Eclipse after plugin installation).

  • CyBench Maven plugin via https://github.com/K2NIO/gocypher-cybench-maven/releases, also located on Maven Central repository, so manually building/installing the plugin is not necessary.

    • NOTE: As of April 2021, CyBench tools (including the Maven plugin) are available on Maven's central repository, and can be referenced in your project's pom.xml without the need to manually build and install the plugin to your local Maven repository.
    • For latest, experimental changes, you have the ability to build the plugin yourself, but we highly suggest using the stable versions released to the Maven Central repository.
    • Instructions for manually building can be found here.

Create Maven project and benchmark your code in the software build process

Open Eclipse and switch it to new empty workspace.

Create a new Maven project (without archetype selection)

  • Select menu File -> New -> Other

  • Expand Maven -> select Maven project

  • Select checkbox "Create a simple project (skip archetype selection)" in the dialog window and click button "Next".

  • Enter project attributes:

    • Group id: com.benchmark.core
    • Artifact id: demo-benchmark
    • Name: Maven Benchmark Demo
    • Other fields leave default
    • Click button "Finish"

  • Eclipse will then generate all required Maven Java project artifacts (folders, packages, files).

  • Using the project explorer, navigate to the newly created project and open the pom.xml file.

    • Depending on your environment, opening the pom.xml file will either open the file in a raw/xml editor, or in a special POM editor (with multiple tabs including Overview, Dependencies, Dependency Hierarchy, etc.) In the event of the latter scenario, simply navigate to the right-most tab pom.xml to view the raw pom file. You can also simply right-click the pom.xml and Open With -> Generic Text Editor

  • Configure project to use JAVA 1.8 (or above) by adding these lines just above project close tag (</project>):

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.0</version>
            <configuration>
                <compilerVersion>1.8</compilerVersion>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

and save the file.

  • Right click mouse button on the project "demo-benchmark", in the context menu select "Maven -> Update Project".
    • This option syncs an Eclipse project's settings to anything you've changed in the pom.xml file. In this case, the Java compiler version used in Eclipse is changed from whichever version the workspace default had, to what we specified in the pom.xml (Java 8)
  • Verify that project "demo-benchmark" is checked in the dialog window and click button "OK".
  • This will apply Java compiler settings which we added to the "pom.xml" file.

Add CyBench Nature

  • NOTE: If you're missing the ability to Add CyBench Nature and Generate Benchmark Stubs, make sure you have the CyBench Eclipse plugin installled!
  • First, right click on the project "demo-benchmark" in the "Project Explorer" and then select "Add CyBench Nature" in the context menu.
  • This action will:
    • update and refresh the project's dependencies (see "pom.xml" file, inside tags <dependencies>, there will be new entries for openJDK's jmh-core and jmh-generator-annprocess)
    • update the annotation processors
    • create a sub-folder for pre-compiled code in the "target" directory.

Create Java class which contains any implementation

  • Right click on the project "demo-benchmark" in the "Project Explorer" to select "New" -> "Class"
  • Enter package name: com.benchmarks.demo
  • Enter class name: StringUtils
  • Example of Class + Package creation
  • Create a public method which concatenates two strings
	public static String concatStrings (String s1, String s2){
		return s1.concat(s2);
	}

Generate benchmark stub class for your implementation

  • First, right click on the "StringUtils.java" file in the "Project Explorer" and select "CyBench Generate Benchmark Stubs" from the context menu.
  • Only one method, "concatStrings", should appear in the dialog. Check it off and click "OK".
    • NOTE: Methods must be visible (public) in order for benchmark stubs to be generated, private methods won't appear in the popup dialog, only public ones!
  • The benchmark stub class "StringUtilsBenchmarks" corresponding to the original class "StringUtils" will be generated in the default Maven source folder for tests "src/test/java", within the same package as the original class.
    • All CyBench generated benchmark stub classes will have the suffix "Benchmarks"
    • All CyBench generated benchmark stub methods will have the suffix "Benchmark"
    • Benchmark stub classes and methods may also contain possible annotations for your benchmark settings and metadata.

Write benchmark for your implementation

  • Open class "StringUtilsBenchmarks" located in "src/test/java" folder.
  • There will be automatically generated JMH methods (setup, setupIteration, tearDown), but also stubs for the methods you chose when generating the benchmark class
    • In this example, only one method ("concatStrings") was generated, and it ("concatStringsBenchmark") can be seen in between the "setupIteration" and "tearDown" methods.
  • Update method concatStringsBenchmark (which is blank by default) to implement the code from your original class.
    String s = StringUtils.concatStrings("Demo", "Benchmark") ;
    bh.consume(s);

Update "pom.xml" to benchmark your code during build process

  • Add CyBench Maven plugin to your project "pom.xml" file in order to hook up it into the build process, the plugin must be added inside the <build><plugins> tags. The below section was grabbed from the Maven plugin's README.
<plugin>
        <groupId>com.gocypher.cybench.launcher.plugin</groupId>
        <artifactId>gocypher-cybench-launch-maven-plugin</artifactId>
        <version>1.0.5</version>
        <executions>
            <execution>
                <phase>test</phase>
                <goals>
                    <goal>cybench</goal>
                </goals>
            </execution>
        </executions>
    <configuration>
        <forks>1</forks>
        <threads>1</threads>
        <measurementIterations>5</measurementIterations>
        <measurementTime>5</measurementTime>
        <warmUpIterations>1</warmUpIterations>
        <warmUpTime>4</warmUpTime>
        <expectedScore>1</expectedScore>
        <shouldSendReportToCyBench>false</shouldSendReportToCyBench>
        <shouldStoreReportToFileSystem>true</shouldStoreReportToFileSystem>
        <reportUploadStatus>private</reportUploadStatus>
        <reportsFolder>C:/CyBench/reports/</reportsFolder>
        <reportName>My Private Build Process Benchmark</reportName>
        <userProperties>project=My Test Project;</userProperties>
        <customBenchmarkMetadata>com.gocypher.benchmarks.client.CollectionsBenchmarks=category:Collections;</customBenchmarkMetadata>
    </configuration>
    </plugin>
  • After updating file "pom.xml", make sure to force a project update by right clicking on the project "demo-benchmark" and selecting "Maven -> Update Project..."
  • It is also beneficial to "Project -> Clean" and then "Project -> Build", if you're not building automatically.
  • The plugin itself will execute when the project is built using Maven.
    • In other words, when you run mvn clean verify, mvn clean install, or any other similar goal, the benchmarks will automatically execute at the end of the build (specifically, after unit tests are ran).
  • For the sake of simplicity, several CyBench plugin parameters have been pre-configured in the above example. Every tag under the <configuration> tag can have it's value modified to suit your project. Each valid tag, their explanation/description, and their default value can be found on the Maven plugin's README

Launch project build using Maven and benchmark the code during build process

Once the previous setup has been completed, you will now be able to execute your benchmarks directly during the build process, with use of the Maven plugin. At this point, an IDE is no longer required, and benchmarks can be ran from the command line, in your CI/CD pipeline, etc. However, for the sake of this tutorial, we will remain in Eclipse.

  • Begin with a right click on project "demo-benchmark" and select "Run As" -> "Maven Build..." in the context menu.
  • In the run configuration dialog window, for the field "Goals" use the value: clean verify.
  • Click "Run".
  • The project build process will start, keep an eye on "Console" for messages. Once the unit tests phase is finished, the CyBench plugin will launch benchmarks for your code.
  • After successful build process report will be generated and stored into folder "C:/CyBench/reports/". Please remember that this output folder can be changed, along with many other options within the <configuration> tag in your pom.xml

Open benchmark report and analyze results using CyBench Eclipse tools

  • Open "CyBench Explorer view", click Eclipse menu item _"Window"->"Show View"->"Other" -> Expand "CyBench Tools" -> select "CyBench Explorer" and click button "OK"
  • This will open the "CyBench Explorer" view, you can drag it into any convenient location.
  • In the "CyBench Explorer" view, click "Open Folder" icon which is located in the top right corner.
  • File browser dialog window opens, there navigate to folder "C:/benchmark_reports/" ,select it and click button "OK"
  • "CyBench Explorer" view will switch to file system mode and display CyBench benchmark reports found in the selected location.
  • In the "CyBench Explorer" view double click on the report record.
  • "Report details" view will display the selected report contents at the bottom of the screen.
  • Analyze report results by double clicking on the entries in "Available Benchmarks" panel and browsing tabs which contains report statistics (at the bottom fo the screen) in the details panel.

Here is what the whole thing will look like

Implementation class (file "StringUtils.java" located in the "src/main/java"):

package com.benchmarks.demo;

public class StringUtils {
	public static String concatStrings (String s1, String s2){
		return s1.concat(s2) ;
	}
}

Benchmark class (file "StringUtils.java" located in the "src/main/test"):

package com.benchmarks.demo;

import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.TearDown;
import org.openjdk.jmh.annotations.Threads;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;

@State(Scope.Benchmark)
public class StringUtilsBenchmarks {


    @Setup(Level.Trial)
    public void setUp() {
        
    }

    @Setup(Level.Iteration)
    public void setUpIteration() {
       
    }

    @Benchmark
    @BenchmarkMode(Mode.Throughput)
    @OutputTimeUnit(TimeUnit.SECONDS)
    @Fork(1)
    @Threads(1)
    @Measurement(iterations = 2, time = 5, timeUnit = TimeUnit.SECONDS)
    @Warmup(iterations = 1, time = 5, timeUnit = TimeUnit.SECONDS)
    public void concatStringsBenchmark(Blackhole bh) {
    	String s = StringUtils.concatStrings("Demo", "Benchmark") ;
        bh.consume(s);
    }

    @TearDown(Level.Trial)
    public void cleanUp() {
       
    }

    @TearDown(Level.Iteration)
    public void cleanUpIteration() {
        
    }

}

Maven Project object model (file "pom.xml" locate in the project root folder):

<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.benchmark.core</groupId>
  <artifactId>demo-benchmark</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>Maven Benchmark Demo</name>
  <dependencies>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-core</artifactId>
      <version>1.26</version>
    </dependency>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-generator-annprocess</artifactId>
      <version>1.26</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.0</version>
        <configuration>
          <compilerVersion>1.8</compilerVersion>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>com.gocypher.cybench.launcher.plugin</groupId>
        <artifactId>gocypher-cybench-launch-maven-plugin</artifactId>
        <version>1.0.5</version>
        <executions>
            <execution>
                <phase>integration-test</phase>
                <goals>
                    <goal>cybench</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <reportsFolder>C:/benchmark_reports/</reportsFolder>
            <reportName>Benchmark using CyBench Maven plugin</reportName>
            <forks>1</forks>
            <threads>1</threads>
            <measurementIterations>2</measurementIterations>
            <measurementTime>2</measurementTime>
            <warmUpIterations>1</warmUpIterations>
            <warmUpTime>2</warmUpTime>
        </configuration>
    </plugin>
    </plugins>
  </build>
</project>

Additional Configuration

Plugin is configurable inside plugin configuration tags. Properties available for plugin behaviour configuration:

Property name Description Default value
forks Number of forks for benchmark execution. 1
threads Number of threads for each benchmark test. 1
measurementIterations Number of iterations for each benchmark. 5
measurementTime Time (in seconds) used for measurement execution (applies only for benchmarks where mode is throughput). 10
warmUpIterations Number of iterations for each benchmark warm-up. 3
warmUpTime Time (in seconds) used for warm-up execution (applies only for benchmarks where mode is throughput). 5
expectedScore Threshold for a total score. If report total score is lower then build fails. -1
shouldSendReportToCyBench A boolean flag which indicates if the benchmark report should be sent to CyBench. false
shouldStoreReportToFileSystem A boolean flag which indicates if the benchmark report should be saved to file system true
reportsFolder Location in a local file system where reports shall be stored. Current execution directory.
reportUploadStatus Parameter which indicates if the report is public or private. Possible values: public, private public
reportName Name of the benchmark report. CyBench Report
customBenchmarkMetadata A property which adds extra properties to the benchmarks report such as category or version or context. Configuration pattern is <fully qualified benchmark class name>=<key1>:<value1>;<key2>:<value2>. Example which adds category for class CollectionsBenchmarks: com.gocypher.benchmarks.client.CollectionsBenchmarks=category:Collections; -
userProperties User defined properties which will be added to benchmarks report section environmentSettings->userDefinedProperties as key/value strings. Configuration pattern:<key1>:<value1>;<key2>:<value2>. Example which adds a library name:library=My Library; -
skip A flag which allows to skip benchmarks execution during build process. Benchmarks execution also can be skipped via JVM system property -DskipCybench. false
benchAccessToken By providing the "bench" token that you get after creating a workspace in CyBench UI, you can send reports to your private directory, which will be visible only to the users that you authorize. -
benchQueryToken By providing the "query" token that you get after creating a workspace in CyBench UI, you can run and send automated comparisons within your project to your private directory, which will be visible only to the users that you authorize. -
email Email property is used to identify report sender while sending reports to both private and public repositories -
shouldFailBuildOnReportDeliveryFailure A flag which triggers build failure if the benchmark report was configured to be sent to CyBench but its delivery failed. false

Adding a Custom Configuration for Automated Performance Regression Testing

Along with the configurations you can specify for the CyBench launcher, you can add more properties under the <configuration> xml tag.

NOTE In order to run automated comparisons, you must add the benchQueryToken to the configuration.

Property name Description Options
automationScope Choose between comparing within current version, or between previous versions. When using BETWEEN, a specific version must be specified with the property automationCompareVersion. WITHIN or BETWEEN
automationCompareVersion Used for BETWEEN version comparisons. Any project version you have previously tested
automationNumLatestReports How many reports do you want to compare against? 1 will compare this report against the most recent report in the version you are comparing against. # > 1 will compare this report against the average of the scores of the most recent # reports in the version you are comparing against. Number >= 1
automationAnomaliesAllowed How many anomalies do you want to allow? If the number of benchmark anomalies surpasses your specified number, CyBench benchmark runner will fail... triggering your CI/CD pipeline to halt. Number >= 0
automationMethod Decide which method of comparison to use. DELTA will compare difference in score, and requires an additional property, automationThreshold. SD will do comparisons regarding standard deviation. SD requires an additional property as well, automationDeviationsAllowed. DELTA or SD
automationThreshold Only used with the DELTA method. GREATER will compare raw scores, PERCENT_CHANGE is used to measure the percent change of the score in comparison to previous scores. PERCENT_CHANGE requires an additional property: automationPercentChangeAllowed. GREATER or PERCENT_CHANGE
automationPercentChangeAllowed This argument is used when running assertions, makes sure your new score is within X percent of the previous scores you're comparing to. Any Double value.
automationDeviationsAllowed Used with assertions to check that the new score is within the given amount of deviations from the mean. (mean being calculated from the scores being compared to). Any Double value.

Example of CyBench Maven Plugin Configuration including Automated Performance Regression Configuration

<plugin>
    <groupId>com.gocypher.cybench.launcher.plugin</groupId>
    <artifactId>cybench-launcher-maven-plugin</artifactId>
    <version>1.0.5</version>
    <executions>
        <execution>
            <phase>test</phase>
            <goals>
                <goal>cybench</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <forks>1</forks>
        <threads>1</threads>
        <measurementIterations>5</measurementIterations>
        <measurementTime>5</measurementTime>
        <warmUpIterations>1</warmUpIterations>
        <warmUpTime>5</warmUpTime>
        <shouldSendReportToCyBench>false</shouldSendReportToCyBench>
        <reportsFolder>./reports/</reportsFolder>
        <reportName>My Report</reportName>
        <userProperties>library=My Library;</userProperties>
        <customBenchmarkMetadata>com.gocypher.benchmarks.client.CollectionsBenchmarks=category:Collections;</customBenchmarkMetadata>
       
        <automationScope>BETWEEN</automationScope>
        <automationCompareVersion>2.0</automationCompareVersion>
        <automationNumLatestReports>1</automationNumLatestReports>
        <automationAnomaliesAllowed>1</automationAnomaliesAllowed>
        <automationMethod>DETLA</automationMethod>
        <automationThreshold>PERCENT_CHANGE</automationThreshold>
        <automationPercentChangeAllowed>10</automationPercentChangeAllowed>
    </configuration>
</plugin>

(c) 2021-2022 CyBench. All rights reserved.

⚠️ **GitHub.com Fallback** ⚠️