CyBench allows users to set up automated performance regression testing. Assuming you have a private workspace configured on the CyBench UI, all you have to do is start benchmarking a project, sending reports to that workspace, and eventually CyBench will start running regression tests without you having to do anything else! You also have the option to configure your own tests, that will run every single time you run a benchmark and send it to CyBench.

In this case, you are integrating CyBench into your CI/CD pipeline. You can specify the degree to what you will allow a performance drop in your benchmarks until CyBench runner halts, stopping your pipeline. You will be able to prevent code deployments with potentially critical impacts to your applications performance.

• GitHub account/CyBench account
  • The CyBench website uses GitHub for authentication, and allows you to save private reports to the CyBench database. You're also granted a private workspace where your reports will be uploaded to, with the option of making reports public. Essentially, your GitHub account also becomes your CyBench account.
  • In order to setup automated performance regression testing, you must have a private workspace setup where your project's reports and comparison info will all be stored.

In order to set up email notification, sign into CyBench and then click on your username at the top right. This will bring you to the workspace admin page. At the top left, you will see a green highlighted box titled "Email for Notifications." Enter your email here, and click verify. You will get an email and will be asked to verify that email address.

Once verified, you will receive email notifications for failures in performance regression testing that you configure! Emails will point you to the details of the test that was ran and will provide bits of information regarding the failures.

CyBench comparisons are done within a project and between specific benchmark reports. These comparisons can be configured based on eight defining properties.

Comparisons work by analyzing each benchmark in a report and comparing them to their previous scores in whatever report(s) they are being compared against.

• The Scope and Compare Version fields represent where to find the reports you want to compare against. If the scope is WITHIN, then comparisons will be ran within whatever project version the recent benchmark report was testing. If the scope is BETWEEN, then a Compare Version must be specified which will inform CyBench to look for the version you specified, and compare against reports in that version.
• The Number of Latest Reports value represents the amount of latest reports your comparison will be run against.
  • Assume you choose to run a BETWEEN comparison between a Compare Version of version 1.2, and specify a Number of Latest Reports as 2. CyBench will take the 2 latest reports recorded in version 1.2 and find all the similar benchmarks between them. Assume each of these reports have two benchmarks, benchmarking a Calculator class with methods "sum" and "difference." CyBench will take the average of the "sum" benchmarks from both reports and the average of the "difference" benchmarks from both reports, and use those values in comparisons against the recent scores recorded for "sum" and "difference" methods.
• The comparison method is what distinguishes how to find benchmark anomalies. The DELTA test will check the score difference in the recent report and compare it to the previous score(s).
  • If DELTA is chosen, a threshold must be clarified. The threshold can be either GREATER which checks solely whether or not a score is greater than its counterpart, or PERCENT_CHANGE which will check whether or not the score is outside of the Percent Change Allowed that you specify. If it is outside, the benchmark is considered an anomaly.
  • If SD is chosen, CyBench will find the mean and standard deviation of the benchmark scores you are comparing against. For this reason, you need to specify at least 2 Number of Latest Reports to compare against (if there is only one score (report) to compare against, no standard deviation can be calculated). Then, CyBench will take your recent score, and find the number deviations from the average of the scores it is compared against. If it is outside of the Deviations Allowed that you specify, the benchmark is considered an anomaly.
• The Number of Allowed Anomalies is what will tell CyBench when to flag a report. If the amount of anomalies found in your recent report surpasses the Number of Allowed Anomalies that you specify, a few things will happen. Firstly, the benchmark runner will halt. What this means is that if your benchmarks are running from within a CI/CD pipeline, your pipeline will not make it past the benchmarking phase! Additionally, if your email is verified on the CyBench UI, you will get an email notification with test results and details regarding the specific anomalies that were flagged so that you can take the appropriate steps to fix your code.

Along with whatever configurations you define for your project, CyBench will always run two individual tests within your project. The first test is a WITHIN version test, and the second one is a BETWEEN versions test.

• The within version test will check your current project version, and see if there was a report already submitted within that version. If so, it will compare all the recent scores to the scores in that most recent, previous report. The comparison method is DELTA and the threshold is GREATER.
• The between versions test will check your current project version, and check to see if there have been any other versions tested. It will alphabetically find the most previous version to the currently tested version, and grab the most recent report within that version and compare the scores between the two reports. The comparison method is DELTA and the threshold is GREATER.

You have the option to place a property file directly in a folder "config" to specify your automated performance regression testing configuration. This is similar to the cybench-launcher.properties file that sits in the config folder.

NOTE In order to run automated comparisons, you must add the benchQueryToken to the launcher configuration. Your bench query token can be found on your workspace admin page within the CyBench website while logged in.

The file is called cybench-automation.properties. Below is an example of what the file might look like once filled out. The rules for the property file follow the same rules as specified above in Comparisons Explained.

CyBench allows you to build a configuration for your project directly in the UI. To do this, navigate to your private workspace and to your project summary page. Both of these can be accessed from the CyBench header on the website. Below is an example of the project summary page for a project titled "NastelTest" hosted in a private workspace "testWS."

At the top right, there is a button titled "Automated Comparison." Click this to access the configuration UI. Below is an example of a filled out configuration. Make sure to click "Save Configuration" in order to properly save your performance regression configuration! Once saved, this configuration will run with every single benchmark sent to the project "NastelTest."

NOTE If you setup a configuration using any of the other plugins on top of the configuration you set up in the UI, both configurations will be run. Therefore, if one configuration has an anomaliesAllowed marked at 2 and the other marked at 4, all CyBench benchmark runs will fail if it finds more than 2 anomalies.

In order to add a automated performance regression testing configuration to the CyBench Maven Plugin, all you have to do is add in properties right next to where you configure the launcher. The launcher configuration is captured in the configuration tag in the root:

<build>
  <plugins>
     <plugin>
        <configuration>
           <!-- configuration here -->
        </configuration>
     </plugin>
  </plugins>
</build>

Here is an example of a Maven CyBench Plugin configuration including both launcher configurations as well as performance regression configurations.

<build>
        <plugins>
            <plugin>
                <groupId>com.gocypher.cybench.launcher.plugin</groupId>
                <artifactId>cybench-launcher-maven-plugin</artifactId>
                <version>1.1-SNAPSHOT</version>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals>
                            <goal>cybench</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <forks>1</forks>
                    <threads>1</threads>
                    <measurementIterations>3</measurementIterations>
                    <measurementTime>5</measurementTime>
                    <warmUpIterations>1</warmUpIterations>
                    <warmUpTime>5</warmUpTime>
                    <shouldSendReportToCyBench>true</shouldSendReportToCyBench>
                    <reportsFolder>./reports/</reportsFolder>
                    <reportName>My Report</reportName>
                    <benchAccessToken>ws_fccof2xp-x2f6-471c-c99d-6c3f20fds964_bench</benchAccessToken>
                    <benchQueryToken>ws_0pecmskc-pe02-qpc2-mfoP-ea8a9d9n7598_query</benchQueryToken>
                    <email>[email protected]</email>

                    <automationScope>within</automationScope>
                    <automationNumLatestReports>1</automationNumLatestReports>
                    <automationAnomaliesAllowed>1</automationAnomaliesAllowed>
                    <automationMethod>delta</automationMethod>
                    <automationThreshold>greater</automationThreshold>
                </configuration>
            </plugin>
        </plugins>
    </build>

NOTE In order to run automated comparisons, you must add the benchQueryToken to the launcher configuration. Your bench query token can be found on your workspace admin page within the CyBench website while logged in.

The property keys are listed below:

To configure automated performance regression testing using the CyBench Gradle plugin, all you have to do is add a configuration section to your build.gradle file titled cybenchAutomation. An example of what this might look like is below:

cybenchAutomation {
    scope = "BETWEEN"
    compareVersion = "2.0"
    numLatestReports = 3
    anomaliesAllowed = 5
    method = "SD"
    deviationsAllowed = 2
}

Here is what a full build.gradle file may look like with the plugin and everything it requires to be there:

/* At the top of the build.gradle file */
buildscript {
    repositories {
        maven {
            url "https://repo1.maven.org/maven2"
        }
    }
    dependencies {
        classpath "com.gocypher.cybench.launcher.plugin:cybench-launcher-gradle-plugin:1.0.4"
    }
}

plugins {
    id 'java'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    testImplementation 'junit:junit:4.13.2'
    implementation 'com.gocypher.cybench.client:gocypher-cybench-annotations:1.3.5'
    annotationProcessor 'com.gocypher.cybench.client:gocypher-cybench-annotations:1.3.5'
    // if benchmarks are in test directory
    testAnnotationProcessor 'com.gocypher.cybench.client:gocypher-cybench-annotations:1.3.5'

    implementation 'org.openjdk.jmh:jmh-core:1.34'
    annotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.34'
    // if benchmarks are in test directory
    testAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.34'
}

/* below the dependencies tag inside the build.gradle */
apply plugin: 'cybench-launcher-gradle-plugin'

/* Configuration properties that could be provided to the plugin */
cybenchJMH {
    forks = 1
    threads = 1
    measurementIterations = 5
    measurementSeconds = 5
    warmUpIterations = 1
    warmUpSeconds = 5
    shouldSendReportToCyBench = true
    benchAccessToken = "ws_fccof2xp-x2f6-471c-c99d-6c3f20fds964_bench"
    benchQueryToken = "ws_0pecmskc-pe02-qpc2-mfoP-ea8a9d9n7598_query"
    email = "[email protected]"
    reportsFolder = "./reports/"
    reportName = "My Report"
    userProperties = "library=My Library;"
}

cybenchAutomation {
    scope = "BETWEEN"
    compareVersion = "2.0"
    numLatestReports = 1
    anomaliesAllowed = 1
    method = "SD"
    deviationsAllowed = 2
}

ant.mkdir(dir: "${projectDir}/config/")
ant.propertyfile(file: "${projectDir}/config/project.properties") {
    entry(key: "PROJECT_ARTIFACT", value: project.name)
    entry(key: "PROJECT_ROOT", value: project.rootProject)
    entry(key: "PROJECT_VERSION", value: project.version)
    entry(key: "PROJECT_PARENT", value: project.parent)
    entry(key: "PROJECT_BUILD_DATE", value: new Date())
    entry(key: "PROJECT_GROUP", value: project.group)
}

NOTE In order to run automated comparisons, you must add the benchQueryToken to the launcher configuration. Your bench query token can be found on your workspace admin page within the CyBench website while logged in.

The property keys are listed below:

When using the Eclipse (or Mars) plugin (v0.3+) to run benchmarks, you also have the option to configure an automatic comparison to be made on the CyBench UI after the benchmark run. Configuring the automatic comparison for Eclipse is done within the Run Configuration, i.e. the same location you set other important data for CyBench (such as access/query tokens, additional JVM settings, etc.).

The Automatic Comparison tab (within your run configuration, adjacent to the Execution Settings and Common tabs) contains a nearly identical layout of configurable options that are found on the CyBench UI.

NOTE Setting an automatic comparison on the UI will not impact the auto comparison configured and requested by the Eclipse plugin, that is, they are independent of each other, and both will run if set.

Below the configurable settings exists brief, yet informative notes for each field. Certain (irrelevant) fields will be disabled depending on your selections (e.g., selecting the SD method will disable the Threshold and Percent Change fields as they are not required when making Standard Deviation comparisons, on the other hand, the # of Deviations Allowed field will enable, and is required).

A sample configuration is show below. In this example, the automatic comparison has been configured to check the last two reports within the same project version. It will compare the average scores of the last two reports to the score of the impending run. Since DELTA and PERCENT_CHANGE are selected, with a Percent Change Allowed value of 12.25, there is a leeway of 12.25% change in score (in either direction), any score beyond this value will result in an anomaly. Since # of Anomalies Allowed has been set to 0, any benchmark scores that go beyond the 12.25% change will fail a CI/CD pipeline, if applicable. Note that the report, scores, and comparisons will still be made and saved in this event, and if you've verified your e-mail on the CyBench UI, you will receive an e-mail detailing the comparison and anomalies.

NOTE Regarding 'quick launching' CyBench (Run As -> Run on CyBench, as opposed to creating a Run Configuration), no automatic comparison is made, even if you have one configured in Eclipse. However, if you have an automatic comparison configured on the CyBench UI, that comparison will be made automatically, even if you 'quick launch' CyBench.

NOTE In order to run automated comparisons, you must add the benchQueryToken to the launcher configuration. The configuration box can be found in the CyBench Configuration tab. Your bench query token can be found on your workspace admin page within the CyBench website while logged in.

• Lowest: 2018.1.x
• Highest: 2020.3.x

The CyBench IntelliJ plugin allows you to easily add a performance regression configuration to your runs. In order to do so, you must first integrate the IntelliJ plugin into your project following the steps on the ReadME. The plugin allows you to create run configurations for CyBench. The configuration includes launcher configurations that tell CyBench how to run the benchmarks and whether or not to send them to CyBench, and also the performance regression configurations.

In order to access this, first familiarize yourself with how to create a CyBench Run Configuration if you aren't already familiarized (info on ReadME). Go to "Run/Debug Configurations" in IntelliJ in order to access the UI where you can easily enter your configurations. The plugin also allows you to check a box to signify whether or not you wish any performance regression tests to be run. Below is an example of what a full configuration may look like.

NOTE In order to run automated comparisons, you must add the benchQueryToken to the launcher configuration. Your bench query token can be found on your workspace admin page within the CyBench website while logged in.