Getting started with CyBench Eclipse UI tools - Nastel/gocypher-cybench-java GitHub Wiki

Quick start with usage of CyBench Eclipse UI Tools

Video Tutorial

This tutorial also includes a video version, which can be found here.

Prerequisites

Must download and install:

Run Benchmarks with Quick Launch

For the sake of simplicity we assume that an Eclipse Maven project is created and benchmarks have been generated. See Eclipse Java project tutorial for instructions on generating CyBench stubs for your project.

Once these stubs have been generated, you'll have a new class in your project (located in src/test/java) containing the benchmarks. These benchmarks can be executed with default settings (forks, iterations, time per iteration, etc.) and whichever metadata you have annotated with the @BenchmarkMetadata tag (see CyBench Annotations for more information on annotated metadata). You can right click the newly created class to see a Run on Cybench option under Run as in the context menu. This is known as 'quick launching' CyBench, and as previously stated, default benchmark settings will be used, and only the metadata that is annotated will be parsed and displayed in the report and on CyBench.

Quick Launch for CyBench

Run Benchmarks using "Run Configuration"

For the sake of simplicity we assume that an Eclipse Maven project is created and benchmarks have been generated. See Eclipse Java project tutorial for instructions on generating CyBench stubs for your project.

You also have the ability (and are encouraged) to use Run Configurations for CyBench. This will allow you to have more control over your benchmark run.

  • Right click your benchmark class

  • Select the menu item "Run As" -> "Run Configuration...".

  • Dialog window opens. Creating a new Run Configuration

  • Select left side navigation item "CyBench" and click toolbar button "New launch configuration".

    • Can also right click "CyBench" and click context option "New configuration".

  • Panel with benchmark launch configuration opens

  • First Tab (CyBench Configuration)

    • Send Report to CyBench - If this flag is checked, the benchmark report will be uploaded to CyBench. If no Bench Access Token is provided, the benchmark report will be sent to CyBench's public repository
    • Include Hardware Properties - If this flag is checked, the benchmark report will contain the machine's specs (CPU, RAM, etc.). When sending reports to CyBench, this option is required and enabled automatically. If you are just running benchmarks locally without sending the report to CyBench, you can omit including hardware properties.
    • Run Project - Dropdown menu consisting of all projects in current workspace. Make sure to select the project you're testing.
    • Reports Folder - Should autofill once your project is selected, directs to "reports" folder in your project's root directory
    • Report Name - Edit the name of the generated report, report name is not changeable once benchmarks are ran/report is uploaded.
    • Bench Access Token - Set this to your CyBench workspace's _bench token. You can find this token on CyBench's website, under workspace admin settings once you're logged in. (Click your username when logged in on CyBench)
    • Bench Query Token - Set this to your CyBench workspace's _query token. You can find this token on CyBench's website, under workspace admin settings once you're logged in. (Click your username when logged in on CyBench) Grabbing benchmark token
    • Email Address - Set this to the e-mail you use to log-in to CyBench (GitHub e-mail). Used so uploaded reports can be associated with your user, and so you can earn rewards.
    • Benchmarks Class Available/Selected - This box will populate when you select a project (Run Project dropdown menu) that contains valid benchmark classes. Make sure to select the right one(s) or all of them.
      • Example filled out CyBench Configuration tab Filled out CyBench Configuration

  • Second Tab (Execution Settings)

    • Use CyBench Benchmark Settings - If this field is checked, CyBench will use the settings on this page/configuration. Otherwise, CyBench will use settings defined in the benchmark class per each method (i.e., will read from @Measurement(...) and @Warmup(...). If these values are also missing, CyBench will resort to default settings.
    • Forks - Defines amount of forks (physical JVMs) created during benchmark execution (placed above class or method) (Default: 1).
    • Threads - Defines amount of threads created in the JVM during benchmark execution (placed above class or method) (Default: 1).
    • Warmup iterations - Number of times benchmark is launched during warmup phase (Default: 1).
    • Warmup time - Duration in seconds of continuous calls to benchmark methods during warmup phase (Default: 10s).
    • Measurement iterations - Number of times benchmark is launched during measurement phase (Default: 5).
    • Measurement time - Duration in seconds of continuous calls to benchmark methods during measurement phase (Default: 10s).
    • Classpath arguments - Extra properties/arguments for your classpath. In most cases, this can be left blank.
    • JVM Properties - Extra properties for the JVM which will be used during benchmark process. For example -Xmx512m -Xms512m. This box will already have content in it when you create a Run Configuration (-Dlog4j.logs.root.path=...\cybenchLogs). The purpose of this is to direct CyBench logs into a logs folder in your home directory. You can change this filepath before running if you'd like logs written elsewhere. You can add additional JVM properties to this field, separated by a space (e.g. -Dlog4j.logs.root.path=C:\logs\cybenchLogs -Xmx512m -Xms512m)
      • Example filled out CyBench Execution tab Filled out CyBench Execution Settings

Explore benchmarks using "CyBench Explorer"

Open "CyBench Explorer" widget using menu "Window"->"Show View"->"Other"-> Expand "CyBench Tools" -> Select "CyBench Explorer".

Enabling CyBench Explorer in Eclipse

CyBench Explorer Widget opens either at the right side, or at the bottom of the screen. It can then be dragged to any other location.

Explorer benchmark reports stored in the workspace

  • "CyBench Explorer" scans all projects in the workspace and displays an "expand" icon near each folder/project that contains "CyBench" report(s). Reports generated via CyBench have an important syntax to their filename. The Eclipse plugin is able to read each report's filename and determine the name of the report, the overall score, and the timestamp of execution. If the benchmark report filename was generated without errors, this metadata (name, score, timestamp) will be displayed in the CyBench Explorer window.

CyBench Explorer example

  • If benchmarks are launched via CyBench Eclipse plugin, then the "CyBench Explorer" window is reloaded automatically once the benchmark process is finished.

  • You are able to manually reload either your workspace or selected file system for reports by clicking on the "Refresh" button which is located on the toolbar for "Cybench Explorer". It is denoted by the two stacked arrows pointing different ways.

CyBench Explorer Refresh

Explore benchmark reports stored outside workspace

  • Select "Open (open folder icon)" button in the toolbar of the "CyBench Explorer" widget, it is to the left of the "Refresh" button.
  • File system explorer will pop-up, expecting a folder containing CyBench reports.
  • Navigate to a folder where reports are stored and select it.
  • "Cybench Explorer" panel loads reports found in the selected folder.
  • To switch back to the workspace view, click on the drop down button on the "CyBench Explorer" toolbar (3 circles stacked, to the right of the "Refresh" button) and select either "Open Workspace" or "Open File System" option. You can swap between the two.

Analyze benchmark report using "Cybench Reports" widget

CyBench report is opened automatically after benchmark execution. Previous reports can be opened manually by double clicking on the report entry in "CyBench Explorer" widget.

"CyBench Reports" widget opens at the bottom of the screen. Each report is opened in a separate "CyBench Reports" widget, if the same report is already opened, then widget is activated (focused).

This view consists of two main sections:

  • Available benchmarks - Displays a list of available benchmarks (entry contains benchmark name and score). In a scenario where you have multiple methods being benchmarked in the same report, they will all be listed here. You can click each Available Benchmark/method to view the specific details of that test.
  • Details - Ddisplays details of selected benchmark (by default the first benchmark is selected). Details section consists of the following:
    • Summary - Displays common benchmark attributes with some brief information (report name, timestamp, total score)
    • Benchmark Details - Displays benchmark specific attributes, includes valuable metadata used by CyBench.
    • JVM Properties - Displays JVM settings and attributes under which the benchmark was launched.
    • HW Properties - Displays hardware and OS attributes under which the benchmark was launched.

To navigate between different benchmark's details, double click on different entries in the "Available Benchmarks" section.

Open report for analysis in CyBench Website

If you opted to send the report to CyBench, you have the ability to view it on CyBench's website. If you omitted a bench access token, the report will live on the public repository, available for all to see. It is highly suggested to create a private workspace, and use the bench access token associated with it, so you can store reports in a private workspace. Whether your report is publicly or privately available, you will be able to view the report in more detail, as well as make comparisons against other reports.

If you choose to send your report to a private workspace, adding your bench query token will allow you to get even more use out of CyBench. With the query token, CyBench will run automated benchmark comparisons for you to show how your new code is keeping up to standard. All of this information will then be available for viewing on your UI in your private workspace.

The attribute "Report URL" indicates if the report was stored in CyBench's repository. If the field is blank, the report wasn't uploaded automatically, but can still be uploaded manually. If not blank, it will contain the URL where your report lives. If the report was uploaded, you are able to click the mini "Google Chrome" logo in the report window's toolbar. This will direct your browser to CyBench's website, specifically to your report.

Open report on CyBench's website

Another way to open your report is:

  • Right click mouse button on the "Report URL" row.
  • Select "Copy Selected Row"
  • Open any Web Browser, paste URL into address field and hit Enter.