Introduction

This plugin measures the difference between two test runs A (Control or Baseline) and B (Variation) using Cohen's d effect size. It shows if there is an increase or a decrease of the average response times between the A and B versions and quantifies the magnitude of this difference by using Sawilowsky's rule of thumb.

The plugin lets you specifiy a maximum acceptable limit for the computed Cohen's d that will be used for a pass/fail test.

Finally, it presents the results of the analysis in a table that you can import into your test report.

Description of magnitudes for d

The following table describes the possible actions that you can take given the computed value of Cohen's d between the Variation and Control versions of your application:

Recommendation

When doing baseline comparisons, it is assumed that both the Control and Variation results are taken under normal operating conditions and that the results are more or less normal. As successful samplers reflect the system's true performance for valid requests, it is recommended to use the successful results files coming from the Outlier Detector plugin for these comparisons.

Usage

Via the GUI

Adding the Comparator to a Test Plan

Add the plugin to a Test Plan via the "Add > Non-Test Elements" menu and then select "Results Comparator".

Specify the Input fields

• Control File: the baseline file A.
• Variation File: the file to compare B.

Specify your pass/fail criteria

• Cohen's d Acceptable Limit: the maximum value for the test to pass (default is 1.2).

Any comparison where their Cohen's d is an increase in the response time equal or above this limit shall be marked as "failed" in the results table. By changing the default value, you can be more or less forgiving

Press the Compare button

The plugin will compare the means of the samplers in the Variation file to the ones in the Control file as per Cohen's d formula. Comparisons will only take place if there are at least two events per sampler in each file.

To help you gauge the global performance of the two runs, the plugin will also compute Cohen's d across all averages (i.e., average of averages) and give you an "OVERALL AVERAGE" score.

Save the Compare Results in a CSV file

By clicking on the "Save" button, the plugin will copy the statistical results into a csv file and place it in the same location of the Variation file with a suffix of "_CompareStats.csv" to the Variation filename which you can then import into your final test report.

Via Command line

In a script or in a Windows Powershell window, you can call the tool with the following command:

java -jar $env:JMETER_HOME\lib\cmdrunner-x.y.jar --tool ResultsCompare --ctrl-file {ControlFilename} --var-file {VariationFilename} --cohensd-alim {CohendMax}

where:

• cmdrunner-x.y.jar: version of the cmdrunner you have in your jmeter/lib directory;
• --ctrl-file: full path to your JMeter Baseline results file;
• --var-file: full path to your JMeter Variation results file;
• --cohensd-alim: maximum acceptable Cohen's d to pass;

The tool will save the results of the computation in an HTML file with a suffix of "_CompareStats.html" to the Variation filename which can be imported into CI/CD tools. It will return the number of failed comparisons in the exit code, allowing to fail the build step in the CI/CD tool if the build process of the tool uses a non zero exit code as a failed indicator.

See also