Introduction

This plugin acts as a "low-pass filter": it will parse the results of your test data and discard the samples at the top end so that when you use the "cleansed" data (i.e., without those upper outliers) for your statistical analysis, you will have a truer and more consistent picture of the application under test instead of your values being "skewed to the right" were you to use the original bulk results for your evaluation. The tool does not touch the samples at the lower end of the distribution.

The reasons as to why some data points are much slower than others in a test are usually independent from the application under test and due to environmental issues, such as ad-hoc network issues, jobs or tasks kicking off on the server under test in the middle of your test such as IIS Application Pool Recycles, Garbage Collection etc.

The plugin will find the upper bounds of the samples by itself using Tukey's fences and then remove the samplers which are bigger than the upper fence. To avoid noise and variability that may be brought by failed samplers, the upper limit is calculated only on successful samplers. This is because failed samplers often have very high or very low response times due to error handling, network timeouts, or other exceptional conditions. Including them could artificially inflate the interquartile range (IQR), leading to an excessively high upper fence and reduce sensitivity to true outliers.

The tool will save the trimmed results and the outliers in three separate files. You can then use the trimmed results files for your performance analysis; you may also review the list of outliers to check for any potential issues.

Finally, the upper bounds and the number of omitted samplers are displayed in a table which you can export should you wish to document the truncation process in your test report.

Recommendation

It is recommended to keep track of the samples that have been excluded from the results, e.g., across multiple tests, in order to determine if a pattern of concern is identified.

Usage

Via the GUI

Adding the detector to a Test Plan

Add the plugin to a Test Plan via the "Add > Non-Test Elements" menu and then select "Upper Outlier Removal".

Specify the Tukey constant k

You have the options between predefined constants and dynamic values.

• Typical constants of k are 1.5 if you want to be very severe and remove all outliers or 3 if you want to be less stringent and remove only the far-out values.
• If you are unsure which constant to use, you can let the plugin compute k dynamically by itself using Carling's definition of k. With Carling, k depends on the sample size and usually varies between 0.99 (for 2 samples) up to 2.28 (for a large set).

By default, the plugin will use Carling's value.

Specify the filename to analyse

• Filename: JMeter results file you want to parse.

Specify your pass/fail criteria

• Removal Acceptable Limit: maximum percentage of excluded outliers to pass (default is 20%). There is no set number of outliers that can be unilaterally dismissed; adjust this threshold according to the Tukey k value that has been selected as well as to your testing environment. For example, by setting k = 1.5, you are using a high detection rate which may result with more than 20% of outliers. Any samples where the percentage of samples that have been excluded exceed this criterion shall be marked as "failed" in the results table.

Press the Detect button

The plugin parses the data and shows the results of the truncation process in a grid. In this table, when the number of successful samples that is left after the truncation is less than 100, the plugin will mark them as "Small Success Group" as a precaution. A "failed" truncation means too many outliers were detected and that you should investigate the events to see if this is acceptable or not.

The plugin will create three new files in the same folder of the original file with the following suffixes:

• Upper outliers: "_outliers"
• Without upper outliers: "_clean"
• Successful samplers without upper outliers: "_success"

You can now use the "clean" or "success" files for your performance analysis instead of the initial results file. If you use a trimmed file for your performance evaluation, it is recommended to also add the truncation results in your test report to indicate this operation to the stake holders.

Save the truncation results in a CSV file

By clicking on the "Save" button, the plugin will copy the truncation results into a csv file with a suffix of "__UpperTrimStats.csv" to the input filename which you can then import into your final test report.

Via Command Line

In a script or in a Windows Powershell window, you can call the tool with the following command:

java -jar $env:JMETER_HOME\lib\cmdrunner-x.y.jar --tool TukeyOutlierDetector --input-file {filenameIn} --tukey-k {k} --rem-alim-pct {RemMaxPct}

where:

• cmdrunner-x.y.jar: version of the cmdrunner you have in your jmeter/lib directory
• --input-file: full path to your JMeter results file
• --tukey-k: Tukey k value (0 for Carling, 1.5 or 3)
• --rem-alim-pct: your maximum dismissal to pass (e.g., 0.4 for 40%)

The tool will save the cleansed results and the outliers in three separate files (see above) as well as save the results of the truncation in an HTML file with a suffix of "_UpperTrimStats.html" to the input filename which can be imported into CI/CD tools. It will return the number of failed samplers in the exit code, allowing to fail the build step in the CI/CD tool if the build process of the tool uses a non zero exit code as a failed indicator.

See also

• Lighthouse Variability