Introduction

This JMeter plugin tests the modality of samplers from a JMeter test results file and calculates their Coefficient of Variation as well. It lets you specifiy thresholds that will be used for a pass/fail test. It presents the results in a table and lets you select samplers to generate their histogram should you wish to see their distribution to confirm the findings of the tool.

Recommendation

The primary goal of modality and coefficient of variation is to assess the results under normal operating conditions. As successful samplers reflect the system's true performance for valid requests, it is recommended to use the successful results file coming from the Outlier Detector plugin for these calculations.

Coefficient of Variation

The Coefficient of Variation (CoV) will give you the variability of your test results. A low CoV indicates a consistent and predictable application (good). The following table may serve as a guideline in your decision-making process whether the amount of variation is within a reasonable range or if it might be considered excessive:

Modality

It is essential to check for the absence of multimodality in your test results as their presence will invalidate your test summary statistics; the average or mean etc. imply a central tendency which is often not the case.

The goal of the plugin is to detect if there are multimodes in your test results; it does not return the number of modes, but tells if multimodality is likely or not. To do this, it follows the mvalue-based modal test approach by Brendan Gregg. Basically, the mvalue is pretty similar to total variation of a histogram normalized by the plot height. As to the modal test, it compares the mvalue with a predefined threshold and makes a decision about multimodality. B. Gregg recommends using 2.4 as a signal that it is worth to manually investigate the distribution by visualising its histogram. In his Lowland multimodality detection post, Andrey Akinshin has been experimenting with the algorithm and suggests to use 2.8 to reduce the false-positive rate. It's hard to say which value is the best one in the general case and you may have to adjust the modality threshold to your System Under Test.

Bin sizes

When computing the mvalue, the tool will run the calculations across two bin sizes and retain the one that returns the largest mvalue. The bin sizes that are used follow:

1. Scott's normal reference rule. which works well for normally distributed data;
2. Freedman-Diaconis rule which works best in the presence of skewed data;

You might see that the tool is using Freedman-Diaconis option most of the time instead of the Scott option; you should not be concerned by this as the response times in performance testing are still likely to follow right-skewed distributions despite removing the upper outliers, hence Freedman-Diaconis remaining the best rule in this scenario.

By selecting the Rows tab, you will see which rule was used and the corresponding bin size.

Histograms

Even though Scott & Freedman-Diaconis rules are reliable, when a sampler is flagged as multimodal, it is always good to use domain knowledge and visual checks to validate the results. To perform a visual inspection, the plugin lets you generate histograms by a few clicks after which the histograms are shown in a "Graph" tab. Hopefully, this picture will help you mitigate the risk of false positives.

Usage

Via the GUI

Adding the calculator to a Test Plan

Add the plugin to a Test Plan via the "Add > Non-Test Elements" menu and select then "Modality & Coefficient of Variation".

Specify the input fields

• MValue Threshold: default value is 2.4 (you may have to increase this value if too much false positives are reported). Any samplers having their mValue equal or higher than this threshold shall be deemed to be multimodal.
• Coefficient of Variation Acceptable Limit: maximum amount of variation to pass (default is 30%).
• Filename: JMeter results file (without upper outliers and failed samplers) that you want to analyze.

If any samplers are detected as multimodal or if their CoV score is higher than the CoV limit, then they shall be marked as "failed" in the results table.

Press the Calculate button

The plugin parses the data and shows the results in a grid, together with their mValue and CoV ratings.

If a sampler is wrongly reported as multimodal, you can set a new mValue threshold slightly above the mValue of this sampler and repeat the calculation. However, it is recommended to do this only after checking the histogram of this sampler to ensure it is truly unimodal. See further below how to produce the histogram.

Save the calculation results in a CSV file

By clicking on the "Save" button, the plugin will copy the results into a csv file with a suffix of "__ModalityCoV.csv" to the input filename which you can then import into your final test report.

Producing a Histogram

In case some samplers are failed because they have been reported as multimodal, you can visualise the distribution of their response times in the "Graph" tab to check how many modes their histogram contains.

To do this, select the "Rows" tab first and select the samplers you are interested in by ticking their "Check" column. Then select the "Graph" tab: the tool will display the histograms of the selected samplers.

Below is an example of a histogram of a bimodal distribution.

Via Command line

In a script or in a Windows Powershell window, you can call the tool with the following command:

java -jar $env:JMETER_HOME\lib\cmdrunner-x.y.jar --tool ModalityCoV --input-file {filenameIn} --mvalue-thold {mValueThold} --cov-alim-pct {CoVMax}

where:

• cmdrunner-x.y.jar: version of the cmdrunner you have in your jmeter/lib directory;
• --input-file: full path to your cleansed JMeter results file;
• --mvalue-thold: your maximum mvalue for unimodality;
• --cov-alim-pct: your maximum variation to pass (e.g., 0.4 for 40%)

The tool will save the results of the computation in an HTML file with a suffix of "_ModalityCoV.html" to the input filename which can be imported into CI/CD tools. It will return the number of failed samplers in the exit code, allowing to fail the build step in the CI/CD tool if the build process of the tool uses a non zero exit code as a failed indicator.

Fixing multimodal samplers

When you have multimodality in your test results, we recommend that you investigate the cause and refactor your test script by splitting the samplers as in many modes that there are in the histogram. For instance, if you have bimodality on a Payment api, you may split this sampler into two separate samplers with their respective names like "Payment-Fast" and "Payment-Slow". In separating the samplers for each mode, your statistical results will refelct the behaviour of the System Under Test in a more reliable way.

See also

• Frequency Trails: Modes and Modality
• Lowland multimodality detection
• Histogram