How to calculate the Metrics - pouryafard75/DiffBenchmark GitHub Wiki

In order to calculate the metrics, its essential to make a universal diff output first. We call this format HumanReadableDiff.

Thus, we divide the process into two stages:

  • Generating the HumanReadableDiff (HRD) for every tool

  • Calculating the metrics based on the generated HRD

HumanReadableDiff Generation

We support two different HRD generators.

HRDGen3 which works only for 3.0 compatible tools and contains all the necessary mappings at sub-expression level + program element level

HRDGen2 which is compatible with all tools (3.0 and 2.1 compatible) but only contains mappings at statement-level + program element level

You can run MakeToolsOutput.java to generate the HRD format based on the passed configuration.

Metric Calculation

Run MetricsToCSV.java to calculate the metrics based on the generated HumanReadableDiffs and write the output into a CSV file.

Predefined Configurations

We also introduced the following pre-defined configurations to run the benchmark:

  • refOracle : Corresponds to the Refactoring Dataset with 3.0 compatible tools.

  • refOracleTwoPointOne : Corresponds to the Refactoring Dataset with all the tools.

  • defects4j : Corresponds to the Defects4J dataset with 3.0 compatible tools.

  • defects4jTwoPointOne : Corresponds to the Defects4J dataset with all the tools.

If you haven't updated the Configuration paths, please check here.