The Benchmark Procedure Format - mulark/factorio-benchmark-helper GitHub Wiki

Benchmark procedures come in two variants:

Benchmark set

Metabenchmark set

Benchmark sets

The benchmark set format contains the following fields:

• save_subdirectory - This is the folder under which the maps for this set will reside
• mods - This is a set of mods that will be enabled when benchmarking this set
  • name - The name of the mod
  • file_name - The file name of the mod
  • version - The mod's version
  • sha1 - The sha1 hash of the mod (mod portal uses sha1)
• maps - The set of saves which will be benchmarked with all mods defined.
  • name - The name of this save
  • sha256 - The sha256 of this save
  • download_link - Where this save can be downloaded from. Must be a direct download.
• ticks - How many ticks should each run of all these maps be peformed
• runs - How many times should each map be ran.

When you run a benchmark set, the mods and maps present in this set will be sourced. Mods will be fetched from the mod portal using your service-username and service-token present in your Factorio folder. Maps will be downloaded from the download link defined with them, or copied from your saves folder if there is a file with the same name and hash.

Meta sets

Meta sets are a list of strings that will recursively populate child meta and benchmark sets for a resultant collection of benchmark sets. You might want to use a meta set if you have two groups of mods you want to benchmark with the same set of maps.

Example: Suppose you have the following procedures:

benchmark_sets: {
    "benchmark_1",
    "benchmark_2",
    "benchmark_3",
}
meta_sets: {
    "meta_1": [
        "meta_2",
        "benchmark_1",
    ],
    "meta_2": [
        "benchmark_3",
    ]
}

If you then perform a benchmark of meta_1, you will benchmark the sets "benchmark_1" and "benchmark_3". Benchmark and meta sets can be named the same thing, however it's not recommended.