factorio-benchmark-helper's primary use case is reproducible benchmarks for Factorio.

To achieve that goal, the two primary modes are benchmark, and create benchmark.

In order to run an already generated benchmark, the following command is supplied

./factorio-benchmark-helper --benchmark <NAME>

This command runs a benchmark of NAME that you have previously created using the create-benchmark section below. This command will fetch all dependencies needed to perform the benchmark.

To run the benchmark created in the next subsection, the following command would work:

./factorio-benchmark-helper "An example benchmark"

After running a benchmark, performance results are inserted into a sqlite3 database. This results.db database can be found in the factorio-benchmark-helper directory (~/.local/share/ usually).

This database contains 3 primary tables: collection, benchmark, verbose. A collection is the results from a single --benchmark command. For each map of the collection, a record is added to the benchmark table. Finally, the verbose table contains the raw per tick timings of each map benchmarked, per run.

Current methods for easy extraction haven't yet been built, but the script found here can be of use in the meantime: https://github.com/mulark/Factorio-benchmarking-tools/blob/master/create_sql_pivot.sh

In order to create a benchmark, you must run a command as follows

./factorio-benchmark-helper --create-benchmark <NAME> --ticks <NUM_TICKS> --runs <NUM_RUNS>

--folder <PATH_TO_SAVES_TO_BE_INCLUDED> --mods [COMMA_SEPARATED_LIST_OF_MODS_OPTIONAL]

If valid, this will produce a JSON representation of the benchmark. For example, the command:

./factorio-benchmark-helper --create-benchmark "An example benchmark" --ticks 260 --runs 100 

--folder some_folder/ --mods region-cloner,creative-world-plus

would produce the following json, if there was a map called "123abc.zip" in the ~/.factorio/saves/some_folder/ directory. (If you're using a Steam based install).

{
  "benchmark_sets": {
    "An example benchmark": {
      "save_subdirectory": "some_folder",
      "mods": [
        {
          "name": "creative-world-plus",
          "version": "0.0.10",
          "sha1": "fd936b59d2befa06808d8dc094df31512fa01c62"
        },
        {
          "name": "region-cloner",
          "version": "1.2.7",
          "sha1": "5a1394a4dd4af5bc1dac47e2a8dcefb3bb2014db"
        }
      ],
      "maps": [
        {
          "name": "123abc.zip",
          "sha256": "3dd8a656233d495546fcbd6adb59acd12e510a710cc7a0fc784301a7997a659f",
          "download_link": "https://f000.backblazeb2.com/file/mulark-maps/some_folder/123abc.zip"
        }
      ],
      "ticks": 260,
      "runs": 100
    }
}

When creating a benchmark, there are a few quality of life tidbits that can help.

When defining a list of mods you wish to add, it's possible to define a specific version for use. Simply specify the mod version with an underscore.

--mods creative-world-plus_0.0.9,region-cloner

It is possible to have factorio-benchmark-helper automatically upload the selected maps to Backblaze b2. Blackblaze provides 10GB of free storage, which makes this a convenient option.

To automatically upload to Backblaze, you must first create a public bucket. Next, for that bucket you need to create an application key. Then, add the b2-backblaze-keyID and b2-backblaze-applicationKey to the config.ini file found within the factorio-benchmark-helper directory.

A meta set is a kind of benchmark set in place to chain together multiple benchmark sets. They can be created with the following command:

./factorio-benchmark-helper --create-meta <NAME> <COMMA_SEPARATED_LIST_OF_META/BENCHMARKS>

and ran with the following

./factorio-benchmark-helper --meta <NAME>

You might want to chain together multiple benchmark sets to, for example, test the same group of maps with a different set of mods. Or to test the same maps at various tick durations.

For example, suppose you have a test with a couple variants that you want to create. This command would chain together these variants:

./factorio-benchmark-helper --create-meta test-123 variant1,variant2,variant3

Meta sets are eagerly evaluated for both benchmark and other meta sets. If we wanted to chain the meta set test-123 above with some other sets, we could:

./factorio-benchmark-helper --create-meta another-test test-123,variant5

which would pull in the members of test-123 recursively to another-test. If all our variants were benchmark sets, running the command:

./factorio-benchmark-helper --meta another-test

would result in running variants 1, 2, 3, and 5.

If you've created a benchmark of particular noteworthiness, you may want to contribute it back to the master.json file.

For that purpose, you can perform a --commit on a locally stored procedure to copy it to the master.json file. This will correctly install the JSON so it can be submitted to the repository in a pull request. In the future this will also perform some validations like checking that the download link is populated, since the maps need to be world downloadable to be considered for inclusion.