Benchmarking - LibertyDSNP/frequency GitHub Wiki

In order to achieve a certain block time, for example 6 seconds per block, there is a limited number of lines of code can be run during that time frame. When writing a pallet function, the developer is responsible for calculating its computational complexity, which is called weight. The process of determining that complexity, or simply put, time cost is called benchmarking.

Release Benchmarks

  • Benchmarks are run again for each Release using Option 2: "Referenced Hardware"
  • Release benchmarks are not merged back into main
  • Benchmarks should still be run on your branches as needed so that PRs highlight changes in weights

Running Benchmarks

There are 2 options to run benchmarks:

  • Locally on a developer laptop
  • Remotely on referenced hardware in the cloud

1. Locally

:exclamation: DO NOT commit weights yourself!

make benchmarks

It can be helpful to temporarily lower the number of iterations when testing benchmarks locally to decrease the time it takes to run them.

2. Referenced Hardware (GitHub Actions)

:exclamation: ATTENTION: DO NOT commit weights yourself! They will be auto-committed by CI job upon completion.

To trigger running benchmarks on referenced hardware in the cloud:

  1. Open a PR and ensure that it is up to-date with main and ready for merging.
  2. Request a core developer to run benchmarks.
  3. A core developer will need to push a new PR commit with a commit message containing [run-benchmarks <pallets|all>] suffix in order to trigger benchmarks. Any text prior to [run-benchmarks <pallets|all>] is optional and can be omitted for empty commits. For example, this command:
    • git commit -m 'Run benchmarks [run-benchmarks msa,schemas]' – commits local changes and runs benchmarks for given msa and schemas pallets once this commit is pushed to remote
    • git commit -m 'Run benchmarks [run-benchmarks all]' – commits local changes and runs benchmarks for all pallets once this commit is pushed to remote
    • git commit -m 'Run benchmarks [run-benchmarks msa,schemas]' --allow-empty – once commit is pushed to remote, runs benchmarks for msa and schemas pallets even if there are no local changes to commit
    • git commit -m '[run-benchmarks msa,schemas]' --allow-empty – same as above
    • git commit -m 'Run benchmarks [run-benchmarks all]' --allow-empty – once commit is pushed to remote, runs benchmarks for all pallets
  4. Upon "Run Benchmarks" job completion, the updated weights will be committed back to the PR branch in a new commit and a comment will be added to the PR similar to this: 
    Finished running benchmarks. Updated weights have been committed to the PR branch 
in commit <commit_sha_link>.

Resources for Understanding and Writing Benchmarks