Parametrized Benchmarking - Manak-org/Manak GitHub Wiki

With the use of advance C++11 features, Manak implements simple to use parametrized benchmarking. Parametrized benchmarking is useful when a same function is benchmarkek with many different parameters.

MANAK_AUTO_BENCHMARK_CASE(ForLoops_1000)
{
  MEASURE
  (
    for(size_t i = 0;i < 1000;i++);
  )
}

MANAK_AUTO_BENCHMARK_CASE(ForLoops_2000)
{
  MEASURE
  (
    for(size_t i = 0;i < 200;i++);
  )
}

Parametrized benchmark cases in Manak helps reduce code duplication. The above code can be written like -

void ForLoops(size_t a)
{
  MEASURE
  (
    for(size_t i = 0;i < a;i++);
  )
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, ForLoops)->AddArgs(1000)->AddArgs(2000));

The output(HTML) can be seen HERE. As you can see there are 2 sub-cases in B1. Notice that all the sub-cases share the same tolerance and iteration values.

Reducing code redundancy helps minimize errors. In the above code 'B1' is a parametrized benchmark holding 2 sets of parameter. With 'AddArgs' you can add more parameter sets to the parametrized benchmark. There can be any number of 'AddArgs' calls linked to a parametrized benchmark case. Parametrized benchmark case will run the given function on all the parameter sets. There is no restriction on type of function accepted by parametrized benchmark but number and type of parameters given to 'AddArgs' must match the given benchmark function. The only thing is that the auto template substitution won't work with functions. For templatized function the entire template signature must be provided.

template<typename T>
int fun(const T& t)
{
  Code;
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, fun<std::string>)->AddArgs("Test"));

Check out Simple Module and Normal Module for more supported parametrized benchmark macros.

##AddArgs_N

'AddArgs' adds an unamed sub-case to the benchmark case. Sub-benchmark cases are characterized by the chronological index rather than a name. Check out @ref hierarchy_sub for more information on the structure. But still it sometimes helps to associate it with a name. For example THIS output is better than THIS output. AddArgs_N can be used to add name to the sub-case.

void ForLoops(size_t a)
{
  MEASURE
  (
    for(size_t i = 0;i < a;i++);
  )
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, ForLoops)->AddArgs_N("With 1000", 1000)->AddArgs_N("With 2000", 2000));

'AddArgs_N' works as same as AddArgs but always accepts first parameter as name. Remember even though name is added to the sub-case they are accessed only by their chronological index. To understand this better see [Manak Hierarchical Structure](Manak Hierarchical Structure).

##AddCustomArgs

If number of parameter sets gets large its inconvenient to write each one down in 'AddArgs'. On top of that they may be part of the sequence which can be generated at runtime. In this case 'AddCustomArgs' is useful.

int fun(int a, int b)
{
  Code;
}

list<tuple<int, int>> GetArgs()
{
  list<tuple<int, int>> out;
  for(size_t i = 0;i < 50;i++)
    out.emplace_back(i, i+1);
  return out;
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, fun)->AddCustomArgs(GetArgs));

'AddCustomArgs' accepts a function which should return list of appropriate tuples. This custom function can have parameters which can be passed to 'AddCustomArgs' after the function name.

int fun(int a, int b)
{
  Code;
}

list<tuple<int, int>> GetArgs(int a)
{
  list<tuple<int, int>> out;
  for(size_t i = 0;i < a;i++)
    out.emplace_back(i, i+1);
  return out;
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, fun)->AddCustomArgs(GetArgs, 50));

To name sub-cases use AddCustomArgs_N.

int fun(int a, int b)
{
  Code;
}

list<tuple<std::string, int, int>> GetArgs(int a)
{
  list<tuple<int, int>> out;
  for(size_t i = 0;i < a;i++)
    out.emplace_back(<name>, i, i+1);
  return out;
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, fun)->AddCustomArgs_N(GetArgs, 50));

For more advance function passing see [Advance Function Registration](Advance Function Registration)

Parametrized benchmark cases can also be used inside a group. for example -

MANAK_AUTO_GROUP()

GINIT()
{

}

void fun(size_t a)
{
  MEASURE
  (
    for(size_t i = 0;i < 100;i++);
  )
}

MANAK_ADD_TO_GROUP(MANAK_CREATE_GROUP_BENCHMARK_WITH_TEMPLATE("InGroup", fun)->AddArgs(100));

MANAK_AUTO_GROUP_END();