Comparison Framework - Manak-org/Manak GitHub Wiki
##Comparison Framework for Simple Module In simple module, benchmark cases are added with a default library name. Understand the library concept by checking out [Manak Hierarchical Structure](Manak Hierarchical Structure). The current results can be saved for comparison by passing '-s ' command line option to the executable. This saved file can be later loaded with '-c ' for comparison. Remember while saving, only the cases which actually ran will be saved. For example, if given '-r' options for running specific cases, only the benchmark cases satisfying the regular expression will be saved. See [Quick Guide](Quick Guide) for more information on running specific benchmarks.
###Comparison Framework Examples
#include <iostream>
#define MANAK_SIMPLE_MODULE bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT
#include <manak/manak.hpp>
MANAK_AUTO_BENCHMARK_CASE(B1)
{
MEASURE
(
for(size_t i = 0;i < 1000;i++);
)
}NOTE: 'MANAK_AUTO_MAIN' or 'MANAK_INIT' has to be defined only once in case of multi file project. Check out [Using Manak Effectively in Multi File Project](Using Manak Effectively in Multi File Project) for more information.
Save the results of this benchmark in 'bench1.txt' for comparison.
g++ -std=c++11 <name>.cpp
./a.out -s bench1.txt
Change .cpp to -
#include <iostream>
#define MANAK_SIMPLE_MODULE bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT
#include <manak/manak.hpp>
MANAK_AUTO_BENCHMARK_CASE(B1)
{
MEASURE
(
for(size_t i = 0;i < 10000;i++);
)
}Compare the new code with the older -
g++ -std=c++11 <name>.cpp
./a.out -c bench1.txt
The output of this procedure will look like Sample HTML ouput
The sample TXT output is -
######################################################################
# Manak C++ Benchmarking Library #
# Version 1.1.0 #
# Created at Jan 31 23:27:53 2015 #
######################################################################
Case Name bench
B1 +30.7(3.4)
Manak by default generates HTML output. Check out [Output Formats](Output Formats) for more information on output formats.
The value in the bracket is the old benchmarked value which was saved in 'bench1.txt'. The '+' sign in front of the entry suggests that the current benchmark is higher in magnitude. On the other hand if the current entry is lower in magnitude, output will show a negative sign. This comparison is done based on tolerance. Lack of sign implies the benchmarks are equivalent with respect to given tolerance. Here no tolerance value is given for 'B1' so default tolerance is assumed(10 microseconds). See [Setting Environment](Setting Environment) for setting default tolerance value. In comparison with tolerance 'value +- tolerance' is considered equivalent. Tolerance value can be given for a perticular benchmark case with 'MANAK_AUTO_BENCHMARK_CASE_T' or 'MANAK_BENCHMARK_CASE_T'. See Simple Module for more information on these macros.
For loading the appropriate old value into the benchmark case each benchmark case is assigned a unique name. This unique name depends on its location in the hierarchy. Hence its not necessary to maintain the exact state of the benchmarking module to load the old saved results.
For example let us change the code of .cpp to -
#include <iostream>
#define MANAK_SIMPLE_MODULE bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT
#include <manak/manak.hpp>
MANAK_AUTO_BENCHMARK_CASE(B1)
{
MEASURE
(
for(size_t i = 0;i < 10000;i++);
)
}
MANAK_AUTO_BENCHMARK_CASE(B2)
{
MEASURE
(
for(size_t i = 0;i < 100000;i++);
)
}In this case the output(TXT... HTML output will be similar) will look like -
######################################################################
# Manak C++ Benchmarking Library #
# Version 1.0.0 #
# Created at Nov 15 04:00:25 2014 #
######################################################################
Case Name base_library
B1 +33.6(3.7)
B2 337.2
Here only 'B1' is compared as only its old data was available. Here if we shift 'B1' inside any suite, its unique signature will change and it won't be compared with the old entry. Also remember the values are loaded against the associated library in a benchmark case, changing the library name will also affect the loading process.
#include <iostream>
#define MANAK_SIMPLE_MODULE bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT
#include <manak/manak.hpp>
MANAK_AUTO_SUITE(Suite1);
MANAK_AUTO_BENCHMARK_CASE(B1)
{
MEASURE
(
for(size_t i = 0;i < 10000;i++);
)
}
MANAK_AUTO_SUITE_END();
MANAK_AUTO_BENCHMARK_CASE(B2)
{
MEASURE
(
for(size_t i = 0;i < 100000;i++);
)
}The output(TXT) will be -
######################################################################
# Manak C++ Benchmarking Library #
# Version master.0.0 #
# Created at Nov 15 04:04:09 2014 #
######################################################################
Case Name base_library
B2 339.7
B1 33.6
##Comparison Framework of Normal Module In normal benchmarking module multiple libraries are compared against each other. The libraries can also be compared along time. The comparison framework of normal module is built around the comparison structure of simple module. Time based comparison structure of both modules is identical hence see [Comparison Framework of Simple Module](#Comparison Framework for Simple Module) for time based comparison. In normal module comparison framework, libraries associed with benchmark case are compared against each other.
For example -
#include <iostream>
#define MANAK_MODULE bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT
#include <manak/manak.hpp>
MANAK_AUTO_BENCHMARK_CASE(B1, lib1)
{
MEASURE
(
for(size_t i = 0;i < 1000;i++);
)
}
MANAK_AUTO_BENCHMARK_CASE(B1, lib2)
{
MEASURE
(
for(size_t i = 0;i < 10000;i++);
)
}
MANAK_AUTO_BENCHMARK_CASE(B2, lib2)
{
MEASURE
(
for(size_t i = 0;i < 1000;i++);
)
}The output of this code can be seen HERE.
The examples shown in [Comparison Framework Examples](#Comparison Framework Examples) would work equivalently on this code in time based comparison. Remember, all suites carry their own namespace.
#include <iostream>
#define MANAK_MODULE bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT
#include <manak/manak.hpp>
MANAK_AUTO_SUITE(Suite1);
MANAK_AUTO_BENCHMARK_CASE(B1, lib1)
{
MEASURE
(
for(size_t i = 0;i < 1000;i++);
)
}
MANAK_AUTO_SUITE_END();
MANAK_AUTO_BENCHMARK_CASE(B1, lib2)
{
MEASURE
(
for(size_t i = 0;i < 10000;i++);
)
}
MANAK_AUTO_BENCHMARK_CASE(B2, lib2)
{
MEASURE
(
for(size_t i = 0;i < 1000;i++);
)
}And the output HERE.