Ostrich

Ostrich is a benchmark suite developed in the Sable Lab at McGill University with the objective of studying the performance of JavaScript and WebCL for numerical code. Version of the benchmarks are available in C/C++, JavaScript, OpenCL and WebCL. Ostrich implements 12 out of 13 dwarfs, classes of numerical algorithm. Here are our benchmarks so far:

• back-prop: a machine-learning algorithm that trains the weights of connecting nodes on a layered neural network
• bfs: a breadth-first search algorithm that assigns to each node of a randomly generated graph its distance from a source node
• crc: an error-detecting code which is designed to detect errors caused by network transmission or any other accidental error
• fft: the Fast Fourier Transform (FFT) function is applied to a random data set
• hmm: a forward-backward algorithm to find the unknown parameters of a hidden Markov model
• lavamd: an algorithm to calculate particle potential and relocation due to mutual forces between particles within a large 3D space
• lud: a LU decomposition is performed on a 1024 x 1024, randomly-generated matrix
• nqueens: an algorithm to compute the number of ways to put down n queens on an n x n chess board where no queens are attacking each other
• nw: an algorithm to compute the optimal alignment of two protein sequences
• page-rank: the algorithm famously used by Google Search to measure the importance of a web site
• spmv: an algorithm to multiply a randomly-generated sparse matrix with a randomly generated vector
• srad: a diffusion method for ultrasonic and radar imaging applications based on partial differential equations

Obtaining the suite

If you have Git installed, you can clone our repository with this command:

$ git clone https://github.com/Sable/Ostrich.git

Dependencies

The Benchmark suite needs a few external tools to compile and run the benchmarks:

• Emscripten (http://kripken.github.io/emscripten-site/docs/getting_started/downloads.html) to compile the C benchmarks to asmjs
• Matjuice (https://github.com/Sable/matjuice) to compile the Matlab benchmarks to js
• Node.js
• Python/Numpy

Both should be put in a 'dependencies' folder at the root of the project.

The runner script in Python also has dependencies that need to be manually installed:

• cherrypy

If you are executing the script from a machine on which you do not have admin rights, the Python dependencies can be installed using pip:

$ pip install --user <dependencies>

Running the suite

The script dwarf_runner.py is used to automatically execute the benchmarks. You can use the --help flag to obtain a short description of its options.

The example below shows how to run the nqueens and bfs benchmarks in C and with asm.js on Firefox 5 times each:

$ python dwarf_runner.py -i 5 -b nqueens,bfs -e c,asmjs-firefox

Notes

If you run the benchmarks on Firefox, make sure that the two following settings (modifiable in about:config) are properly configured:

dom.max_script_run_time = 0
toolkit.startup.max_resumed_crashes = -1

The first setting prevents Firefox from prompting the user if the script takes too long; the second will prevent Firefox from prompting the user to start the browser in safe mode due to being killed too often by the script.