Summary

After tests are run and sent to the database, users are brought to a page displaying their results. A full suite of test results is known as a test report.

Purpose

Useful for providing and visualizing meaningful metrics and statistics for device performance. Suppose a client wants to test their application on a set-top box, the reporting page will provide a way to measure and visualize the application's performance and allow for comparison with other devices.

Requirements

The reporting page imports the libraries necessary for usage. These include Chart.js, the keyboard navigation tool, Bootstrap, and JQuery, as well as the necessary stylesheets. These libraries are compatible with the following browser versions: Firefox 2.0+, Internet Explorer 6+, Safari 3+, Opera 10.6+, and Chrome 8+. The page is served via Express.js by the Node.js server that runs Axiom.

Usage

On page load, the reporting page will automatically generate buttons and charts, as well as populate the fields necessary for displaying this data. If the user requested a comparison of two benchmarks, then this data is automatically populated as well.

Report Object Structure

Reports are passed into the report.ejs file, which is a modified .html file capable of serving information from the Node.js server. The test reports are passed in as JSON, and are then parsed into a Javascript object. See the appendices for an example.

Charts

The report object is separated into two objects, statistics and the benchmark results, and optionally a third if comparison is requested. A bar graph is used for displaying individual test's runtime speeds, a radar graph is used for displaying standard deviation, and a polar graph is used for displaying percentile. The data for these charts is populated parsed from the report object. For the bar graph, the test names are used as the x-axis labels and for the standard deviation chart, the test names are used as the labels.

Standard Deviation and Percentile

The radar graph displays the z-score of benchmark's test results following a normal distribution. The z-score is the number of standard deviations from the mean for a given data point. Given the standard deviation and average for that test across all benchmarks in the database and the value of the test result, the z-score is calculated using the formula . This is used as the value for the radar graph for each test result label.

The polar graph displays the percentile for the benchmark's results as a whole. The percentile graph indicates how well the device performs in comparison to all other devices tested. This is calculated with a database query and attached as an attribute of the benchmark before it is passed to the reporting page.

Banner

The banner displays information and icons based off of the attributes of the benchmark object, as well as the score calculated from the benchmark's results.

Icons

Icons are computed using a dictionary mapping benchmark object attributes to the Font Awesome icon html class.

"benchmark": {
    "id": "",
    "timestamp": "",
    "operatingsystem": "macOS",
    "operatingsystemversion": "1.0.0",
    "browser": "Safari",
    "browserversion": "1.0.0",
    "hardwaretype": "Desktop",
}

var os = {"macOS": "apple", "Windows": "windows", "Linux": "linux", "Android": "android", "Comcast":"copyright", "iOS":"apple"}

For example, in this benchmark, the apple logo will be displayed for the operating system icon.

Score

The benchmark score is the key to measuring a device's benchmark performance. It is the average of the percentiles for each tested framework. The score represents how well a certain device performs against all other devices ever tested.

URL

The URL for the reporting page follows the format:

https://axiom-benchmark.herokuapp.com/report?benchmark={benchmark_id}

Where {benchmark_id} is replaced with the identifier for the given benchmark.

If the user requests a second benchmark for comparison, the URL follows the format:

https://axiom-benchmark.herokuapp.com/report?benchmark={benchmark_id}&benchmark2={benchmark2_id}

Comparing Two Benchmarks

If the user requested a second benchmark for comparison, then another benchmark object will be added within the report object with the key benchmark2.

For the bar graph, if the user requested a second benchmark for comparison, then the average will be replaced with the second benchmark's results. For the standard deviation and percentile charts, the second benchmark's computed values will be overlayed.

Icons and scores for the second benchmark will also be displayed on the banner.

Navigation

Based on the number of frameworks run during the test, buttons along the sidebar will be generated per framework. If only one benchmark was reported then only one display benchmark button will be rendered. If two benchmarks are reported, then a second button will be rendered.

Appendix

Reporting JSON

{
    report: {
        "statistics": {
            "id": "statistics",
            "frameworks": {
                "Framework 1": {
                    "results": {
                        "Test Suite 1": {
                            "Test 1": {
                                "avg": 0,
                                "stddev": 0
                            },
                            "Test 2": {
                                "avg": 0,
                                "stddev": 0
                            }
                        }alt_text
                    }
                },
                "Framework 2": {
                    "results": {
                        "Test Suite 1": {
                            "Test 1": {
                                "avg": 0,
                                "stddev": 0
                            }
                        }
                    }
                }
            }
        }
    },
    "benchmark": {
        "id": "",
        "timestamp": "",
        "operatingsystem": "",
        "operatingsystemversion": "",
        "browser": "",
        "browserversion": "",
        "hardwaretype": "",
        "frameworks": {
            "Framework 1": {
                "results": {
                    "Test Suite 1": {
                        "Test 1": {
                            "result": 0,
                            "desc": ""
                        }
                    },
                    "version": "",
                    "percentile": "0"
                }
            }
        }
    }
}