Storing results - crocs-muni/randomness-testing-toolkit GitHub Wiki

Results of RTT execution are stored using interface Battery::IStorage. Each implemented interface defines certain way in which the data will be stored. Overview of currently implemented storages follows.

File Storage

For setting up locations of reports and main result table see Toolkit settings.

Stores results in report files. It will generate single report file for each RTT execution that will contain full results of the battery execution, executed tests, their settings and so on. Then it will write result of the run (proportion of passed and total tests) into file with result table. Result table may exist before execution, then the result will be added into it. Separate RTT executions in the table are distingushed by the used battery and data file name. If there is already entry for the battery and filename, the old result will be overwritten.

Report file format:

***** Randomness Testing Toolkit data stream analysis report *****
Date:    05-04-2017
File:    data.bin
Battery: Dieharder

Alpha:   0.01
Epsilon: 1e-08

Passed/Total tests: 1/1

Battery errors:

Battery warnings:


-----------------------------------------------------------
Diehard Runs Test test results:
    Result: Passed
    Test partial alpha: 0.00250943

    Variant 1:
        User settings: 
            P-sample count: 10
        ************

        Subtest 1:
            Kolmogorov-Smirnov statistic p-value: 0.78507772 Passed
            p-values: 
                0.05899849 0.14303914 0.25813353 0.26205674 0.29277325 
                0.35258302 0.56733382 0.65239775 0.72661662 0.97024989 
            ============
        ############

        Subtest 2:
            Kolmogorov-Smirnov statistic p-value: 0.98126677 Passed
            p-values: 
                0.06708958 0.13845018 0.24143043 0.30884227 0.34883845 
                0.45856887 0.50099963 0.72226328 0.79368168 0.94420838 
            ============
        ############

    ^^^^^^^^^^^^

    Variant 2:
        User settings: 
            P-sample count: 20
        ************

        Subtest 1:
            Kolmogorov-Smirnov statistic p-value: 0.94522158 Passed
            p-values: 
                0.02636030 0.05899849 0.07020394 0.11462969 0.14303914 
                0.25813353 0.26205674 0.29277325 0.35258302 0.42201984 
                0.49409172 0.50249791 0.50852031 0.56733382 0.65239775 
                0.72661662 0.87379533 0.94805962 0.95109797 0.97024989 
            ============
        ############

        Subtest 2:
            Kolmogorov-Smirnov statistic p-value: 0.97098651 Passed
            p-values: 
                0.05940416 0.06708958 0.10176118 0.13845018 0.18591525 
                0.20171335 0.24143043 0.30884227 0.34883845 0.45856887 
                0.47711566 0.50099963 0.51679868 0.57821548 0.72226328 
                0.73270941 0.79368168 0.91155189 0.94420838 0.99216956 
            ============
        ############

    ^^^^^^^^^^^^

-----------------------------------------------------------

MySQL Storage

For setting up connection to the database see Toolkit settings.

This storage will save all relevant results and messages into MySQL Database. RTT program will insert the results of a battery execution into table batteries and related subtables.

Experiment groups multiple RTT executions into one pack. Usually it is analysis of single file with multiple batteries.

Before execution of RTT, there must exist experiment in the table experiments and RTT must be provided ID of this experiment.

Table jobs is not used by RTT directly, it server to scheduling cumputational jobs on deployed servers.

This database storage can be used for deploying RTT as a service.

For inspecting all table fields please refer to the file rtt/storage/create_rtt_tables.sql.

Table hierarchy:

experiments                    //RTT executions on single file
|_ jobs                        //Computation scheduling
|_ batteries                   //Single RTT execution
   |_ battery_erros            //RTT runtime errors
   |_ battery_warnings         //RTT runtime warnings
   |_ tests                    //Single test in battery
      |_ variants              //Single variant of the test
         |_ variant_errors     //Errors during variant execution
         |_ variant_stderr     //Standard error output of the variant
         |_ variant_warnings   //Warnings during variant execution
         |_ user_settings      //User defined settings of the variant
         |_ subtests           //Subtests of the variant
            |_ test_parameters //Parameters of the subtest
            |_ statistics      //Statistics calculated by the subtest
            |_ p_values        //p-values of the subtest