Battery execution configuration - crocs-muni/randomness-testing-toolkit GitHub Wiki

To run RTT, you have to supply a file with battery execution configuration. This configuration modifies the way in which the tests contained in the chosen battery will be executed. You can find expected structure of configuration file below.

Disclaimer: The descriptions may contain certain RTT specific terms. To avoid confusion, you can read their explanations here.

The repository contains few predefined configuration files that can be used for RTT execution without further changes. Here is description of the configurations.

Name of the file tells you how much data (at minumum) you should use with given configuration. E.g. it's okay to analyze two megabytes of data with file 1MB.json but it is not okay to use configuration in file 10MB.json and so on.

This configuration needs at least 8 388 608 000 (8 000 * 1 024 * 1 024) bytes of data. It is intended to run all batteries (except for TestU01 Big Crush, which consumes too much data) in their default settings. Therefore all tests in the batteries will be executed (almost) as if you provided the data directly into the battery (not through RTT). Execution time should not exceed one hour.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. We consider testing 1 000 of 1 000 000 bit long sequences as default.
• DIEHARDER All 76 variants of all tests will be executed. Test RGB Lagged Sum from DIEHARDER is by default executed in 33 variants (with -n [0..32]). The -n setting affects the amount data the test will process. Starting from -n 20, this would be too much for 8GB, therefore variants with -n [20..32] have accordingly limited repetitions of the test.
• TestU01 Small Crush All 10 variants of all tests will be executed.
• TestU01 Crush All 96 variants of all tests will be executed.
• TestU01 Big Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 8 388 608 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

Configuration with focus on processing as little data as posible. Will not process more than 1MB of data (1 000 000 bytes). The run will finish very quickly (seconds).

Individual battery notes:

• DIEHARDER Only 14 variants of tests will be executed. Repetitions of these variants will be largely reduced.
• NIST STS, TestU01 (Small, Big) Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 1 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 10MB (10 000 000 bytes) of data at most and will finish in seconds.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 80 streams, each 1 000 000 bits long.
• DIEHARDER 39 variants of the tests will be executed. Repetitions are reduced.
• TestU01 (Small, Big) Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 10 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 50MB (50 000 000) of data at most and will finish in few minutes.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 400 streams, each 1 000 000 bits long.
• DIEHARDER 44 variants of the tests will be executed. Repetitions are reduced.
• TestU01 Small Crush 6 variants of the tests will be executed.
• TestU01 (Big) Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 50 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 75MB (75 000 000) of data at most and will finish in few minutes.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 400 streams, each 1 000 000 bits long.
• DIEHARDER 44 variants of the tests will be executed. Repetitions are reduced.
• TestU01 Small Crush 6 variants of the tests will be executed.
• TestU01 (Big) Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 75 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 100MB (100 000 000) of data at most and will finish in about 10 minutes. More serious biases can be discovered even in this configuration.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 800 streams, each 1 000 000 bits long.
• DIEHARDER 63 variants of the tests will be executed. Repetitions are reduced.
• TestU01 Small Crush 7 variants of the tests will be executed.
• TestU01 (Big) Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 100 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 250MB (250 000 000) of data at most and will finish in about 20 minutes. More serious biases can be discovered in this configuration.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 2 000 streams, each 1 000 000 bits long.
• DIEHARDER 73 variants of the tests will be executed. Repetitions are reduced.
• TestU01 Small Crush All 10 variants of all tests will be executed.
• TestU01 (Big) Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 250 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 500MB (500 000 000) of data at most and will finish in about 30-40 minutes. Serious biases can be discovered in this configuration.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 4 000 streams, each 1 000 000 bits long.
• DIEHARDER 75 variants of the tests will be executed. Repetitions are reduced slightly, some tests are already running on their default settings.
• TestU01 Small Crush All 10 variants of all tests will be executed.
• TestU01 Crush 24 variants of the tests will be executed.
• TestU01 Big Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 500 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This configuration will process 1 000MB (1 000 000 000) of data at most and will finish in about an hour (will vary between batteries). Very serious biases can be discovered in this configuration.

Individual battery notes:

• NIST STS All 15 variants of all tests will be executed. Each test will process 8 000 streams, each 1 000 000 bits long.
• DIEHARDER All 76 variants of all tests will be executed. Repetitions are reduced slightly, some tests are already running on their default settings.
• TestU01 Small Crush All 10 variants of all tests will be executed.
• TestU01 Crush 37 variants of the tests will be executed.
• TestU01 Big Crush No tests will be executed.
• TestU01 Rabbit, Alphabit, Block Alphabit Batteries are set so that they will not process more than 1 000 000 000 bytes of data. All 26, 9, 54 variants of all tests (respectively) will be executed.

This is allowed structure of the JSON file. The tags are explained below. This is only example of the structure and the options.

//Let's assume RTT was executed with arguments 
//    "-b dieharder -f file.dat -c cfg.json"
//Tests from DIEHARDER battery will be executed
{
  "randomness-testing-toolkit": {
    //dieharder-settings tag will be used
    "dieharder-settings": {
      "defaults": {
        //Tests ID was not defined in the arguments.
        //Tests 1, 8, 9 will be executed.
        "test-ids": ["1","8-9"]
        "psamples": 100
      },
      "test-specific-settings": [
        //Test 1 doesn't have its special tag. Value of psamples
        //will fall back to default 100
        {
          //Test 8 will have value of psamples set to 50
          "test-id": 8,
          "psamples": 50
        },
        {
          //This test will be executed in two variants
          "test-id": 9,
          "psamples": 200,
          "variants": [
            {
              //This variant will have psamples set to 100
              "psamples": 100
            },
            {
              //This variant doesn't have defined psamples options.
              //Value of psamples will fall back to test, which is 200
            }
          ]
        }
      ]
    },
    //nist-sts-settings tag will be ignored
    "nist-sts-settings": {
      "defaults": {
        "test-ids": ["1-15"],    
        "stream-size": "1000000",
        "stream-count": "1000"
      },
    },
    //tu01-smallcrush-settings tag will be ignored
    "tu01-smallcrush-settings": {
      "defaults": {
        "test-ids": [1-10]
      }
    }
    //tu01-crush-settings tag will be ignored
    "tu01-crush-settings": {
      "defaults": {
        "test-ids": [1-96]
      }
    }
  }
}

These settings are common for all implemented batteries and tests. They can be present in various tags. Their possible use is listed.

• randomness-testing-toolkit
  Root tag of the configuration file.
• defaults
  Tag should be inside settings of given battery. Options inside of this tag are considered defaults. If certain test is executed and needs to have some option defined but it does not, the option falls back to the value in defaults tag inside given battery.
• test-ids
  For use in defaults tag. Sets which tests will be executed in specific battery, if no tests were specified through command line arguments. Value of this tag must be list of strings with single numbers or number ranges.
• test-specific-settings
  For use in tag with settings of specific battery. Tags inside are specific for certain tests and batteries. Options set there will be used for chosen test if that test won't have variants defined. In this tag must be list of JSON objects. Each of those objects must contain option test-id for identifying test and can contain additional options.
• test-id
  Used for identifying JSON objects inside test-specific-settings tag.
• variants
  Must be inside JSON object which is in test-specific-settings. If it is there multiple times, the test will be executed multiple times, each time with options as defined in respective variants. If some options are not defined in variant, value will at first fall back to option in test and then to defaults.
• <battery>-settings
  <battery> can be dieharder, nist-sts, tu01-smallcrush, tu01-crush, tu01-bigcrush, tu01-rabbit, tu01-alphabit, tu01-blockalphabit. This tag contains settings for specific battery.

Following settings are specific to each battery and should be used only in corresponding <battery>-settings tag.

• stream-size
  Sets bit size of single stream of data that will be used for testing.
• stream-count
  Sets number of streams of data that will be used for testing. E.g. if this value would be 100, then the test would be repeated 100 times each time with different stream of length stream-size
• block-length
  Some tests split data into blocks, you can modify bit length of the blocks. For details on which tests use block, see documentation of tests in NIST STS.

• psamples Sets how many times will be the test repeated before final result is calculated.

• arguments Allows you to pass additional arguments to the DIEHARDER binary. Format is expected as single string of argument-value pairs separated by spaces. For summary of arguments allowed by DIEHARDER, see its documentation.

    "arguments": "-n 2 -t 10000"
  

• repetitions
  Sets how many times will be the test repeated.

• bit-nb
  Usable only with rabbit and (block) alphabit batteries. Sets how many bytes can be processed by the test. The test will not process more bytes.

• bit-r
  Usable only with (block) alphabit batteries. Sets the offset in each group of 32 bits. If set to r then first r bits from each 32 bits will be ignored. Only the rest of bits in data will be tested.

• bit-s
  Usable only with (block) alphabit batteries. Sets how many bits from each 32 bit group will be processed, starting from bit-r. bit-r=0, bit-s=32 will process whole 32 bit group.

• bit-w
  Usable only with block alphabit battery. Will set reorder of bits in 32 bit groups.

• parameters
  Usable only with (small, big) crush batteries. Will set custom parameters on specific test. Each test has different parameters, you can find them in TestU01 documentation. You have to set values for all available parameters.

    //Parameters of test smars_BirthdaySpacings
    "parameters": {
      "N": "1",
      "n": "10000",
      "r": "0",
      "d": "1000000",
      "t": "2",
      "p": "1"
    }