tutorials: comma test run - acfr/comma GitHub Wiki

Table of Contents comma-test-run by example Basic usage Specifying custom resource limits Summary Bonus question

 mkdir success
 echo "hello=\"world\"" > success/input
 echo "hello=\"world\"" > success/expected
 echo "cat" > success/test
 chmod +x success/test

 comma-test-run

 mkdir failure
 echo "hello=\"world\"" > failure/input
 echo "hello=\"moon\"" > failure/expected
 echo "cat" > failure/test
 chmod +x failure/test

 comma-test-run

Create a sample directory of "tests":

 echo "#!/bin/bash" > test
 echo "sleep 5" >> test
 chmod u+x test

 echo "resources/robots/queen_bee = 1" > custom.limits
 echo "resources/robots/working_bee = 100" >> custom.limits

 mkdir test.0
 echo "resources/robots/queen_bee = 1" > test.0/config
 echo "resources/robots/working_bee = 10" >> test.0/config
 touch test.0/input

 for num in $( seq 5 ); do
   mkdir test.$num
   echo "resources/robots/working_bee = 10" >> test.$num/config
   touch test.$num/input
 done

Run the command

 time comma-test-run --max-parallel=6 --max-resources=custom.limits --debug

a few times. Note:

• there is only one queen bee; therefore, test.0 runs serially, everyone else waits.
• other tests run in parallel; the maximal number of working bees involved in 5 * 10 = 50, which is less than the limit of 100
• the total run time is either just above 10s (if test.0 happened to run first or last) or just above 15s (if some tests from 1 to 5 run before test.0, then test.0, then the remainder of non-zero tests)
• test.0 and all tests running after it display the "waiting for resources" message (turned on by the --debug) unless test.0 runs first and therefore, does not wait.

Now, run the command

 time comma-test-run --max-parallel=6 --debug

I.e., same "tests" and same input files are used but no custom limits are given on the command line. Note:

• now the run-time is just over 30s
• all tests run serially; this happens because comma-test-run does not know the limits for robots counters and effectively sets them to zero. Therefore, whenever a test makes a request for robots, the limits are exceeded and tests are serialized.

You may specify custom limits explicitly in the config files. When doing so, you must provide custom resource limits explicitly via the --max-resources option, otherwise, your tests will never run in parallel.

Decrease the limit for working bees to 30 (in the custom.limits file). Re-run the 'comma-test-run' commands above. What do you observe?