tutorials: comma test run - acfr/comma GitHub Wiki
mkdir success echo "hello=\"world\"" > success/input echo "hello=\"world\"" > success/expected echo "cat" > success/test chmod +x success/test
comma-test-run
mkdir failure echo "hello=\"world\"" > failure/input echo "hello=\"moon\"" > failure/expected echo "cat" > failure/test chmod +x failure/test
comma-test-run
Create a sample directory of "tests":
echo "#!/bin/bash" > test echo "sleep 5" >> test chmod u+x test
echo "resources/robots/queen_bee = 1" > custom.limits echo "resources/robots/working_bee = 100" >> custom.limits
mkdir test.0 echo "resources/robots/queen_bee = 1" > test.0/config echo "resources/robots/working_bee = 10" >> test.0/config touch test.0/input
for num in $( seq 5 ); do mkdir test.$num echo "resources/robots/working_bee = 10" >> test.$num/config touch test.$num/input done
Run the command
time comma-test-run --max-parallel=6 --max-resources=custom.limits --debug
a few times. Note:
- there is only one queen bee; therefore, test.0 runs serially, everyone else waits.
- other tests run in parallel; the maximal number of working bees involved in 5 * 10 = 50, which is less than the limit of 100
- the total run time is either just above 10s (if test.0 happened to run first or last) or just above 15s (if some tests from 1 to 5 run before test.0, then test.0, then the remainder of non-zero tests)
- test.0 and all tests running after it display the "waiting for resources" message (turned on by the --debug) unless test.0 runs first and therefore, does not wait.
time comma-test-run --max-parallel=6 --debug
I.e., same "tests" and same input files are used but no custom limits are given on the command line. Note:
- now the run-time is just over 30s
- all tests run serially; this happens because comma-test-run does not know the limits for robots counters and effectively sets them to zero. Therefore, whenever a test makes a request for robots, the limits are exceeded and tests are serialized.
You may specify custom limits explicitly in the config files. When doing so, you must provide custom resource limits explicitly via the --max-resources option, otherwise, your tests will never run in parallel.
Decrease the limit for working bees to 30 (in the custom.limits file). Re-run the 'comma-test-run' commands above. What do you observe?