legend - CICE-Consortium/Test-Results GitHub Wiki
Test Results Legend
There are two sets of tables, one shows a summary of test results, and the other shows detailed results from individual tests.
The columns in the summary table are
- machine
- compiler
- version = model version
- date = date of testing
- test fail = number of tests that failed. For smoke tests, that means the run did not complete. For exact restart, it means the run did not complete or did not pass exact restart.
- comp fail = number of regression comparisons that failed using --bcmp and comparing current results to prior results. There are two numbers in that column. The first number is the number of tests that outright failed with different results. The second number is the number of tests that could not be compared, either because the test failed or because the baselines did not exist. If no regression testing was done, then both of those numbers will always be zero and the result will be gray.
- total = total number of tests. Clicking on this link will take you to the details page.
The summary tables color legend is as follows,
= all tests pass
= at least 1 test fails
= more than 10% of tests fail
= more than 20% of tests fail
= unknown
The columns in the detailed results are
- Bld = build success
- Run = run success
- Test = did the specific test pass
- Regr = did a bit-for-bit regression comparison to a prior set of test results pass
- Compare = did a bit-for-bit comparison to another test pass
- Timing = timing ouput, TL=timeloop, dyn=dynamics, col=column. Numbers in parens are comparison to prior test results (green is less than 20% degradation, yellow is less than 50%, red is more than 50%).
- Case = case tested
The individual test results legend is as follows,
= pass
= fail
= other
= unknown
TravisCI reports results to the wiki only when tests fail. Completely passing tests are not reported.