legend - CICE-Consortium/Test-Results GitHub Wiki

Test Results Legend

There are two sets of tables, one shows a summary of test results, and the other shows detailed results from individual tests.

The columns in the summary table are

machine
compiler
version = model version
date = date of testing
test fail = number of tests that failed. For smoke tests, that means the run did not complete. For exact restart, it means the run did not complete or did not pass exact restart.
comp fail = number of regression comparisons that failed using --bcmp and comparing current results to prior results. There are two numbers in that column. The first number is the number of tests that outright failed with different results. The second number is the number of tests that could not be compared, either because the test failed or because the baselines did not exist. If no regression testing was done, then both of those numbers will always be zero and the result will be gray.
total = total number of tests. Clicking on this link will take you to the details page.

The summary tables color legend is as follows,

The columns in the detailed results are

Bld = build success
Run = run success
Test = did the specific test pass
Regr = did a bit-for-bit regression comparison to a prior set of test results pass
Compare = did a bit-for-bit comparison to another test pass
Timing = timing ouput, TL=timeloop, dyn=dynamics, col=column. Numbers in parens are comparison to prior test results (green is less than 20% degradation, yellow is less than 50%, red is more than 50%).
Case = case tested

The individual test results legend is as follows,

TravisCI reports results to the wiki only when tests fail. Completely passing tests are not reported.