System Computing Consistency Check - calab-ntu/gpu-cluster GitHub Wiki

Steps

  1. Run burn-in test
    1. cd /work1/xuanshan/[eureka/spock]/maintenance/gamer.2026-04/bin
    2. Edit start and stop node in the file start.sh
    3. Run burn-in test sh start.sh
  2. Determine the test results
    1. Check if the job is still running by node.
    2. If the job is done, check if the job fails. tail log && ls spockXX/Data_000005 -> FAIL or not.
    3. If not fail. Check Data consistency. diff Record__Conservation ../ref/Record__Conservation. -> OK or GOOD

Node category rules

Queue Burn-in Test Result (Recent 6 runs) Crash Limit (Past 6 months)
stableq All GOOD < 2 crashes
workq All OK or GOOD < 4 crashes
unstableq ≥ 1 FAIL > 4 crashes

Log

Result definition FAIL : GAMER crashed during the test. OK : Simulation completed. (Data_000005 generated) GOOD : Record__Conservation file matches reference.

eureka

node name 2026/10 2026/09 2026/08 2026/07 2026/06
eureka01 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka02 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka03 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
eureka04 OK; OK; DOWN:1 OK; OK; DOWN:2 OK; OK OK; OK OK; OK
eureka05 OK; OK OK; OK OK; OK OK; OK; DOWN:1 OK; OK
eureka06 OK; OK OK; OK; DOWN:1 OK; OK OK; OK OK; OK
eureka07 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka08 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka09 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka10 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka11 OK; OK OK; OK; DOWN:1 OK; OK OK; OK OK; OK
eureka12 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka13 OK; DOWN:1 OK; OK; DOWN:3 OK; OK OK; OK OK; OK
eureka14 OK OK; OK OK; OK OK; OK OK; OK
eureka15 OK OK; OK OK; OK OK; OK OK; OK
eureka16 OK OK; OK OK; OK OK; OK OK; OK
eureka17 OK OK; FAIL OK;FAIL OK; OK FAIL; OK; DOWN:1
eureka18 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka19 OK; OK OK; OK OK; OK; DOWN:1 OK; OK; DOWN:1 OK; OK
eureka20 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka21 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka22 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka23 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka24 OK; OK OK; OK; DOWN:1 OK; OK OK; OK OK; OK
eureka25 OK; OK; DOWN:2 OK; OK OK; OK; DOWN:3 OK; OK OK; OK; DOWN:1
eureka26 OK; OK OK; OK; DOWN:2 OK; OK; DONW:2 OK; OK OK; OK
eureka27 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka28 OK; OK OK; OK OK; OK OK; OK
eureka29 OK; OK OK; OK OK; OK OK; OK OK; OK
eureka30 OK; OK OK; OK; DOWN:4 OK; OK; DOWN:3 OK; OK; DOWN:1 OK; OK
eureka31 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
eureka32 FAIL; FAIL (cuda arch not compatible) FAIL; FAIL FAIL; FAIL FAIL; FAIL FAIL; FAIL
eureka33 FAIL; FAIL; DOWN:3 (cuda arch not compatible) FAIL; FAIL; DOWN:4 FAIL; FAIL; DOWN:3 FAIL; FAIL; DOWN:3 FAIL; FAIL

spock

node name 2026/10 2026/09 2026/08 2026/07 2026/06
spock01 OK; GOOD GOOD; OK GOOD; OK GOOD; GOOD GOOD; GOOD
spock02 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock03 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD; DOWN:1
spock04 GOOD; GOOD GOOD; GOOD GOOD; OK GOOD; GOOD OK; OK
spock05 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock06 OK; GOOD OK; GOOD OK; OK GOOD; GOOD GOOD; GOOD
spock07 GOOD; GOOD; DOWN:1 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock08 GOOD; GOOD; DOWN:1 GOOD; GOOD GOOD; OK GOOD; GOOD; DOWN:1 GOOD; GOOD
spock09 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock10 GOOD; FAIL GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock11 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock12 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock13 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock14 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock15 GOOD; OK GOOD; GOOD GOOD; GOOD OK; GOOD GOOD; GOOD
spock16 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock17 GOOD; OK GOOD; GOOD GOOD; FAIL FAIL; GOOD GOOD; GOOD
spock18 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock19 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock20 GOOD; OK GOOD; GOOD GOOD; GOOD GOOD; GOOD; DOWN:1 GOOD; GOOD
spock21 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock22 GOOD; GOOD; DOWN:1 GOOD; GOOD GOOD; OK GOOD; GOOD GOOD; GOOD
spock23 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock24 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock25 OK; OK GOOD; GOOD GOOD; OK GOOD; GOOD OK; OK
spock26 FAIL; FAIL; DOWN:1 (cuda arch not compatible) FAIL; FAIL FAIL; FAIL FAIL; FAIL; DOWN:3 FAIL; FAIL
spock27 GOOD; GOOD; DOWN:1 GOOD; GOOD GOOD; GOOD GOOD; GOOD GOOD; GOOD
spock28 GOOD; GOOD GOOD; GOOD GOOD; OK GOOD; GOOD; DOWN:1 GOOD; GOOD