C48_S2SW gfs_waveinit Orion srun layout Error Analysis - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

Failure on Orion Root Cause Analysis

C48_S2SW gfs_waveinit

Date: April 27, 2026
Case: C48_S2SW_c176171a-13777
Cycle / Job: gfs.2021032312 ยท waveinit
Platform: Orion (orion-16-11, orion-16-12)
Slurm JobId: 22929648
Build: nightly_0_c176171a_13777 (GFS CI/CD GitLab nightly)
Log: emcbot gist 19778eedaaf41e80a46a0f3e7aceec5f (gfs_waveinit.log, 1430 lines)
Analyst: EIB MCP-RAG (global-workflow-unified-mcp v3.6.2)

TL;DR

The reported fatal in the log is misleading:

FATAL ERROR: No model definition file for grid ep_10m  RETURN CODE 3

That is a downstream symptom. The real failure is that the MPMD srun launching the 8 wave_grid_moddef.sh tasks aborted before any task ran:

srun: error: JobId=22929648 failed to distribute tasks
       (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=22929648.0 failed on node orion-16-11:
       Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: error: orion-16-12: task 7: Killed

Because the launch failed, no mod_def.*.bin files were produced, and the file-existence check inside exgfs_wave_init.sh then raised RETURN CODE 3 for the first missing grid (ep_10m).

Evidence

1. The actual srun launch failure

From ush/run_mpmd.sh line 222 (after unset_strict.sh disables set -eu):

+ run_mpmd.sh[222] srun -l --export=ALL --hint=nomultithread --multi-prog \
    --output=mpmd.%j.%t.out -n 8 \
    /work2/.../waveinit.127424/mpmd_cmdfile.chunk1
srun: error: JobId=22929648 failed to distribute tasks
       (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=22929648.0 failed on node orion-16-11:
       Unable to layout tasks on given cpus
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: orion-16-12: task 7: Killed
srun: Terminating StepId=22929648.0

The Slurm comment "this should never happen" is its own assertion that the requested task layout is internally inconsistent with the node's CPU/affinity mask.

2. The downstream "fatal" in exgfs_wave_init.sh

+ exgfs_wave_init.sh[66][ -f .../gfs.t12z.mod_def.ep_10m.bin ](/TerrenceMcGuinness-NOAA/global-workflow/wiki/--f-.../gfs.t12z.mod_def.ep_10m.bin-)
+ exgfs_wave_init.sh[69] export err=3
+ exgfs_wave_init.sh[70] err_exit 'No model definition file for grid ep_10m'

    -- FATAL ERROR: No model definition file for grid ep_10m RETURN CODE 3
    -- ABNORMAL EXIT at Sun Apr 26 20:56:09 CDT 2026 on orion-16-11

3. Working directory confirms โ€” only inputs, no outputs

ww3_grid.inp.gsh_15m, ww3_grid.inp.ep_10m, ww3_grid.inp.aoc_9km,
ww3_grid.inp.wc_10m, ww3_grid.inp.uglo_100km, ww3_grid.inp.gnh_10m,
ww3_grid.inp.glo_30m, ww3_grid.inp.at_10m, uglo_100km.msh,
mpmd_script (1072 bytes), mpmd.out (78 bytes โ€” empty body)

No mod_def.*.bin files exist because the MPMD tasks never executed.

Why srun failed on Orion

Unable to layout tasks on given cpus with bind_type:threads,mask_cpu,one_thread is a Slurm/affinity layout mismatch โ€” not a workflow content error. Common Orion triggers for this exact message:

  1. Affinity / hyperthread mismatch. wavempexec includes --hint=nomultithread, which forbids placing tasks on hyperthread siblings. Combined with the cgroup mask the controller hands to the step on orion-16-11, the requested 8 single-thread tasks cannot be cleanly mapped onto allowed cores.
  2. Allocation vs. step inconsistency. Rocoto-assigned cpus-per-task/threads-per-core for the waveinit task may not match the per-step bindings srun is asking for.
  3. Transient node state. Orion has a recurring history of intermittent CPU-mask issues on individual compute nodes; the same job re-submitted to a different node usually succeeds.

Latent bugs surfaced in the same log

These are not the cause of this failure, but they should be filed:

A. run_mpmd.sh swallows the srun exit code (real bug โ€” masks failures)

Trace immediately after the failed srun:

+ run_mpmd.sh[223] source .../ush/set_strict.sh
+ run_mpmd.sh[235] err=0
+ run_mpmd.sh[236] [ 0 -ne 0 ](/TerrenceMcGuinness-NOAA/global-workflow/wiki/-0--ne-0-)

run_mpmd.sh re-enables strict mode and reports err=0 even though srun exited non-zero. There is no err=$? capture immediately after the srun invocation, so a launch failure is silently propagated as success. This is the reason the log surfaces a misleading "missing mod_def file" fatal instead of an immediate, accurate srun fatal.

Fix: capture err=$? immediately after the srun line in run_mpmd.sh, and propagate it through cat_outputs / chunk accounting.

B. JGLOBAL_WAVE_INIT line 21 โ€” invalid arithmetic

+ JGLOBAL_WAVE_INIT[21] [ 10#-1 -ge 0 ](/TerrenceMcGuinness-NOAA/global-workflow/wiki/-10#-1--ge-0-)
.../jobs/JGLOBAL_WAVE_INIT: line 21: [[: 10#:
       invalid integer constant (error token is "10#")

Some variable expanding to -1 is being prefixed with 10# (forced base-10 parse). Bash rejects 10#-1. With set -eu active just above, this should normally abort; it didn't only because [ ](/TerrenceMcGuinness-NOAA/global-workflow/wiki/-) returns non-zero on parse error and bash continued anyway.

Fix: guard the value (e.g., ${VAR#-} or explicit (( VAR >= 0 ))).

C. setpdy.sh missing seed file (cosmetic)

sed: can't read .../RUNTESTS/COMROOT/date/t12z: No such file or directory

The CI test case isn't staging date/t12z; setpdy.sh falls back to PDY env so this is harmless, but it clutters logs.

Recommendation

Action Owner Priority
Re-run the case (transient Orion layout issues are common) CI operator Immediate
If reproducible: drain orion-16-11/16-12, retry on different nodes RDHPCS / CI Immediate
Try --cpu-bind=none (or drop --hint=nomultithread) in wavempexec for wave init MPMD on Orion global-workflow dev Investigate
File issue: run_mpmd.sh not capturing srun exit code global-workflow dev High (masks real failures)
File issue: JGLOBAL_WAVE_INIT:21 10#-1 parse error global-workflow dev Medium
File issue: stage date/t12z for CI cases (or silence the sed) CI Low

Tools used

  • eib-mcp-gateway v3.6.2 โ€” mcp_health_check, get_server_info, get_knowledge_base_status, search_documentation, search_issues
  • Knowledge base at time of analysis: 85,995 docs (ChromaDB) ยท 5,174 nodes / 2.65M relationships (Neo4j)