Date: May 6, 2026
Platform: Orion (orion-18-02)
Build: nightly_0_22b3cf70_14769
JobId: 22996374
Case: C96C48mx500_S2SW_cyc_gfs
Error Source: https://gist.github.com/emcbot/d5d4298bc5727c80f6d12f70a44a50a6

FATAL ERROR: No model definition file for grid ep_10m RETURN CODE 3

The reported error is a downstream symptom. The mod_def files were never generated because the srun MPMD job that creates them was killed due to a CPU binding failure. The run_mpmd.sh script then silently swallowed the srun exit code, allowing execution to continue to the file-existence check — which correctly reported the missing file but gave a misleading error message.

File: JGLOBAL_WAVE_INIT line 21
Error:

[[: 10#: invalid integer constant (error token is "10#")

The variable RUNMEM is -1 (from config.wave). The script evaluates [[ 10#${RUNMEM} -ge 0 ]] which expands to [[ 10#-1 -ge 0 ]]. The 10# prefix means "interpret in base 10" but negative numbers are not valid in bash base-N notation. This error is logged but does not abort the script.

File: exgfs_wave_init.sh line 58
Action: Calls run_mpmd.sh to generate mod_def files for 8 grids in parallel:

• ep_10m, aoc_9km, gsh_15m, uglo_100km, wc_10m, at_10m, gnh_10m, glo_30m

Each grid runs wave_grid_moddef.sh <grid_name> as a separate srun task.

File: run_mpmd.sh line 222
Command:

srun -l --export=ALL --hint=nomultithread --multi-prog --output=mpmd.%j.%t.out -n 8 mpmd_cmdfile.chunk1

Error output:

srun: error: JobId=22996374 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=22996374.0 failed on node orion-18-02: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: orion-18-03: task 7: Killed
srun: Terminating StepId=22996374.0

The job was allocated nodes but srun cannot map 8 tasks to the available CPUs with the requested binding (--hint=nomultithread). This is a Slurm task-layout failure — the node's CPU topology doesn't support the requested binding configuration for this step.

File: run_mpmd.sh line 235
Bug: After srun fails, the script sets err=0 unconditionally:

err=0          # ← line 235: unconditionally clears the error
[[ ${err} -ne 0 ]] && ...  # never triggers

The srun exit code is never captured or propagated. Only one output file exists (mpmd.22996374.7.out) and it's empty — confirming all tasks were killed before producing any output.

File: exgfs_wave_init.sh line 66
Check:

[[ -f ${COMOUT_WAVE_PREP}/gfs.t18z.mod_def.ep_10m.bin ]]

The file doesn't exist because the wave_grid_moddef.sh tasks never ran. The script reports:

FATAL ERROR: No model definition file for grid ep_10m RETURN CODE 3

Severity: High
Location: ush/run_mpmd.sh line 235
Impact: Any srun failure in MPMD mode is silently swallowed, causing downstream tasks to fail with misleading errors about missing output files.

Fix: Capture the srun exit code and propagate it:

srun ... ${chunk_file}
err=$?
[[ ${err} -ne 0 ]] && echo "FATAL: srun MPMD failed with exit code ${err}" && return ${err}

Severity: Low (non-fatal, but noisy)
Location: jobs/JGLOBAL_WAVE_INIT line 21
Impact: Logs a bash error but doesn't abort. Confuses log analysis.

Fix: Guard the base-10 conversion:

if [[ ${RUNMEM:--1} -ge 0 ]]; then
  # ... member-specific logic
fi

Severity: Medium
Location: Orion node orion-18-02 / Slurm configuration
Impact: The --hint=nomultithread flag combined with the job's CPU allocation doesn't match the node's topology for 8 tasks.

Possible causes:

• Node was allocated with fewer CPUs than expected (resource contention)
• tasks_per_node=40 in config but actual allocation is smaller
• --hint=nomultithread conflicts with the node's hyperthreading configuration
• Slurm's SelectTypeParameters or TaskPlugin configuration on Orion

1. Immediate: Fix run_mpmd.sh to capture and propagate srun exit codes
2. Short-term: Fix the 10#${RUNMEM} parse error in JGLOBAL_WAVE_INIT
3. Investigation: Check Orion Slurm configuration for orion-18-02 — why can't it layout 8 tasks with --hint=nomultithread? May need --hint=compute_bound or removal of the hint for waveinit tasks.
4. Monitoring: The zstd/1.5.6 module dependency warning (repeated 3 times) suggests a spack-stack configuration issue — c-blosc/1.21.5 and netcdf-c/4.9.2 expect zstd/1.5.6 but only zstd/1.5.2 is loaded.

Analysis performed May 6, 2026 using error log from CI nightly build 14769 on Orion.