Date of Failure: 2026-02-18 22:26–22:27 CST
Date of Analysis: 2026-02-19
Analyst: EIB MCP-RAG Automated Analysis
Source Log: emcbot/enkfgdas_fcst_mem001.log

The nightly CI/CD run on Hercules failed in the enkfgdas ensemble forecast (member 001) due to a MOM6 NaN numerical instability. The ocean model's mpp_reproducing_sum detected NaN values in a parallel summation field and issued a FATAL abort, killing all 80 MPI ranks. This is an instance of the known transient instability on Hercules tracked in issue #4348.

MOM6 NaN numerical instability — the ocean model detected NaN (Not a Number) values in a summation field, triggering a fatal abort:

FATAL from PE 2: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability

This originated from mpp_reproducing_sum in MOM6's FMS infrastructure. The function performs bit-reproducible parallel summation and includes NaN guards — when it encounters NaN values in the input array, it fatally aborts rather than propagating corrupted math.

Line 8695:  FATAL from PE 2: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability
Line 8697:  srun: error: hercules-05-08: tasks 0-79: Killed
Line 8698:  srun: Terminating StepId=7968733.0
Line 8700:  err=137
Line 8706:  -- FATAL ERROR: The forecast failed to run to completion! RETURN CODE 137
Line 8707:  -- ABNORMAL EXIT at Wed Feb 18 22:27:44 CST 2026 on hercules-05-08
Line 8854:  [2026-02-18T22:27:45.103] error: *** JOB 7968733 ON hercules-05-08 CANCELLED AT 2026-02-18T22:27:45 DUE to SIGNAL Terminated ***

HOMEgfs:  /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/nightly_0_3b291f69_9296/global-workflow
EXPDIR:   .../RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_3b291f69-9296
ROTDIR:   .../RUNTESTS/COMROOT/C48mx500_hybAOWCDA_3b291f69-9296
DATA:     /work2/noaa/global/role-global/stmp/HERCULES/RUNDIRS/C48mx500_hybAOWCDA_3b291f69-9296/enkfgdas.2021032500/enkfgdasefcs001.2021032500/fcst.3283753

The forecast job sourced the following configuration chain (all loaded successfully):

1. config.base — Machine: HERCULES, PSLOT: C48mx500_hybAOWCDA_3b291f69-9296
2. config.com — COM path templates
3. config.fcst — Forecast settings, FHMAX_HF=0, FHOUT_HF=0
4. config.ufs — UFS coupling settings (S2S: FV3+MOM6+CICE6)
5. config.ocn — Ocean: MESH_OCN=mesh.mx500.nc, ODA_INCUPD=True
6. config.ice — Ice: min_seaice=1.0e-6
7. config.efcs — EnKF forecast: CASE=C48
8. config.nsst — NSST: NST_MODEL=2
9. config.resources — Resources: walltime=00:20:00
10. config.resources.HERCULES — Hercules-specific resource overrides
11. HERCULES.env — Environment: OMP_STACKSIZE=512M, FI_MLX_INJECT_LIMIT=0

These warnings are not causal to the failure.

This failure is an exact match for the open tracking issue:

• Filed by: @DavidHuber-NOAA on 2025-12-17
• Status: Open (reopened)
• Assignee: @DavidHuber-NOAA

From the issue description:

"The nightly runs are periodically reporting NaNs when running ensemble forecast jobs. Last night's (12/17/2025) nightly showed the failure on the first full cycle of the C96C48mx500_S2SW_cyc_gfs test and the night before on the C96C48_hybatmDA test. Since the pattern seems to be the ensemble forecasts in different jobs, it may suggest an issue with the eupd job."

This confirms the issue is still active as of February 2026.

1. EnKF update (eupd) introduced bad increments — the issue author suspects the eupd job may be the upstream source, since different ensemble members in different tests fail
2. MOM6 ocean state corruption — NaN propagation from an unstable ocean grid cell at mx500 resolution
3. Hercules-specific hardware/compiler issue — only observed on Hercules, not Hera or other platforms
4. Intel oneAPI 2024.2.1 floating-point behavior — the loaded compiler stack might handle denormals or edge cases differently
5. Memory/interconnect issue on specific nodes — hercules-05-08 may have intermittent hardware issues

1. Update issue #4348 with this data point (commit 3b291f69, Feb 18 nightly, enkfgdas_fcst_mem001 in C48mx500_hybAOWCDA)
2. Check if this nightly was retried — transient failures often pass on retry, confirming non-determinism
3. Inspect the eupd output for this cycle — look for anomalous increment values in member 001
4. Check other members — did mem002–mem030 also fail, or just mem001?
5. Cross-reference with Hera nightly — if the same commit passed on Hera, it confirms Hercules-specific behavior
6. Node health check — request MSS/admin review of hercules-05-08 for ECC memory errors or interconnect issues

 1) contrib/0.1                    40) antlr/2.7.7
 2) intel-oneapi-compilers/2024.2.1 41) gsl/2.8
 ...
39) esmf/8.8.0                     78) gw_run.hercules

This report was generated by the EIB MCP-RAG analysis system from the emcbot gist log.