Date: June 17, 2026 Failed job: JGLOBAL_ENKF_ECEN on Hercules (GitLab nightly CI) Pipeline: nightly_0_7b0cab44_31970 Slurm job: 9117380 on hercules-08-52 Source log: https://gist.github.com/emcbot/9b9c80fed952cd5eaf80f3e26a96b663 Tooling used: agentcore-mcp-rag MCP server (tenant gw)

A nightly CI run of the GFS C96C48mx500_S2SW_cyc_gfs experiment failed in the EnKF ensemble recentering step. The failure was caused by a transient Lustre stale file handle error between two consecutive MPI invocations within the same script. The compute work itself succeeded; the failure occurred in a routine bookkeeping step (prep_step) when redirecting stdout to the job's output file.

The failure was amplified by three workflow design choices that turned a transient infrastructure flake into a hard CI failure with no post-mortem evidence:

1. No retry logic on transient filesystem errors.
2. KEEPDATA=NO cleanup destroyed the working directory before the error could be reported.
3. The strict set -eu plus EXIT trap left no opportunity to capture diagnostic context.

This document explains the root cause, the cascade that made it worse, and recommends concrete best practices that would have either prevented the failure or made it self-recovering.

/apps/contrib/spack-stack/.../prod-util-2.1.1-cmrirb3/bin/prep_step: line 4: OUTPUT.850135: Stale file handle

prep_step is a NCO/EE2 production utility that clears stale Fortran unit assignments before each Fortran program execution. Line 4 is the echo $pgm statement that redirects stdout to $pgmout (which equals OUTPUT.850135). The file existed and was successfully written to about one second earlier by the first MPI run.

Stale file handle (POSIX ESTALE) on Lustre or NFS arises when the kernel-level file descriptor a process holds no longer corresponds to a valid inode on the server. On a parallel filesystem, the typical causes are:

This is a known class of transient HPC filesystem flake. The same job retried 30 seconds later usually succeeds. Hercules and other Lustre-based MSU systems are particularly prone to this under load.

The MCP-RAG search confirmed this is consistent with general Lustre flock/ESTALE behaviour documented in the Spack and global-workflow troubleshooting pages: "parallel filesystems or NFS volumes may be configured without flock support enabled" and similar caching artefacts can produce ESTALE on otherwise valid files.

prep_step followed by err_chk is the EE2-required pattern (confirmed via agentcore-mcp-rag.search_ee2_standards):

. prep_step                            # clear Fortran unit assignments
export FORT11=$FIXpmb/inputfile.tbl
mpiexec <options> $EXECmodel/$pgm >>$pgmout 2>errfile
export err=$?; err_chk

err_chk correctly aborts on any non-zero return code, but it does not distinguish between deterministic failures (model instability, FATAL ERROR in stdout, missing input file) and transient failures (ESTALE, EAGAIN, ETIMEDOUT, ENOMEM during a process burst).

A transient class of error should retry once or twice with a short back-off before being escalated to a fatal abort.

From jjob_standard_vars.sh:

+ jjob_standard_vars.sh[63]export KEEPDATA=NO

From jjob_shell_setup.sh:

+ jjob_shell_setup.sh[56]trap '/.../postamble.sh 1781666438' EXIT

When err_exit fires, the EXIT trap runs postamble.sh, which honours KEEPDATA=NO and removes the entire run directory. This races with the error reporting itself:

ls: cannot access '/.../enkfgfs.2021122018/ecen.850069': No such file or directory
cat: OUTPUT.850135: Stale file handle
/.../JGLOBAL_ENKF_ECEN: line 68: cd: /.../enkfgfs.2021122018: No such file or directory

The forensic evidence (the OUTPUT file containing the actual program logs) is wiped before any operator or CI tool can capture it. For CI nightlies in particular, this turns an investigable flake into a black box.

From set_strict.sh:

[[ YES == YES ]]
set -eu
set -o pipefail

Strict mode plus the always-runs EXIT trap means the moment any command returns non-zero, the script exits AND the cleanup runs immediately. There is no opportunity for the script to:

• Capture diagnostic context.
• Decide if the failure class is transient.
• Retry the failed step.
• Tag the run directory as .failed_${jobid} for later inspection.

The strict-mode pattern is correct for catching real bugs early, but it interacts poorly with KEEPDATA=NO cleanup and a non-discriminating err_chk.

This kind of failure shows up periodically in nightly runs and is treated as a code regression by automation. Each instance:

• Burns operator time chasing a non-existent code bug (the RUNDIR is gone, so they cannot investigate).
• Erodes trust in the CI signal: every nightly red light is potentially an infrastructure flake, not a real regression.
• Costs a full re-run of an expensive coupled-model integration when the original computation actually succeeded.

The MCP-RAG get_operational_guidance query for "stale file handle Lustre transient filesystem error retry HPC job recovery" returned no specific guidance for the global-workflow nightly pipeline, which suggests this is currently a tribal-knowledge issue rather than a documented procedure.

A new helper at ush/prep_step_with_retry.sh:

# Wrap prep_step with bounded retry for transient filesystem errors.
prep_step_retry() {
  local max_attempts=3
  local backoff_s=5
  local attempt=1
  local err_log
  err_log="$(mktemp -p "${DATA:-/tmp}" prep_step.err.XXXXXX)"

  while (( attempt <= max_attempts )); do
    if . prep_step 2> "${err_log}"; then
      rm -f "${err_log}"
      return 0
    fi

    # Classify the failure
    if grep -qE "(Stale file handle|Resource temporarily unavailable|Cannot allocate memory|Input/output error)" "${err_log}"; then
      echo "[WARN] prep_step transient error (attempt ${attempt}/${max_attempts}): $(cat "${err_log}")" >&2
      sleep "${backoff_s}"
      backoff_s=$(( backoff_s * 2 ))
      ((attempt++))
      continue
    fi

    # Deterministic failure: surface and abort
    cat "${err_log}" >&2
    rm -f "${err_log}"
    return 1
  done

  echo "[ERROR] prep_step failed after ${max_attempts} attempts" >&2
  cat "${err_log}" >&2
  rm -f "${err_log}"
  return 1
}

Adopt across ex-scripts:

# Replace
. prep_step
# With
prep_step_retry || err_exit "prep_step failed after retries"

In postamble.sh, condition the cleanup on the calling script's exit code AND a new KEEPDATA_ON_FAILURE flag:

# In jjob_shell_setup.sh
trap '_RC=$?; postamble.sh "${start_time}" "${_RC}"' EXIT

# In postamble.sh
exit_code=${2:-0}
if (( exit_code != 0 )) && [[ "${KEEPDATA_ON_FAILURE:-YES}" == "YES" ]]; then
  failed_dir="${DATA}.failed_${jobid:-$$}"
  mv "${DATA}" "${failed_dir}" || true
  echo "[INFO] Preserved failed RUNDIR for post-mortem: ${failed_dir}" >&2
  exit "${exit_code}"
fi
if [[ "${KEEPDATA}" != "YES" ]]; then
  rm -rf "${DATA}"
fi

This preserves the run directory under <original_path>.failed_<jobid> whenever the job exits non-zero, without forcing operators to set KEEPDATA=YES globally (which would also retain successful runs and bloat scratch).

In err_exit.sh, snapshot the working directory and key files before exiting:

err_exit() {
  local message="${1:-}"
  echo "FATAL ERROR: ${message}" >&2

  # Snapshot critical diagnostic files into a side directory.
  local snapshot="${DATA}/.err_exit_snapshot"
  mkdir -p "${snapshot}" 2>/dev/null || true
  cp -p "${pgmout:-OUTPUT.${pid:-unknown}}" "${snapshot}/" 2>/dev/null || true
  cp -p errfile "${snapshot}/" 2>/dev/null || true
  echo "${message}" > "${snapshot}/err_exit_message"
  date -u +%Y%m%dT%H%M%SZ > "${snapshot}/err_exit_timestamp"

  exit 1
}

Combined with Best Practice 2, this means the post-mortem directory always has a .err_exit_snapshot/ subdirectory containing the exact context at the moment of failure.

The GitLab CI pipeline can pattern-match the job log for the well-known transient signatures and retry once before reporting failure:

# .gitlab-ci.yml fragment
nightly_test:
  script:
    - bash run_test.sh
  retry:
    max: 1
    when:
      - script_failure
  # Custom retry logic via after_script that inspects the log
  after_script:
    - |
      if grep -qE "(Stale file handle|LNetError|Cannot allocate memory)" job.log && ! grep -q "FATAL ERROR" job.log; then
        echo "[INFO] Detected transient infrastructure failure; flagging for auto-retry" >&2
        exit 75   # EX_TEMPFAIL — triggers GitLab retry
      fi

This way, infrastructure flakes do not show up as code regressions on nightly dashboards.

The first MPI run launched 80 ranks with srun --hint=nomultithread. The output file OUTPUT.850135 was written by rank 0 on whichever compute node Slurm placed it. The subsequent prep_step call ran in the launcher shell. If those landed on different physical nodes, the Lustre metadata cache disagreement window is wider.

For setup-then-MPI-then-setup patterns, force the bookkeeping to a single node:

# Before
. prep_step
srun -l ... -n 80 ./pgm.x ...
. prep_step

# After
. prep_step
srun -l ... -n 80 ./pgm.x ...
sync                       # flush local filesystem cache
sleep 1                    # let metadata caches reconcile
. prep_step

The sync; sleep 1 is a cheap insurance policy that costs about 1 wall-clock second but eliminates the most common race window for ESTALE.

The cleanest path to addressing this is a small spec under .kiro/specs/ (or a focused PR) that lands all four code changes together:

• ush/prep_step_with_retry.sh — new helper.
• ush/err_exit.sh — snapshot diagnostic files before exit.
• ush/postamble.sh — preserve RUNDIR on non-zero exit (gated by KEEPDATA_ON_FAILURE).
• Ex-script updates that currently call . prep_step directly — switch to prep_step_retry.
• Documentation update at docs/source/troubleshooting/transient_filesystem_errors.rst.

The CI-level auto-retry (Best Practice 4) is independent and can land separately in the GitLab pipeline configuration.

• Source log (gist): https://gist.github.com/emcbot/9b9c80fed952cd5eaf80f3e26a96b663
• EE2 standards for prep_step / err_chk / err_exit / cpfs: confirmed via agentcore-mcp-rag.search_ee2_standards (category error_handling)
• Spack filesystem requirements documenting Lustre flock / metadata cache behaviour: agentcore-mcp-rag.search_documentation
• Global Workflow troubleshooting: section 10 "Common Errors / Known Issues" in the global-workflow RTD docs
• JGLOBAL_ENKF_ECEN task source: dev/jobs/JGLOBAL_ENKF_ECEN
• exglobal_enkf_ecen.sh: scripts/exglobal_enkf_ecen.sh
• prep_step utility: provided by prod-util/2.1.1 via spack-stack 1.9.2

• How frequently does this exact signature show up in nightly runs? A two-week scan of gitlab-ci logs grepping for Stale file handle would quantify the cost and prioritise the fix.
• Are there other steps (postsnd, fcst init) where prep_step is called in close succession? Those would benefit from the retry wrapper too.
• Would the global-workflow infrastructure team accept a KEEPDATA_ON_FAILURE=YES default? If yes, this reduces investigation friction across the entire workflow at near-zero scratch cost (failed runs are infrequent).