Lingering restart files can cause re‐ran cases to fail - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

Looking at the JGLOBAL_FORECAST script, I can see how the paths are constructed, but you're right - we need to understand what specific case/run is causing the issue and where exactly it's looking for and finding these problematic restart files.

Let's analyze how the paths are determined:

  1. For non-ensemble runs:

    export DATAjob="${DATAROOT}/${RUN}fcst.${PDY:-}${cyc}"

  2. For ensemble members:

    export DATAjob="${DATAROOT}/${RUN}efcs${ENSMEM}.${PDY:-}${cyc}"

  3. Then the restart directory is set as:

    export DATArestart="${DATAjob}/restart"

From the error log, we can extract some information to identify the specific case:

  1. Looking at the paths in the error log, I can see:

    /gpfs/f6/drsa-precip3/scratch/role.glopara/RUNDIRS/C48_S2SWA_gefs_28431ae/gefs.2021032312/gefsfcst.2021032312

  2. Breaking this down:

    • DATAROOT appears to be /gpfs/f6/drsa-precip3/scratch/role.glopara/RUNDIRS/C48_S2SWA_gefs_28431ae/gefs.2021032312
    • RUN appears to be gefs
    • PDY is 20210323
    • cyc is 12

  3. The job is finding restart files with timestamp 20210325.120000 in:

    /gpfs/f6/drsa-precip3/scratch/role.glopara/RUNDIRS/C48_S2SWA_gefs_28431ae/gefs.2021032312/gefsfcst.2021032312/restart/FV3_RESTART

  4. These restart files are two days ahead of the current cycle date (March 23 vs March 25), which is why the job is trying to use them as a warm start, even though they shouldn't be used.

This means we're looking at a GEFS (Global Ensemble Forecast System) run for March 23, 2021, 12Z, but the restart directory contains files from a future date (March 25, 2021), which is causing the issue.

The configuration C48_S2SWA_gefs_28431ae suggests this is a low-resolution (C48) Subseasonal-to-Seasonal (S2S) configuration with wave and aerosol components (S2SWA) for GEFS, with a specific commit ID or branch identifier (28431ae).

To summarize:

  1. The run is a GEFS forecast for 2021-03-23 12Z
  2. It's using the C48_S2SWA_gefs configuration
  3. The restart directory contains files from 2021-03-25 12Z
  4. The code is finding these restart files and trying to warm start from them
  5. This is causing the forecast to fail

The cleanup code you've added to JGLOBAL_FORECAST should prevent this by removing any existing restart directories at the start of the run, ensuring that old restart files won't be detected and incorrectly used.