jgfs_atmos_gempak_master ecf (v17) Analysis - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

jgfs_atmos_gempak_master.ecf (GFS v17)

Branch / tenant: dev/gfs.v17 (gw_v17) Path: ecf/scripts/gfs/product/atmos/gempak/gempak/jgfs_atmos_gempak_master.ecf Analysis date: 2026-06-10 Method: AWS Bedrock AgentCore agentcore-mcp-rag MCP (graph traversal, GraphRAG, code-context) cross-checked against the on-disk v17 worktree.

A companion page, agentcore-mcp-rag Tool Usage Analysis, records which MCP tools were used, how useful each was, and a separation-of-concerns assessment of the toolset.


Summary

jgfs_atmos_gempak_master.ecf is the NCO production ecFlow task template that turns "GFS produced GRIB2 for forecast hour NNN" into "GEMPAK/NAWIPS files for hour NNN, on every output grid that hour qualifies for." It is the single PBS-batch entry point shared by every per-forecast-hour GEMPAK task in the operational GFS suite. At runtime it loads modules and immediately hands off to the J-Job ${HOMEgfs}/jobs/JGFS_ATMOS_GEMPAK, which drives exgfs_atmos_nawips.sh once per grid/forecast-hour combination.

The _master suffix is the NCO/global-workflow convention for one ecf template that is copied and renamed to every task instance in a family — here jgfs_atmos_gempak_f000 through jgfs_atmos_gempak_f384.


1. The script itself

#PBS -S /bin/bash
#PBS -N gfs_atmos_gempak_%CYC%_f%FHR%
#PBS -j oe
#PBS -q %QUEUE%
#PBS -A %PROJ%-%PROJENVIR%
#PBS -l walltime=00:20:00
#PBS -l select=1:mpiprocs=28:ompthreads=1:ncpus=28:mem=6GB
#PBS -l place=vscatter:shared
#PBS -l debug=true

model=gfs
export cyc="%CYC%"
%include <head.h>
%include <envir-p1.h>

############################################################
# Load modules
############################################################
#TODO: load only the modules needed for each job
source ${HOMEgfs}/dev/ush/load_modules.sh run

module list
export WGF="%WGF%"
export FHR3="%FHR%"
export FHR_LIST="%FHR_LIST%"
export COMPONENT=${WGF}
############################################################
# CALL executable job script here
############################################################
export CDATE=${PDY}${cyc}
export HOMEglobal=${HOMEgfs}

#### ${HOMEgfs}/dev/job_cards/rocoto/gempak.sh
${HOMEgfs}/jobs/JGFS_ATMOS_GEMPAK
if [ $? -ne 0 ]; then
   ecflow_client --msg="***JOB ${ECF_NAME} ERROR RUNNING J-SCRIPT ***"
   ecflow_client --abort
   exit
fi

%include <tail.h>
%manual

%end

What each part does

Section Role
#PBS directives PBS Pro batch resources: 1 node, 28 cpus/mpiprocs, 6 GB, 20-min wall, shared placement. Job name gfs_atmos_gempak_%CYC%_f%FHR%.
%include <head.h> / <envir-p1.h> NCO boilerplate: environment bootstrap, ecflow_client init, production environment (envir-p1).
load_modules.sh run v17 centralizes module loading in one ush script instead of per-task module load lines.
WGF, FHR3, FHR_LIST, COMPONENT ecFlow %VAR% placeholders injected per task by the suite definition. WGF = Weather-Generating-Forecast component (atmos).
HOMEglobal=${HOMEgfs} Bridges the v17 HOMEglobal naming used inside the J-Job to the NCO HOMEgfs.
${HOMEgfs}/jobs/JGFS_ATMOS_GEMPAK The actual work — the J-Job. Non-zero exit triggers ecflow_client --abort.
%include <tail.h> / %manual NCO completion handling and the operator manual block.

2. Why "_master" — the template mechanism

A single .ecf file backs hundreds of ecFlow tasks. The global-workflow ecFlow generator (dev/workflow/ecflow/ecflow_definitions.py, add_task(..., template=...)) looks up the named template and copies it to the destination, renaming it to the task name:

"If template is set, it will look for that template and then copy and change the name of the template at the destination to the name of the task." — ecflow_definitions.py, add_task docstring (rephrased for compliance)

So jgfs_atmos_gempak_master.ecf is deployed as jgfs_atmos_gempak_f000.ecf, jgfs_atmos_gempak_f001.ecf, … by the build step. The generated suite definition (ecf/defs/gfs_prod.def) shows the expansion:

family gempak
  family gempak
    task jgfs_atmos_gempak_f000
      edit FHR '000'
      edit FHR_LIST '0'
      trigger ../../product/jgfs_atmos_product_f000==complete
    task jgfs_atmos_gempak_f001
      edit FHR '001'
      edit FHR_LIST '1'
      trigger ../../product/jgfs_atmos_product_f001==complete
    ...
    task jgfs_atmos_gempak_f384
  • The GFS atmos GEMPAK family runs f000 → f384.
  • Each task overrides FHR, FHR_LIST, and inherits WGF 'atmos' from the product family.
  • edit ECF_FILES '%PACKAGEHOME%/ecf/scripts' sets the template search root.
  • The trigger gate is the matching upstream product/jgfs_atmos_product_fNNN==complete, so a given hour's GEMPAK conversion only fires once that hour's GRIB2 products exist.

The higher-level NCO suite intent is declared in dev/workflow/prod.yml under the gempak: family (sibling tasks jgfs_atmos_gempak, jgfs_atmos_gempak_meta, jgfs_atmos_gempak_ncdc_upapgif, jgfs_atmos_pgrb2_spec_gempak).


3. Execution chain

ecf/defs/gfs_prod.def : task jgfs_atmos_gempak_fNNN   (gated on product_fNNN == complete)
  └─ jgfs_atmos_gempak_master.ecf            (PBS submit; deployed as *_fNNN.ecf)
       └─ ${HOMEgfs}/jobs/JGFS_ATMOS_GEMPAK   (J-Job)
            └─ ${SCRglobal}/exgfs_atmos_nawips.sh   (ex-script, per grid)
                 └─ ${USHglobal}/product_functions.sh   (shared helper, sourced)

This chain was confirmed via the MCP graph (trace_data_flow, find_callers_callees, find_dependencies) against the gw_v17 tenant:

  • JGFS_ATMOS_GEMPAK INVOKES exgfs_atmos_nawips.sh and SOURCES jjob_header.sh, jjob_standard_vars.sh, jjob_shell_setup.sh, set_strict.sh.
  • exgfs_atmos_nawips.sh callers = {JGFS_ATMOS_GEMPAK} (fan-in 1); callee = product_functions.sh (fan-out 3, low complexity).
  • product_functions.sh is a leaf helper shared by 5 NAWIPS/AWIPS ex-scripts (exgfs_atmos_nawips.sh, exgfs_atmos_goes_nawips.sh, exgfs_atmos_awips_20km_1p0deg.sh, exgdas_atmos_nawips.sh, interp_atmos_master.sh).

Fuller J-Job import set (from find_dependencies)

JGFS_ATMOS_GEMPAK upstream imports (10): jjob_header.sh, jjob_standard_vars.sh, set_strict.sh, jjob_shell_setup.sh, exgfs_atmos_gempak_gif_ncdc_skew_t.sh, exgfs_atmos_nawips.sh, exgfs_atmos_goes_nawips.sh, exgfs_atmos_gempak_meta.sh, plus the ush/-prefixed header/vars. No circular dependencies; no downstream importers (it is a suite leaf).


4. What the J-Job actually produces

JGFS_ATMOS_GEMPAK defines COM directories and runs exgfs_atmos_nawips.sh once per grid, gated by forecast hour. Output lands under ${ROTDIR}/${RUN}.${PDY}/${cyc}/products/atmos/gempak/<grid>/.

Output grid Hour range Cadence DBN alert
1p00 0–240 every 3 h GFS_GEMPAK
1p00 252–384 every 12 h GFS_GEMPAK
0p50 0–240 every 3 h GFS_GEMPAK
0p50 246–276 every 6 h GFS_GEMPAK
0p50 288–384 every 12 h GFS_GEMPAK
0p25 0–120 every 1 h GFS_GEMPAK
0p25 123–240 every 3 h GFS_GEMPAK
0p25 252–384 every 12 h GFS_GEMPAK
35km_atl / 35km_pac / 40km 0 – min(180, FHMAX_GFS) every 3 h GFS_GEMPAK_WWB

Error handling uses the EE2-correct err_exit pattern (not set -e) on each grid block.


5. Key environment variables

Variable Origin Meaning
WGF suite edit WGF 'atmos' Weather-Generating-Forecast component; also exported as COMPONENT.
FHR / FHR3 per-task edit FHR Forecast hour (FHR3 zero-padded), drives the grid/cadence gates.
FHR_LIST per-task edit FHR_LIST Forecast-hour list. Degenerate (single hour) in the NCO def; the Rocoto counterpart dev/job_cards/rocoto/gempak.sh loops over it to bundle hours.
COMOUT_ATMOS_GEMPAK_<grid> J-Job (declare -rx) Output COM dirs per grid; consumers per graph: JGFS_ATMOS_GEMPAK, JGDAS_ATMOS_GEMPAK, JGFS_ATMOS_GEMPAK_PGRB2_SPEC, exgfs_atmos_goes_nawips.sh.
COMIN_ATMOS_GRIB_<grid> J-Job Input GRIB2 COM dirs (0p25/0p50/1p00).
DBN_ALERT_TYPE J-Job (:-GFS_GEMPAK) DBNet alert type for product distribution.
NAGRIB, GEMGRD, GRIBIN, WGRIB2, SENDDBN, DBNROOT nawips ex-script GEMPAK conversion + distribution controls (from find_callers_callees GGSR env edges).

config.gempak (v17)

########## config.gempak ##########
# GFS gempak step specific
export MAX_TASKS=42
source "${EXPDIR}/config.resources" gempak

Minimal — sets the per-member task cap and pulls task resources from config.resources.


6. v17-specific notes

  • Centralized module loading via load_modules.sh run (vs develop's per-task module load block) — single source of truth for module versions across the suite.
  • product/ family layer: the v17 path inserts product/ ( ecf/scripts/gfs/product/atmos/gempak/gempak/…) compared with develop's ecf/scripts/gfs/atmos/gempak/…, reflecting the reorganized product family tree.
  • HOMEglobal shim: export HOMEglobal=${HOMEgfs} is present in the v17 master.ecf (not in the develop analog), bridging the v17 J-Job's HOMEglobal/USHglobal/SCRglobal naming to the NCO HOMEgfs package layout.
  • FHR_LIST plumbed but single-valued in the NCO def — the interface is identical between Rocoto (where bundling can happen) and NCO ecFlow (one task = one hour), so a future change to bundle hours only needs edit FHR_LIST '0,3,6' on a task.
  • The commented #### ${HOMEgfs}/dev/job_cards/rocoto/gempak.sh is a breadcrumb to the Rocoto wrapper; in NCO the master.ecf calls the J-Job directly.

7. Change-impact snapshot (from MCP, gw_v17)

Symbol Change type Risk Direct dependents
JGFS_ATMOS_GEMPAK behavior LOW (0.10) 0 (suite leaf)
exgfs_atmos_nawips.sh signature LOW (0.30) 1 (JGFS_ATMOS_GEMPAK)

Editing exgfs_atmos_nawips.sh's argument contract (grid fhr3 alert comout) requires updating only its single J-Job caller, but note product_functions.sh is shared by five ex-scripts — changes there have wider blast radius.


Sources

  • On-disk v17 worktree: ecf/scripts/gfs/product/atmos/gempak/gempak/jgfs_atmos_gempak_master.ecf, dev/jobs/JGFS_ATMOS_GEMPAK, dev/parm/config/gfs/config.gempak, ecf/defs/gfs_prod.def, dev/workflow/prod.yml, dev/workflow/ecflow/ecflow_definitions.py.
  • agentcore-mcp-rag MCP (tenant gw_v17): trace_data_flow, find_dependencies, find_callers_callees, trace_execution_path, find_env_dependencies, get_change_impact, get_code_context.

Content was rephrased for compliance with licensing restrictions where source docstrings were quoted.

⚠️ **GitHub.com Fallback** ⚠️