GFS v17 EE2 Operational Compliance Review 2026 07 29 - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

GFS v17 — Full EE2 Operational Compliance Review

Date: 2026-07-29 Scope: dev/gfs.v17 branch (MCP tenant gw_v17) Reviewed tree: supported_repos/global-workflow_dev-gfs.v17 (on-disk worktree) Tooling Used: agentcore-mcp-rag MCP server (EE2 compliance module + GraphRAG) Standard: NCEP WCOSS Implementation Standards (EE2), Phase-2 SME-corrected patterns


Executive Summary

This is a complete EE2 operational-compliance review of 420 files on the dev/gfs.v17 branch, spanning every file the user scoped: all ush/ shell and Python code, all ecf/ ecFlow task scripts, all J-Jobs and ex-scripts that dev/ush/setup_gfs_for_nco.py stages for NCO, and all dev/parm/config/gfs configuration files. The rubric was drawn directly from the EE2 standards ingested in the MCP GraphRAG (generate_compliance_report, search_ee2_standards), and every category was exercised through the MCP EE2 tools (scan_repository_compliance, extract_code_for_analysis) backed by an exhaustive per-file static pass.

Overall posture: STRONG structural compliance with a small number of targeted documentation and observability gaps. The v17 suite faithfully follows the EE2 workflow contract — J-Job → ex-script → ush layering, JAAAAA job naming, err_exit-based error handling (the correct Phase-2 pattern), $SENDDBN-gated DBN alerts, prep_step/pgmout production-utility usage, and env-var-resolved executable paths (no hard-coded absolute product paths). No critical or production-blocking defects were found.

The findings that do exist are NCO-handoff readiness items, appropriate to this branch's staging lifecycle: (1) the per-line job-timing export (PS4) is missing on 65 of 66 J-Jobs; (2) the ecFlow %manual operator blocks are present but empty on 83 of 84 scripts (the exact work drafted in PR #5027, not yet merged to this branch); (3) explicit set -x debug logging is declared in only 4 of 38 shell ex-scripts (runtime tracing still works via inheritance); (4) formal DOCBLOCK headers are missing on 16 shell ex-scripts; and (5) one NCO-staged ex-script (exglobal_prep_ocean_obs.py) referenced by the setup script is not present on disk, which would break setup_gfs_for_nco.py.


What Was Reviewed

The authoritative scope was resolved from dev/ush/setup_gfs_for_nco.py (base_job_list, base_script_list) plus a full directory walk of ush/, ecf/, and dev/parm/config/gfs/.

Group Location Count Notes
J-Jobs dev/jobs/ 66 Every name in base_job_list; all present on disk
Ex-scripts dev/scripts/ 57 38 shell + 18 Python + 1 missing (exglobal_prep_ocean_obs.py)
ush shell ush/ 66 Task drivers + sourced function libraries
ush Python ush/ (incl. ush/python/pygfs/) 40 pygfs task classes + standalone helpers
ecFlow scripts ecf/scripts/ 84 GFS / GDAS / EnKF .ecf task templates
Config files dev/parm/config/gfs/ 107 config.*, *.j2 templates, yaml/defaults.yaml
Total 420

Path note. The user referenced parm/config/gfs; on dev/gfs.v17 the source-of-truth configs live under dev/parm/config/gfs (the same dev/-staged pattern as jobs and scripts — setup_gfs_for_nco.py copies them into place for NCO). Top-level parm/config does not exist on this branch.


Compliance Scorecard

Rates below are "files meeting the EE2 expectation / files where the expectation applies."

J-Jobs (dev/jobs/, 66)

EE2 check Rate Verdict
Shebang #! /usr/bin/env bash (line 1) 66/66
set -x debug logging 66/66
Error handling (err=$? / export err=$?err_exit/exit) 66/66
jjob_header.sh + jjob_shell_setup.sh sourced 65/66 ✅ (exception: JGLOBAL_FSM, the new forecast-state-machine job)
Working-dir cleanup (KEEPDATA/DATAROOT) 58/66 ⚠ 8 marine-analysis jobs defer cleanup to a finalize step
No hard-coded absolute executable paths 66/66
JAAAAA naming convention 66/66
export PS4 per-line timing 1/66 primary gap

Ex-scripts (dev/scripts/, 57)

EE2 check Rate Verdict
Shebang (bash/python) 56/56 present
Shell error handling (err_chk/err_exit/set_strict) 37/38 ✅ (exception: exglobal_cleanup.sh, delegates to caller)
$SENDDBN gating on all DBN alerts 13/13 ✅ 100% of scripts issuing alerts
prep_step + >> $pgmout 2> errfile on compiled execs present in producers ✅ (exgfs_atmos_fbwind.sh exemplary)
Python __main__ guard 18/18
Explicit set -x 4/38 shell ⚠ 34 rely on inherited trace
Formal DOCBLOCK header 22/38 shell ⚠ 16 missing (ad-hoc history comments present)
File present on disk 56/57 exglobal_prep_ocean_obs.py missing

ush (ush/, 106 — 66 shell + 40 Python)

EE2 check Rate Verdict
Shebang 106/106
No absolute executable paths 65/66 shell gfs_bfr2gpk.sh uses /bin/rm
Python module docstrings 31/40 ⚠ 9 minimal (__init__.py, legacy calc helpers)
Sourced-library structure (functions, set +x/-x discipline) clean ✅ (timer.sh exemplary)

ecFlow scripts (ecf/scripts/, 84)

EE2 check Rate Verdict
%include <head.h> 84/84
%include <tail.h> 84/84
ecflow_client --abort J-script error handling present
%manual/%end block present 83/84 ✅ (cycle_end.ecf is suite-control, exempt)
%manual populated with PURPOSE + TROUBLESHOOTING 1/84 primary gap

Config files (dev/parm/config/gfs/, 107)

EE2 check Rate Verdict
Shebang on executable config.* compliant
No lowercase boolean exports (YES/NO caps) 107/107
BEGIN:/END: echo + source config.resources pattern standard

Findings and Root Cause

Finding 1 — [MEDIUM] PS4 per-line timing missing on 65/66 J-Jobs

What. EE2 (standards.rst lines 868–919) calls for J-Jobs to export a timing-annotated prompt string so every traced line in the job log carries an elapsed-seconds stamp:

export PS4='+ $SECONDS + '

scan_repository_compliance flagged this on every J-Job it sampled, and the full static pass confirmed only 1 of 66 J-Jobs sets PS4.

Why it matters. Without PS4 timing, operators triaging a slow or hung job cannot read per-step wall-clock directly from the log — they must cross- reference external timers. It is a log-observability requirement, not a correctness one, which is why the jobs still run correctly today.

Finding 2 — [MEDIUM] ecFlow %manual operator blocks are empty (83/84)

What. Every .ecf task template carries a %manual … %end block (the text ecFlow surfaces in the GUI when an operator right-clicks a task), but on v17 those blocks are empty: 0 of 84 contain a PURPOSE section and 0 contain a TROUBLESHOOTING section. The lone documented script is ecf/scripts/gdas/atmos/analysis/jgdas_atmos_analysis_wdqms.ecf.

Root cause. This is precisely the work drafted in PR #5027 "(dev/gfs.v17) Add %manual documentation blocks to 77 ecFlow scripts" and reconciled in the wiki analyses v17-ecf-scripts-summary and v17-ecf-script-reconciliation. That PR was not merged into the branch state reviewed here, so the operator-facing guidance is still absent from the %manual blocks. This is a documentation-completeness gap, not a code defect — the blocks are invisible to PBS/Slurm and do not affect execution.

Finding 3 — [LOW-MEDIUM] Explicit set -x absent in 34/38 shell ex-scripts

What. The MCP scan flags "Missing debug logging" on shell ex-scripts that do not declare set -x. Only 4 of 38 shell ex-scripts declare it explicitly.

Why the runtime impact is limited. Every J-Job sets set -x on line 2 and sources set_strict.sh before invoking its ex-script, so the child inherits the trace flag at runtime and logs are in fact traced. EE2 nonetheless expects each operational script to declare set -x so it is self-tracing when run or tested in isolation. Treat this as a defense-in-depth / standalone-testability item.

Finding 4 — [LOW] Formal DOCBLOCK headers missing on 16 shell ex-scripts

What. EE2 Example 4 defines a DOCBLOCK template (Program Name, Abstract, History Log, Usage, Input/Output Files, Condition codes). 16 shell ex-scripts lack a structured DOCBLOCK. Many still carry useful ad-hoc history comments (e.g. exgfs_atmos_fbwind.sh has a dated change log), so this is a formatting/consistency gap rather than an absence of documentation. Python ex-scripts use module docstrings/logit logging rather than the shell DOCBLOCK form.

Finding 5 — [LOW] gfs_bfr2gpk.sh uses absolute /bin/rm

What. ush/gfs_bfr2gpk.sh invokes /bin/rm by absolute path. EE2 prefers PATH-resolved system utilities (rm). This is minor (a system utility, not a product executable) and the same script is already the subject of open PR #5171.

Finding 6 — [INFO / correctness] NCO-staged ex-script missing on disk

What. setup_gfs_for_nco.py::base_script_list lists exglobal_prep_ocean_obs.py, and JGLOBAL_PREP_OCEAN_OBS exists in dev/jobs/, but the ex-script is not present at dev/scripts/exglobal_prep_ocean_obs.py. Because copy_files() raises FileNotFoundError on any missing source, running setup_gfs_for_nco.py in its current state would abort when it reaches this entry.

Likely explanation. The ocean-obs prep step is provided by the gdas.cd submodule rather than the top-level dev/scripts, so either the list entry should be removed/redirected or the file needs to be staged. Flagged for the v17 team to confirm — this is the one finding with functional (not stylistic) impact.


Code-Level Tracing (GraphRAG / MCP Insights)

The MCP EE2 tools were run against real file content, not just heuristics:

  • scan_repository_compliance (J-Job batch) classified JGLOBAL_CLEANUP and JGDAS_ATMOS_VERFRAD as job_cards and returned exactly one automated violation each — "J-job missing PS4 timing export" — with no error-handling, environment-variable, or production-utility violations, confirming the err_exit pattern is accepted under the Phase-2 SME rules (set -eu/set -e are not required).

  • scan_repository_compliance (ex-script + ecf batch) flagged exgfs_atmos_fbwind.sh only for "Missing debug logging (add set -x)" and otherwise found it clean — validating that its prep_step, export err=$?; err_exit, and >> ${pgmout} 2> errfile usage is fully compliant.

  • extract_code_for_analysis (categories output_file_naming, error_handling, env_var_validation) on exgfs_atmos_fbwind.sh confirmed the output filename ${COMOUT}/${RUN}.atmos.t${cyc}z.fbwind.pacific.ascii is EE2-compliant (lowercase, period-separated categories, no embedded date, no special characters) and that err_exit is the correct fatal-error mechanism.

  • scan_repository_compliance (ush + config batch) on ush/timer.sh and dev/parm/config/gfs/config.fbwind returned zero issues — the sourced function-library and config-resources patterns are clean.

  • A heuristic false-positive was caught and corrected during tracing: JGLOBAL_FETCH, JGDAS_FIT2OBS, and JGFS_ATMOS_VERIFICATION initially looked non-compliant on error handling because they use err=$? (without export); reading the files confirmed all three follow err=$? with a proper err_exit/exit ${err} guard and are compliant. Error handling is therefore effectively universal (66/66) across the J-Jobs.


Suggested Fixes / Best Practices

Ordered by operational value. None are production-blocking.

  1. Add PS4 timing to all J-Jobs (Finding 1). Insert immediately after set -x in each dev/jobs/J*:

    export PS4='+ $SECONDS + '

    Best handled centrally — e.g. export it once in ush/jjob_header.sh so all 66 jobs inherit it in a single change rather than 66 edits.

  2. Populate the ecFlow %manual blocks (Finding 2). Merge / rebase PR #5027 onto the current dev/gfs.v17. The PURPOSE/TROUBLESHOOTING text already exists in the v17-ecf-scripts-catalog.csv reconciliation artifact; this is a re-application, not new authoring.

  3. Declare set -x in ex-scripts (Finding 3). Add set -x after the shebang in the 34 shell ex-scripts that lack it (or standardize on sourcing a shared preamble). Low effort, improves standalone testability.

  4. Add DOCBLOCK headers (Finding 4). Apply the EE2 Example-4 template to the 16 shell ex-scripts missing it; fold the existing ad-hoc history comments into the History Log section.

  5. Resolve the missing ocean-obs ex-script (Finding 6). Confirm whether exglobal_prep_ocean_obs.py should be staged from gdas.cd or removed from base_script_list; today setup_gfs_for_nco.py would fail on it.

  6. Minor: switch /bin/rmrm in ush/gfs_bfr2gpk.sh (tracks with open PR #5171).


MCP Tool Effectiveness Tally

Tool Invoked Parameters Accuracy & Effectiveness Notes & Recommendations
generate_compliance_report scope: detailed, format: markdown High — returned the full 7-category EE2 rubric that anchored the review The ENVIRONMENT VARIABLES section rendered as a placeholder ("being retrieved"); the other six categories returned rich standards text. Worth fixing that one section's retrieval.
search_ee2_standards query, tenant_id: gw_v17 Low for this query — returned 0 hits Expected: EE2 standards are branch-agnostic/org-wide and not ingested under the gw_v17 prefix. Confirms tenant scoping works; standards should be queried tenant-agnostically.
scan_repository_compliance 3 batches: J-Jobs; ex-script+ecf; ush+config ({name,content}, all 5 categories) High — correct file-type classification, surfaced the PS4 and set -x findings, and correctly returned 0 false positives on err_exit/set -eu (Phase-2 SME patterns held) Automated check set is intentionally narrow (PS4, shebang, set -x, type); depth comes from the passthrough. repository_path is unusable against the AWS backend (EFS /mnt/workflow not mounted) — content-passing is required.
extract_code_for_analysis categories: output_file_naming, error_handling, env_var_validation, content_type: bash High — returned per-category prompts + SME corrections + extracted signals (set -x: NOT FOUND, err_exit line) enabling authoritative naming/error judgments The passthrough-reasoning design (tool provides signal + rubric, LLM classifies) worked well and is the right pattern for the categories the scanner does not auto-check.

Overall MCP effectiveness: High. The EE2 module correctly anchored the rubric, classified file types, enforced the Phase-2 SME corrections (no set -eu false positives), and surfaced the two dominant real findings (PS4 timing, empty %manual blocks). The GraphRAG issue tooling (search_issues) independently tied the %manual finding to PR #5027. Terminal fallback was used only for exhaustive per-file enumeration and to correct one heuristic false-positive, per the CI-error-reporting tool policy.


Methodology Note

Compliance judgments were produced by the agentcore-mcp-rag EE2 tools applying the Phase-2 SME-corrected patterns (err_chk/err_exit is correct; set -eu is not required). Complete 420-file coverage was achieved by pairing those tool calls with an exhaustive static enumeration of the same rubric markers, so every in-scope file is accounted for in the scorecard rather than sampled. dev/gfs.v17 is a staging-lifecycle branch; findings are framed as NCO-handoff readiness items, not production defects.

⚠️ **GitHub.com Fallback** ⚠️