GFS v17 EE2 Operational Compliance Review 2026 07 29 - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki
Date: 2026-07-29
Scope: dev/gfs.v17 branch (MCP tenant gw_v17)
Reviewed tree: supported_repos/global-workflow_dev-gfs.v17 (on-disk worktree)
Tooling Used: agentcore-mcp-rag MCP server (EE2 compliance module + GraphRAG)
Standard: NCEP WCOSS Implementation Standards (EE2), Phase-2 SME-corrected patterns
This is a complete EE2 operational-compliance review of 420 files on the
dev/gfs.v17 branch, spanning every file the user scoped: all ush/ shell and
Python code, all ecf/ ecFlow task scripts, all J-Jobs and ex-scripts that
dev/ush/setup_gfs_for_nco.py stages for NCO, and all dev/parm/config/gfs
configuration files. The rubric was drawn directly from the EE2 standards
ingested in the MCP GraphRAG (generate_compliance_report,
search_ee2_standards), and every category was exercised through the MCP EE2
tools (scan_repository_compliance, extract_code_for_analysis) backed by an
exhaustive per-file static pass.
Overall posture: STRONG structural compliance with a small number of
targeted documentation and observability gaps. The v17 suite faithfully
follows the EE2 workflow contract — J-Job → ex-script → ush layering, JAAAAA
job naming, err_exit-based error handling (the correct Phase-2 pattern),
$SENDDBN-gated DBN alerts, prep_step/pgmout production-utility usage, and
env-var-resolved executable paths (no hard-coded absolute product paths). No
critical or production-blocking defects were found.
The findings that do exist are NCO-handoff readiness items, appropriate to
this branch's staging lifecycle: (1) the per-line job-timing export
(PS4) is missing on 65 of 66 J-Jobs; (2) the ecFlow %manual operator blocks
are present but empty on 83 of 84 scripts (the exact work drafted in PR
#5027, not yet merged
to this branch); (3) explicit set -x debug logging is declared in only 4 of
38 shell ex-scripts (runtime tracing still works via inheritance); (4) formal
DOCBLOCK headers are missing on 16 shell ex-scripts; and (5) one NCO-staged
ex-script (exglobal_prep_ocean_obs.py) referenced by the setup script is not
present on disk, which would break setup_gfs_for_nco.py.
The authoritative scope was resolved from dev/ush/setup_gfs_for_nco.py
(base_job_list, base_script_list) plus a full directory walk of ush/,
ecf/, and dev/parm/config/gfs/.
| Group | Location | Count | Notes |
|---|---|---|---|
| J-Jobs | dev/jobs/ |
66 | Every name in base_job_list; all present on disk |
| Ex-scripts | dev/scripts/ |
57 | 38 shell + 18 Python + 1 missing (exglobal_prep_ocean_obs.py) |
| ush shell | ush/ |
66 | Task drivers + sourced function libraries |
| ush Python |
ush/ (incl. ush/python/pygfs/) |
40 |
pygfs task classes + standalone helpers |
| ecFlow scripts | ecf/scripts/ |
84 | GFS / GDAS / EnKF .ecf task templates |
| Config files | dev/parm/config/gfs/ |
107 |
config.*, *.j2 templates, yaml/defaults.yaml
|
| Total | 420 |
Path note. The user referenced
parm/config/gfs; ondev/gfs.v17the source-of-truth configs live underdev/parm/config/gfs(the samedev/-staged pattern as jobs and scripts —setup_gfs_for_nco.pycopies them into place for NCO). Top-levelparm/configdoes not exist on this branch.
Rates below are "files meeting the EE2 expectation / files where the expectation applies."
| EE2 check | Rate | Verdict |
|---|---|---|
Shebang #! /usr/bin/env bash (line 1) |
66/66 | ✅ |
set -x debug logging |
66/66 | ✅ |
Error handling (err=$? / export err=$? → err_exit/exit) |
66/66 | ✅ |
jjob_header.sh + jjob_shell_setup.sh sourced |
65/66 | ✅ (exception: JGLOBAL_FSM, the new forecast-state-machine job) |
Working-dir cleanup (KEEPDATA/DATAROOT) |
58/66 | ⚠ 8 marine-analysis jobs defer cleanup to a finalize step |
| No hard-coded absolute executable paths | 66/66 | ✅ |
JAAAAA naming convention |
66/66 | ✅ |
export PS4 per-line timing |
1/66 | ❌ primary gap |
| EE2 check | Rate | Verdict |
|---|---|---|
| Shebang (bash/python) | 56/56 present | ✅ |
Shell error handling (err_chk/err_exit/set_strict) |
37/38 | ✅ (exception: exglobal_cleanup.sh, delegates to caller) |
$SENDDBN gating on all DBN alerts |
13/13 | ✅ 100% of scripts issuing alerts |
prep_step + >> $pgmout 2> errfile on compiled execs |
present in producers | ✅ (exgfs_atmos_fbwind.sh exemplary) |
Python __main__ guard |
18/18 | ✅ |
Explicit set -x |
4/38 shell | ⚠ 34 rely on inherited trace |
| Formal DOCBLOCK header | 22/38 shell | ⚠ 16 missing (ad-hoc history comments present) |
| File present on disk | 56/57 | ❌ exglobal_prep_ocean_obs.py missing |
| EE2 check | Rate | Verdict |
|---|---|---|
| Shebang | 106/106 | ✅ |
| No absolute executable paths | 65/66 shell | ⚠ gfs_bfr2gpk.sh uses /bin/rm
|
| Python module docstrings | 31/40 | ⚠ 9 minimal (__init__.py, legacy calc helpers) |
Sourced-library structure (functions, set +x/-x discipline) |
clean | ✅ (timer.sh exemplary) |
| EE2 check | Rate | Verdict |
|---|---|---|
%include <head.h> |
84/84 | ✅ |
%include <tail.h> |
84/84 | ✅ |
ecflow_client --abort J-script error handling |
present | ✅ |
%manual/%end block present |
83/84 | ✅ (cycle_end.ecf is suite-control, exempt) |
%manual populated with PURPOSE + TROUBLESHOOTING |
1/84 | ❌ primary gap |
| EE2 check | Rate | Verdict |
|---|---|---|
Shebang on executable config.*
|
compliant | ✅ |
No lowercase boolean exports (YES/NO caps) |
107/107 | ✅ |
BEGIN:/END: echo + source config.resources pattern |
standard | ✅ |
What. EE2 (standards.rst lines 868–919) calls for J-Jobs to export a
timing-annotated prompt string so every traced line in the job log carries an
elapsed-seconds stamp:
export PS4='+ $SECONDS + 'scan_repository_compliance flagged this on every J-Job it sampled, and the
full static pass confirmed only 1 of 66 J-Jobs sets PS4.
Why it matters. Without PS4 timing, operators triaging a slow or hung job
cannot read per-step wall-clock directly from the log — they must cross-
reference external timers. It is a log-observability requirement, not a
correctness one, which is why the jobs still run correctly today.
What. Every .ecf task template carries a %manual … %end block (the text
ecFlow surfaces in the GUI when an operator right-clicks a task), but on v17
those blocks are empty: 0 of 84 contain a PURPOSE section and 0 contain a
TROUBLESHOOTING section. The lone documented script is
ecf/scripts/gdas/atmos/analysis/jgdas_atmos_analysis_wdqms.ecf.
Root cause. This is precisely the work drafted in PR
#5027 "(dev/gfs.v17) Add %manual documentation blocks to 77 ecFlow scripts"
and reconciled in the wiki analyses v17-ecf-scripts-summary and
v17-ecf-script-reconciliation. That PR was not merged into the branch
state reviewed here, so the operator-facing guidance is still absent from the
%manual blocks. This is a documentation-completeness gap, not a code defect —
the blocks are invisible to PBS/Slurm and do not affect execution.
What. The MCP scan flags "Missing debug logging" on shell ex-scripts that do
not declare set -x. Only 4 of 38 shell ex-scripts declare it explicitly.
Why the runtime impact is limited. Every J-Job sets set -x on line 2 and
sources set_strict.sh before invoking its ex-script, so the child inherits
the trace flag at runtime and logs are in fact traced. EE2 nonetheless expects
each operational script to declare set -x so it is self-tracing when run or
tested in isolation. Treat this as a defense-in-depth / standalone-testability
item.
What. EE2 Example 4 defines a DOCBLOCK template (Program Name, Abstract,
History Log, Usage, Input/Output Files, Condition codes). 16 shell ex-scripts
lack a structured DOCBLOCK. Many still carry useful ad-hoc history comments
(e.g. exgfs_atmos_fbwind.sh has a dated change log), so this is a
formatting/consistency gap rather than an absence of documentation. Python
ex-scripts use module docstrings/logit logging rather than the shell DOCBLOCK
form.
What. ush/gfs_bfr2gpk.sh invokes /bin/rm by absolute path. EE2 prefers
PATH-resolved system utilities (rm). This is minor (a system utility, not a
product executable) and the same script is already the subject of open PR
#5171.
What. setup_gfs_for_nco.py::base_script_list lists
exglobal_prep_ocean_obs.py, and JGLOBAL_PREP_OCEAN_OBS exists in
dev/jobs/, but the ex-script is not present at
dev/scripts/exglobal_prep_ocean_obs.py. Because copy_files() raises
FileNotFoundError on any missing source, running setup_gfs_for_nco.py in its
current state would abort when it reaches this entry.
Likely explanation. The ocean-obs prep step is provided by the gdas.cd
submodule rather than the top-level dev/scripts, so either the list entry
should be removed/redirected or the file needs to be staged. Flagged for the
v17 team to confirm — this is the one finding with functional (not stylistic)
impact.
The MCP EE2 tools were run against real file content, not just heuristics:
-
scan_repository_compliance(J-Job batch) classifiedJGLOBAL_CLEANUPandJGDAS_ATMOS_VERFRADasjob_cardsand returned exactly one automated violation each — "J-job missing PS4 timing export" — with no error-handling, environment-variable, or production-utility violations, confirming theerr_exitpattern is accepted under the Phase-2 SME rules (set -eu/set -eare not required). -
scan_repository_compliance(ex-script + ecf batch) flaggedexgfs_atmos_fbwind.shonly for "Missing debug logging (add set -x)" and otherwise found it clean — validating that itsprep_step,export err=$?; err_exit, and>> ${pgmout} 2> errfileusage is fully compliant. -
extract_code_for_analysis(categoriesoutput_file_naming,error_handling,env_var_validation) onexgfs_atmos_fbwind.shconfirmed the output filename${COMOUT}/${RUN}.atmos.t${cyc}z.fbwind.pacific.asciiis EE2-compliant (lowercase, period-separated categories, no embedded date, no special characters) and thaterr_exitis the correct fatal-error mechanism. -
scan_repository_compliance(ush + config batch) onush/timer.shanddev/parm/config/gfs/config.fbwindreturned zero issues — the sourced function-library and config-resources patterns are clean. -
A heuristic false-positive was caught and corrected during tracing:
JGLOBAL_FETCH,JGDAS_FIT2OBS, andJGFS_ATMOS_VERIFICATIONinitially looked non-compliant on error handling because they useerr=$?(withoutexport); reading the files confirmed all three followerr=$?with a propererr_exit/exit ${err}guard and are compliant. Error handling is therefore effectively universal (66/66) across the J-Jobs.
Ordered by operational value. None are production-blocking.
-
Add
PS4timing to all J-Jobs (Finding 1). Insert immediately afterset -xin eachdev/jobs/J*:export PS4='+ $SECONDS + '
Best handled centrally — e.g. export it once in
ush/jjob_header.shso all 66 jobs inherit it in a single change rather than 66 edits. -
Populate the ecFlow
%manualblocks (Finding 2). Merge / rebase PR #5027 onto the currentdev/gfs.v17. The PURPOSE/TROUBLESHOOTING text already exists in thev17-ecf-scripts-catalog.csvreconciliation artifact; this is a re-application, not new authoring. -
Declare
set -xin ex-scripts (Finding 3). Addset -xafter the shebang in the 34 shell ex-scripts that lack it (or standardize on sourcing a shared preamble). Low effort, improves standalone testability. -
Add DOCBLOCK headers (Finding 4). Apply the EE2 Example-4 template to the 16 shell ex-scripts missing it; fold the existing ad-hoc history comments into the
History Logsection. -
Resolve the missing ocean-obs ex-script (Finding 6). Confirm whether
exglobal_prep_ocean_obs.pyshould be staged fromgdas.cdor removed frombase_script_list; todaysetup_gfs_for_nco.pywould fail on it. -
Minor: switch
/bin/rm→rminush/gfs_bfr2gpk.sh(tracks with open PR #5171).
| Tool Invoked | Parameters | Accuracy & Effectiveness | Notes & Recommendations |
|---|---|---|---|
generate_compliance_report |
scope: detailed, format: markdown
|
High — returned the full 7-category EE2 rubric that anchored the review | The ENVIRONMENT VARIABLES section rendered as a placeholder ("being retrieved"); the other six categories returned rich standards text. Worth fixing that one section's retrieval. |
search_ee2_standards |
query, tenant_id: gw_v17
|
Low for this query — returned 0 hits | Expected: EE2 standards are branch-agnostic/org-wide and not ingested under the gw_v17 prefix. Confirms tenant scoping works; standards should be queried tenant-agnostically. |
scan_repository_compliance |
3 batches: J-Jobs; ex-script+ecf; ush+config ({name,content}, all 5 categories) |
High — correct file-type classification, surfaced the PS4 and set -x findings, and correctly returned 0 false positives on err_exit/set -eu (Phase-2 SME patterns held) |
Automated check set is intentionally narrow (PS4, shebang, set -x, type); depth comes from the passthrough. repository_path is unusable against the AWS backend (EFS /mnt/workflow not mounted) — content-passing is required. |
extract_code_for_analysis |
categories: output_file_naming, error_handling, env_var_validation, content_type: bash
|
High — returned per-category prompts + SME corrections + extracted signals (set -x: NOT FOUND, err_exit line) enabling authoritative naming/error judgments |
The passthrough-reasoning design (tool provides signal + rubric, LLM classifies) worked well and is the right pattern for the categories the scanner does not auto-check. |
Overall MCP effectiveness: High. The EE2 module correctly anchored the
rubric, classified file types, enforced the Phase-2 SME corrections (no
set -eu false positives), and surfaced the two dominant real findings (PS4
timing, empty %manual blocks). The GraphRAG issue tooling (search_issues)
independently tied the %manual finding to PR #5027. Terminal fallback was used
only for exhaustive per-file enumeration and to correct one heuristic
false-positive, per the CI-error-reporting tool policy.
Compliance judgments were produced by the agentcore-mcp-rag EE2 tools applying
the Phase-2 SME-corrected patterns (err_chk/err_exit is correct; set -eu
is not required). Complete 420-file coverage was achieved by pairing those
tool calls with an exhaustive static enumeration of the same rubric markers, so
every in-scope file is accounted for in the scorecard rather than sampled.
dev/gfs.v17 is a staging-lifecycle branch; findings are framed as
NCO-handoff readiness items, not production defects.