C48 ATM ecFlow Suite Generator Report - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki
Analysis by GitHub Copilot (Claude Opus 4.8, VS Code), directed and reviewed by Terry.McGuinness. Implementation work described here is by Anton Fernando ([email protected]).
Report date: 2026-09-22.
The branch feature/ecflow-c48-atm turns the forecast-only ecFlow
scaffold — documented in the companion page
GFS-Forecast-Only-ecFlow-and-Rocoto-Report — into a working,
loadable ecFlow suite for the smallest atmosphere-only CI case, C48_ATM.
Where the September 2025 scaffold left get_cycledefs() and write() as
pass, this branch supplies a Rocoto-parallel task-class hierarchy, a shared
.ecf script + header library, and an end-to-end case loader that stands the
suite up on a running ecFlow server.
This is the first concrete implementation increment of NOAA-EMC epic #4106 "Dynamically Generate EcFlow Suites" (author AnnaSmoot-NOAA, open), whose stated goal is to remove the situation where "Rocoto is the only workflow engine option for developers and the EcFlow engine is hard coded for production."
All implementation on this branch is by Anton Fernando. The work is nine
non-merge commits authored 2026-09-17 → 2026-09-21, adding 21 files and
~1,751 lines under dev/workflow/ecflow/ and dev/ecflow/.
- Inspected checkout:
supported_repos/global-workflow_forkedon branchfeature/ecflow-c48-atm, trackinganton/feature/ecflow-c48-atm(https://github.com/AntonMFernando-NOAA/global-workflow.git). - Merge-base with
noaa-emc/develop:5e6a11c60. The branch is 37 commits ahead, of which 9 are branch-authored and 28 are pass-through merges bringingdevelopforward. This report describes only the 9 authored commits; the develop merges are out of scope. - Line/file counts are restricted to the two directories the branch owns
(
dev/workflow/ecflow/,dev/ecflow/) viagit diff --numstat noaa-emc/develop...HEAD -- <dirs>. - No source files were modified during this analysis; findings are read-only.
- EIB MCP graph/doc queries were issued against tenant
gw, branchdevelop. Direct source and Git history are authoritative for provenance and implementation claims; MCP results are supporting context.
| Evidence | Recorded value |
|---|---|
| Implementer (all 9 commits) | Anton Fernando <[email protected]>
|
| Remote |
anton/feature/ecflow-c48-atm (AntonMFernando-NOAA fork) |
| First authored commit |
e47fa99fb — 2026-09-17 20:40:59 +0000 |
| Latest authored commit |
f6520ebd0 — 2026-09-21 20:07:44 +0000 |
| Authored commits | 9 (non-merge) |
| Files changed (branch dirs) | 21 |
| Line delta (branch dirs) | +1,751 / −19 |
| Base epic | NOAA-EMC #4106 "Dynamically Generate EcFlow Suites" |
| Predecessor scaffold | Rahul Mahajan, PR #4071 (2025-09-17), see companion report |
| Commit | Date (UTC) | Subject |
|---|---|---|
e47fa99fb |
2026-09-17 | chore(ecflow): Add ECF scripts and workflow utilities for GFS forecasting |
a91692b53 |
2026-09-17 | fix(ecflow): Escape printf format string in atmos_prod.ecf |
42ea94620 |
2026-09-17 | feat(ecflow): Add cleanup status tracking and conditional experiment overwrite |
58eebed4e |
2026-09-18 | docs(ecflow): Update documentation from symlink to script directory approach |
85e340a6a |
2026-09-18 | feat(ecflow): Add C48_ATM test case loader and generalize ecFlow case loading |
3f56bebb9 |
2026-09-21 | feat(ecflow): Add C48_ATM ecFlow suite with task mappings and triggers |
8fc8005e4 |
2026-09-21 | refactor(ecflow): Restructure task definitions into modular factory pattern |
2c32c1bd5 |
2026-09-21 | feat(ecflow): Add CLI overrides for pslot, expdir, and comroot |
f6520ebd0 |
2026-09-21 | feat(ecflow): Add CLI override for STMP path |
The design mirrors the existing dev/workflow/rocoto/ package so the two
engines stay symmetrical and share the resource / forecast-hour / system
plumbing. The only intended divergence is the output shape: dictionaries
with ecFlow trigger strings instead of Rocoto XML fragments.
| File | Role |
|---|---|
ecflow_tasks.py |
EcFlowTasks base class, subclass of rocoto.tasks.Tasks. Reuses resource extraction and forecast-hour grouping; get_ecflow_task() dispatches to a per-task method returning a dict. |
gfs_ecflow_tasks.py |
GFSEcFlowTasks: one method per task (fetch, stage_ic, aerosol_init, waveinit, fcst, atmupp, atmos_prod, …). Dependencies expressed as ecFlow triggers such as fcst == complete. |
ecflow_tasks_factory.py |
wxflow.Factory registering gfs → GFSEcFlowTasks. gefs / sfs / gcafs registered-but-commented, reserved for follow-on branches. |
ecflow_workflow_tasks.py |
get_ecflow_tasks(app_config) standalone driver that walks every RUN/task and returns the full task-dict list. |
gfs_forecast_only_ecflow.py |
+385/−19: the forecast-only suite generator that consumes the task dicts and emits the .def families and triggers. |
load_ecflow_case.py |
Generic case loader: parse case YAML → setup_expt → setup_workflow (ecflow engine) → load .def into the ecFlow server → optionally begin. Reads ECF_HOST / ECF_PORT / HOMEglobal. |
c48_atm_ecflow.py |
20-line entry point; sets the default case to dev/ci/cases/pr/C48_ATM.yaml and delegates to load_ecflow_case.run(). |
sync_ecf_scripts.sh |
Refreshes ECF_FILES from repo .ecf sources via a manifest, so a script edit does not require regenerating the whole .def. |
Nine task scripts under dev/ecflow/scripts/ — stage_ic, fcst,
atmos_prod, tracker, genesis, metp, arch_vrfy, arch_tars,
cleanup. Each is a thin ecFlow wrapper that %includes the shared headers,
exports the ecFlow-substituted variables (%HOMEglobal%, %EXPDIR%,
%COMROOT%, …), sources dev/ush/load_modules.sh run, invokes the existing
J-Job (e.g. dev/jobs/JGLOBAL_FCST), and signals completion via tail.h.
Shared headers under dev/ecflow/utils/:
| Header | Purpose |
|---|---|
slurm.h |
#SBATCH directives templated from ecFlow vars (%NODES%, %WALLTIME%, %QUEUE%, …). |
head.h |
Sets ECF_NAME/HOST/PORT/PASS/TRYNO/RID/JOB, calls ecflow_client --init, and defines the ERROR trap. Notably it saves _ECF_HOST_SAVED/_ECF_PORT_SAVED before load_modules.sh resets the module environment, because the module reset can clobber those exports. |
tail.h |
ecflow_client --complete and trap teardown. |
envir.h |
NCO-style environment defaults. |
Notable fix (a91692b53): atmos_prod.ecf loops over a comma-separated
%FHR_LIST% (one script serving a fanned-out forecast-hour family). The
printf format had to be escaped as %%03d because ecFlow performs its own
%VAR% substitution on the script before Slurm ever runs it — an unescaped
%03d would be eaten by the ecFlow pre-processor.
-
dev/ctests/cases/C48_S2SW*YAML anddev/ctests/CMakeLists.txtupdated so the CI harness recognizes ecFlow-loaded cases. -
dev/workflow/setup_buildxml.pyandecflow_suite_factory.pyupdated sosetup_workflow.py --engine ecflowyields a valid GFS forecast-only.def. -
dev/workflow/hosts/gaeac6.yamlone-line default the suite needs. -
.gitlab-ci.ymlchange to exercise the C48_ATM ecFlow load in CI. - The last two commits add
--pslot,--expdir,--comroot, and--stmpCLI overrides toload_ecflow_case.py, so multiple developers can run C48_ATM side-by-side on a shared box (URSA / Gaea-c6) without editing YAML.
flowchart TD
A[AppConfig<br/>experiment config] --> B[EcFlowTasks / GFSEcFlowTasks<br/>+ ecflow_tasks_factory]
B -->|task dicts w/ triggers| C[GFSForecastOnlyEcFlowSuite<br/>gfs_forecast_only_ecflow.py]
C -->|ecFlow Python API| D[.def suite definition]
D --> E[load_ecflow_case.py<br/>load + begin]
E --> F[(ecFlow server<br/>ECF_HOST:ECF_PORT)]
G[dev/ecflow/scripts/*.ecf<br/>+ utils/*.h] -->|ECF_FILES via manifest| F
H[sync_ecf_scripts.sh] -.hot-refresh.-> G
The equivalent Rocoto path is AppConfig → rocoto.tasks + tasks_factory → Rocoto XML. The branch deliberately keeps the two engines symmetric so shared
resource/config logic lives in rocoto.tasks.Tasks (the ecFlow base inherits
from it).
# On URSA / gaeac6, with an ecflow_server already up and ECF_HOST/ECF_PORT set:
python3 dev/workflow/ecflow/c48_atm_ecflow.py \
--pslot my_c48_atm --expdir "$STMP/EXPDIR" --overwriteThis runs setup_expt, setup_workflow (ecflow engine), copies the .ecf
files into EXPDIR/…/ecf_scripts/, writes ecf_scripts.manifest (later used
by sync_ecf_scripts.sh), loads the .def into the server, and begins the
suite.
-
gefs,sfs,gcafstask classes are registered-but-commented — reserved for later increments of epic #4106. - No cycled DA support — forecast-only only.
- No wave / ocean / ice tasks in the C48_ATM DAG (those remain exercised on the Rocoto side by the C48_S2SW cases).
The companion report GFS-Forecast-Only-ecFlow-and-Rocoto-Report documents
that, as of the June 2026 checkout, GFSForecastOnlyEcFlowSuite.get_cycledefs()
and .write() were both pass — an extension point with zero of two
generation hooks implemented. This branch is the direct sequel: it implements
those hooks (the file grows +385/−19), adds the task-dict producers they
consume, and supplies the .ecf payloads the generated .def references.
Reading the two reports together gives the full arc from scaffold (Rahul
Mahajan, Sep 2025) to first working case (Anton Fernando, Sep 2026).
Agent: GitHub Copilot running Claude Opus 4.8 (VS Code). The agent selected
and issued all tool calls; Terry.McGuinness directed the investigation and
reviewed results. Server: eib-mcp-localhost (Docker MCP Gateway via pw
tunnel, tenant gw, branch develop). Git history and direct source were
used as the authoritative cross-check.
The inventory below is complete: every OCA MCP call issued while drafting this report is listed. Per the originating request it carries an evaluation rating (Usefulness), a separate Assessment column explaining that rating, and an inline Recommendation to Improve column for the calls that under-performed.
| # | Tool | Query / Args | Result | Usefulness | Assessment | Recommendation to Improve |
|---|---|---|---|---|---|---|
| 1 | explain_workflow_component |
dev/workflow/ecflow/gfs_forecast_only_ecflow.py, detailed |
Returned forecast-only user-guide docs (setup_expt invocation examples) but nothing about the ecFlow generator internals | Low | Correct file, but the doc context was Rocoto setup, not the ecFlow code; contributed nothing the source did not already show. | Prefer the code-graph symbol summary for a concrete source path; fall back to doc RAG only when no symbol is indexed. The tool matched on the filename's forecast_only tokens and pulled the user guide instead of the class. |
| 2 |
search_issues (global-workflow, all) |
"ecflow C48_ATM suite" | Surfaced #4106 "Dynamically Generate EcFlow Suites" (Epic, open) plus two related PRs | High | Established the strategic "why" (the epic parent) that Git history alone does not show — the single most valuable result in the session. | None needed. Exemplary: returned the framing epic on a natural-language query. |
| 3 | explain_with_context |
"ecFlow suite generation … setup_workflow engine ecflow, dev/ecflow, load_ecflow_case", technical | Returned ecFlow Python-API doc excerpts + the user-guide note that an ecFlow system "is also under development" | Medium | Confirmed the native ecFlow Python API design basis and the documented intent; useful corroboration but not decisive. | Fix whitespace/normalization in ingested doc chunks: excerpts had run-together words ("aPython API", "ecFlowbased systm"). Repair at ingestion time so retrieved text is quotable. |
| 4 | analyze_code_structure |
dev/workflow/ecflow/gfs_forecast_only_ecflow.py, depth 2, deps |
Listed class GFSForecastOnlyEcFlowSuite, 3 methods, 4 imports, Imported By: 0
|
Medium | Corroborated the class structure, but the Imported By: 0 reading is a false-negative (the class is loaded dynamically via factory), so it needed manual correction. |
Detect wxflow.Factory string-name registration and label such classes "dynamically dispatched (factory-registered)" instead of implying they are unused. |
| 5 |
search_issues (global-workflow, open) |
"dynamically generate ecflow suites epic" | Cleanly returned only #4106 | High | Precise epic confirmation on a targeted query; validated result #2 with zero noise. | None needed. |
| — | (all graph/doc tools above) | tenant gw, branch develop
|
Answers reflected develop, but this work lives on feature/ecflow-c48-atm (not ingested) |
Caveat | Not a per-call rating but a cross-cutting risk: the branch under analysis was never ingested, so every structural answer risked being stale. | Add a one-line banner ("graph reflects develop; requested code may be newer") whenever the queried branch differs from the ingested branch. |
Tally: 5 calls — 2 high-value (search_issues #2 and #5), 2 medium, 1 low,
plus 1 cross-cutting branch-staleness caveat. As the Assessment column shows,
the graph/doc tools mostly corroborated structure the source already exposed,
while the issue search delivered the context Git could not — the epic that
frames the whole branch. Net: the OCA MCP was most valuable for provenance and
strategic framing (why this branch exists),
and least valuable for code-internal detail (which direct source read better).
-
dev/workflow/ecflow/ecflow_tasks.py—EcFlowTasksbase and task-dict contract -
dev/workflow/ecflow/gfs_ecflow_tasks.py— per-task GFS definitions (best entry point for adding a task) -
dev/workflow/ecflow/ecflow_tasks_factory.py— engine registration -
dev/workflow/ecflow/gfs_forecast_only_ecflow.py— forecast-only suite generator (the filled-in scaffold) -
dev/workflow/ecflow/load_ecflow_case.py— generic case loader -
dev/workflow/ecflow/c48_atm_ecflow.py— C48_ATM entry point -
dev/ecflow/scripts/*.ecf,dev/ecflow/utils/*.h— script + header library - NOAA-EMC #4106 — umbrella epic
- Companion: GFS-Forecast-Only-ecFlow-and-Rocoto-Report