GFS Forecast Only ecFlow and Rocoto Report - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki
Prepared with contributions from Asta.
Report date: 2026-09-08.
The forecast-only ecFlow module is an architectural scaffold,
not an implemented forecast-only ecFlow suite generator. Its constructor
delegates to the shared configuration layer, but both get_cycledefs() and
write() contain only pass. Calling either method returns None; neither
creates cycles, tasks, dependencies, or workflow files.
Rahul Mahajan introduced the file on September 17, 2025, at 15:25:58 -0400,
in commit 490a0a45415eaeedfa0d498cb3c1cff2dac7a891, associated with PR
#4071, "Make room for generating ecflow suites." The file has no subsequent
commits in the inspected checkout's git log --follow history. This is the
recorded Git author date, not proof of when the author first drafted the code.
The introducing commit explicitly describes parallel ecFlow and Rocoto
backends, shared attributes moved into workflow_suite.py, and preservation
of existing Rocoto functionality. The intended relationship is therefore
alternative workflow-engine backends sharing experiment configuration, not
two schedulers jointly controlling the same tasks.
- Inspected checkout:
supported_repos/global-workflow_developat commit6703c6973039aafc08f049fb593675f9feb6a91c(detached HEAD), dated 2026-06-29, with commit subject "Fix documentation errors (#5072)." Upstream changes after that revision were not assessed. - The source checkout has local CI-file changes, but none were reported in the workflow files discussed here. No source files were modified.
- Findings describe this local revision, not necessarily the latest upstream branch or an independently deployed NCO production workflow.
- EIB MCP graph analysis was requested with tenant
gw, branchdevelop. This session exposes EIB tools through its gateway connection, not a separately selectableeib-mcp-rag-fullconnection. Direct source and Git history are authoritative for implementation and provenance claims. - The MCP graph recognized
EcFlowSuiteinheritance and the three methods, but returned no callers or callees for the class. That absence is not proof that the class is unused.
| Evidence | Recorded value |
|---|---|
| Introducing author | Rahul Mahajan |
| Author date | 2025-09-17 15:25:58 -0400 |
| Committer | GitHub |
| Commit date | 2025-09-17 15:25:58 -0400 |
| Commit | 490a0a45415eaeedfa0d498cb3c1cff2dac7a891 |
| Pull request | #4071, "Make room for generating ecflow suites" |
| File size at introduction | 49 added lines |
| Subsequent file-history entries at this revision | None |
The introducing change touched 38 files. In addition to creating this class,
it introduced EcFlowSuite, an ecFlow factory, and WorkflowSuite; renamed
the Rocoto base module to rocoto_xml.py; moved older helpers from ecFlow/
to ecflow/; and adjusted workflow setup and CI configuration. The commit
message expressly says that current Rocoto functionality is maintained.
That scope supports reading this file as an extension point, not as a
completed ecFlow implementation.
| Element | Implementation | Consequence |
|---|---|---|
| Module and class documentation | Present | States the intended forecast-only purpose |
__init__(app_config, ecflow_config) |
Calls the parent constructor | Uses existing shared initialization |
get_cycledefs() |
pass |
No cycle definitions; returns None
|
write() |
pass |
No suite output; returns None
|
| ecFlow task and dependency construction | Absent from this class | No executable task graph is generated here |
to_timedelta, timedelta_to_HMS imports |
Unused in this file | Imports do not implement cycle scheduling |
There are two meaningful workflow-generation hooks and zero of the two is implemented. This is a precise hook-level measure, not an estimate that all surrounding ecFlow work is 0% complete. Constructor/configuration plumbing does exist, and other ecFlow helper modules must be assessed separately.
The inheritance chain is:
WorkflowSuite -> EcFlowSuite -> GFSForecastOnlyEcFlowSuite
EcFlowSuite only forwards initialization to WorkflowSuite. The latter
validates workflow_engine as rocoto or ecflow, stores the appropriate
configuration dictionary, selects the first application's base configuration,
converts GDAS/GFS intervals to timedeltas, resolves host information, and stores
ACCOUNT, HOMEglobal, EXPDIR, and PSLOT context.
WorkflowSuite declares get_cycledefs() and write() abstract. The target
class overrides both with concrete no-op methods. Consequently, the abstract
interface alone does not enforce actual suite generation: after successful
configuration initialization, a call to write() can return normally without
producing a suite. There is no fail-fast NotImplementedError in these hooks.
The controlling code is the workflow setup entry point. It:
- Accepts the experiment directory and a required
rocotoorecflowsubcommand. - Reads the experiment configuration, verifies that the requested experiment
directory matches
EXPDIR, and checks writable working directories. - Uses
NETandMODEto select an application configuration throughapp_config_factory. Forgfs_forecast-only, this isGFSForecastOnlyAppConfig. - Builds a workflow configuration dictionary. Both engines receive their name and verbosity; Rocoto additionally receives retry and throttle values.
- Selects exactly one engine factory and creates the matching generator.
- Calls that generator's
write()method without checking for an output file.
flowchart TD
A[Experiment config files] --> B[setup_workflow.py]
B --> C[GFSForecastOnlyAppConfig]
C --> D{Selected engine}
D -->|rocoto| E[GFSForecastOnlyRocotoXML]
D -->|ecflow| F[GFSForecastOnlyEcFlowSuite]
E --> G[RocotoXML: cycles, tasks, dependencies, XML]
G --> H[Experiment XML and optional cron files]
H --> I[rocotorun and its state database]
F --> J[Shared configuration initialization]
J --> K[write returns None: no suite output]
The ecFlow factory actively registers only
gfs_forecast-only. Other advertised application registrations are comments,
not implemented registrations. The Rocoto factory maps the
same GFS key to GFSForecastOnlyRocotoXML.
"In tandem" means common inputs and corresponding backend implementations. It does not mean Rocoto first generates XML that this ecFlow class converts, nor that both schedulers run together. There is no XML-to-ecFlow conversion, cross-scheduler state synchronization, or ecFlow-server deployment in the target class or its current setup path.
AppConfig finalizes run options, task names, and sourced per-run configuration. The GFS forecast-only application selects tasks according to configuration flags. Its conceptual ordering is:
optional fetch -> stage_ic -> optional aerosol/wave initialization -> fcst
-> selected atmosphere/ocean/ice/wave products and verification
-> optional archive transfers -> arch_vrfy -> cleanup
This is a task-selection outline, not a strictly serial execution graph. Dependencies determine actual readiness and concurrency. Important options include fetching initial conditions, warm versus cold starts, atmosphere, ocean, ice, waves, inline versus offline UPP, aerosol forecasts, METplus, tracking, sounding products, GEMPAK/AWIPS, and archiving.
Rocoto then calls get_wf_tasks(), which selects GFSTasks
through the task factory and asks each selected task for its
XML. Tasks supplies environment and resource handling;
GFSTasks supplies task-specific dependencies and execution details.
Those task builders already contain Rocoto XML construction calls. They are not a scheduler-independent graph that ecFlow can consume unchanged. The new ecFlow branch does not call them or translate their results. The shared base classes provide configuration reuse, not full task-graph parity.
GFSForecastOnlyRocotoXML.get_cycledefs() implements:
| Group | Current Rocoto rule | Why it matters |
|---|---|---|
gfs |
SDATE_GFS through EDATE, at INTERVAL_GFS
|
Defines forecast initialization cycles |
gfs_seq |
Starts one GFS interval after SDATE_GFS; omitted if beyond EDATE
|
Supplies cycles after the initial one, including aerosol initialization that needs prior-cycle restarts |
metp |
Present only when DO_METP is enabled |
Separates verification scheduling from forecast scheduling |
For metp, a GFS interval below 24 hours produces a 24-hour verification
schedule by replacing the start/end hours with 18. At intervals of 24 hours
or more, it reuses the GFS schedule. The implementation normalizes both
endpoint hours; it does not clamp the resulting verification times to the
original forecast endpoint hours.
For example, with SDATE_GFS=2026030100, EDATE=2026030200, a six-hour
interval, and DO_METP=True, the cycle definitions are:
<cycledef group="gfs">202603010000 202603020000 06:00:00</cycledef>
<cycledef group="gfs_seq">202603010600 202603020000 06:00:00</cycledef>
<cycledef group="metp">202603011800 202603021800 24:00:00</cycledef>An ecFlow implementation would need to express equivalent cycle identities
and dependencies using its own suite/family/task and scheduling mechanisms.
Copying the XML strings into get_cycledefs() would not accomplish that.
The scaffold does not yet establish the ecFlow hook's output representation.
Examples from GFSTasks show the required semantic detail:
-
stage_icwaits forfetchwhen remote/local fetching is enabled. - The forecast-only forecast waits for staged initial conditions and adds configured wave and aerosol prerequisites.
- Forecast segments become a serial metatask, with
FCST_SEGMENTpassed to each segment. The number of GFS segments comes fromFCST_SEGMENTS. - Aerosol initialization uses
gfs_seqand previous-cycle restart data; the forecast dependency includes a first-cycle exception. - Offline UPP can begin when per-hour atmospheric, surface, and log files satisfy readiness checks, including file-age thresholds. It need not wait for the entire forecast to finish.
- Product tasks group forecast hours, account for segment boundaries, and can use file readiness or forecast-metatask completion as alternatives.
These are concrete parity requirements, not evidence that all optional Rocoto configurations have been runtime-validated in this analysis.
RocotoXML assembles the XML during construction: preamble,
entities, scheduler/throttle header, cycle definitions, generated tasks, and
footer. Its inherited write() then writes the experiment XML and attempts
to write the periodic Rocoto driver configuration.
| Artifact or operation | Rocoto behavior | Current forecast-only ecFlow class |
|---|---|---|
| Workflow definition | Writes EXPDIR/PSLOT.xml
|
Writes nothing |
| Runtime state | Generated driver command references EXPDIR/PSLOT.db; Rocoto manages it at runtime |
No ecFlow state/server connection |
| Periodic driver | Writes PSLOT.crontab when rocotorun is available; default interval is five minutes |
None |
| Slurm cron support | Can also write executable PSLOT.scron.sh and validate Rocoto configuration |
None |
| Archive-server scheduling | Additional server cron file for configured Globus/HPSS archiving | None |
| Installation/launch | Writing cron files does not itself install them or run the forecast | No suite loading or launch |
Example current CLI selection, after preparing a valid experiment and the required host environment, from the workflow setup directory:
./setup_workflow.py /path/to/experiment rocoto
./setup_workflow.py /path/to/experiment ecflowThe second command demonstrates selection syntax only. It is not a working ecFlow generation recipe at this revision. Earlier configuration or host checks can still fail; if they succeed, this backend reaches the no-op writer.
| Date | Evidence | Significance |
|---|---|---|
| 2022-07-18 | Commit aa2542eb4c95827bc1fc7a4a76d4d0f5bc701f74, PR #912 |
Added the older ecFlow generator to develop
|
| 2022-07-19 | Commit 5c9639dba419ef51608aa8962b54ae69c8ad8c60, PR #916 |
Consolidated generator locations but explicitly said the engines remained separate and required separate inputs |
| 2025-09-17 | Commit 490a0a45415eaeedfa0d498cb3c1cff2dac7a891, PR #4071 |
Added the target scaffold and current shared-base/factory architecture |
| 2026-06-29 | Inspected checkout revision | Target still contains its original two no-op hooks |
The older setup script, YAML setup helper, and definition builder credit Kyle Nevins in their source headers. The 2022 integration commits above are Git-authored by Rahul Mahajan. Those are different forms of attribution: Kyle's existing generator work must not be mistaken for authorship of the 2025 target file, or vice versa.
Immediately before PR #4071, the main workflow setup script explicitly raised
NotImplementedError when ecFlow was selected. The introducing commit replaced
that rejection with factory dispatch. Consequently, the current route is
better connected architecturally, but no longer fails explicitly at engine
selection to announce that generation is unimplemented.
The legacy entry point constructs Ecflowsetup(args, envconfigs), invokes
generate_workflow(), then save(). Its input contract includes:
- A user-supplied ecFlow YAML definition, selected with
--ecflow-config. - Experiment
config.base, selected with--expdir. - A definition-output directory, selected with
--savedir. -
ECFgfsand an optional script repository; the helper defaults the latter toECFgfs/scriptsif no override exists.
Implemented helper code loads YAML, substitutes environment/configuration
values, expands suite names, constructs families/tasks, applies edits and
repeat/time/status attributes, and adds triggers/events after nodes exist.
It places the resulting suites in an ecFlow Defs object and saves
ecflow_suite.def. It also includes a Defs.check() call in its diagnostic
print() method; the ordinary legacy save() path does not itself call that
diagnostic method.
The definition builder creates directory trees and can copy an existing
task-named or template-named .ecf script into the appropriate suite/family
directory. That is script deployment, not automatic synthesis of a modern GFS
dependency graph from AppConfig or conversion of Rocoto XML.
- The new class does not import or invoke
Ecflowsetup. - The current
ecflowCLI subparser exposes verbosity, not the legacy--ecflow-configand--savedirinputs. There is no adapter between the application configuration and the legacy YAML contract. - The legacy entry point and setup helper retain
ecFlow.*imports although PR #4071 moved the directory to lowercaseecflow/. The checkout's local package search finds no uppercase package at the workflow root. - The local lowercase
ecflowpackage also shares its name with the external ecFlow Python bindings. With the workflow directory first on the module search path, the local package wins. Integration must resolve this naming conflict before relying on imports such asfrom ecflow import Defs. - The legacy entry point also imports
Configurationfrom an old top-level module name, whereas the current setup path imports it fromwxflow. Compatibility for that old import was not established here.
These are source-level integration gaps. The legacy generator was not run against an installed ecFlow binding or server, and this report does not certify that its remaining code is otherwise correct.
The legacy README describes its status as of June 2022. Its claim that "this application only works for ecFlow" applies to that older utility, not to the current GFS factory-based interface or the entire global-workflow system.
The Rocoto forecast job card loads the development runtime
modules and invokes the development forecast J-job. That
J-job sources configuration, prepares forecast environment and paths, and
invokes FORECASTSH, defaulting to exglobal_forecast.sh under SCRglobal.
The existing ecFlow forecast wrapper invokes
${HOMEgfs}/jobs/JGLOBAL_FORECAST. It uses PBS directives, ecFlow %...%
substitutions, and shared ecFlow headers. The header supplies
client connection variables, initialization, and failure trapping;
the tail reports completion.
Rocoto XML -> Rocoto readiness/batch submission -> development job card
-> development JGLOBAL_FORECAST -> forecast execution script
ecFlow definition -> ecFlow readiness/batch submission -> .ecf wrapper
-> deployed JGLOBAL_FORECAST -> forecast execution script
The second line describes the existing ecFlow wrapper architecture, not output generated by the target scaffold. The development and deployed J-job paths differ. The ecFlow template has operations-specific PBS resources, module loading, and environment assumptions, so it is not automatically a portable replacement for the Rocoto job card on every supported host.
For a future side-by-side comparison, use separate experiment identifiers, output/work directories, and scheduler state. Running both engines against the same experiment outputs could duplicate execution or race on products and cleanup. No coordination mechanism was found in the inspected setup code.
| Capability | Assessment at the inspected revision |
|---|---|
| Shared experiment parsing and application selection | Implemented |
| Common workflow configuration base | Implemented |
| GFS forecast-only ecFlow factory selection | Implemented |
| New class's cycle generation | Not implemented |
| New class's task/dependency/resource generation | Not implemented |
| New class's definition/script output | Not implemented |
| New class's ecFlow validation/deployment | Not implemented |
| Older YAML-based ecFlow building blocks | Substantial code exists, but is not integrated into this class |
| Rocoto generation and runtime-driver file creation | Implemented; not an end-to-end runtime certification |
| Demonstrated modern ecFlow/Rocoto parity | Not established |
There is a C48 atmosphere ecFlow CI case, but it skips nine named
hosts: WCOSS2, Hera, Ursa, Gaea C6, Orion, Hercules, Derecho,
awsepicglobalworkflow, and aws-ec2. Its workflow.engine: ecflow setting
demonstrates configuration intent, not successful generation or execution.
No CI results demonstrating this backend's correctness were examined.
Recommended future work, not implemented by this documentation task:
- Restore an explicit unsupported-backend error until usable output exists, or add a verified output requirement so a no-op cannot look successful.
- Specify the initial supported application/host scope and the ecFlow cycle representation before completing the two hooks.
- Decide whether to adapt the YAML helper or build a new renderer, resolving legacy imports and external-binding package resolution first.
- Preserve a common task/configuration source. If dependency rules are to be reused by both engines, separate those rules from Rocoto XML emission; do not assume the current task builders are already engine-neutral.
- Implement matching first/subsequent-cycle behavior, forecast segmentation, file readiness, optional tasks, resources, retries, and concurrency limits.
- Produce and validate definitions and
.ecfscripts, then separately handle server loading and execution. Definition generation alone is not deployment. - Add focused parity tests and an enabled small-host CI case before claiming readiness: one cycle, multiple cycles, METplus cadence, segmented forecast, disabled optional tasks, missing input files, and abort/retry behavior.
Practical recommendation: use the Rocoto path for forecast-only experiment generation at this revision. Treat the target as the designated location for future ecFlow implementation, not as an available alternate scheduler backend.
- Git history established the target's introducing author/date and lack of subsequent edits in this checkout. Historical commit contents established the older explicit rejection and the intended backend separation.
- EIB MCP
find_callers_callees,analyze_code_structure, andsearch_documentationsupplied graph and documentation context. The graph's missing importer/caller results were contradicted by the directly inspected factory import; they were not treated as evidence that the class is unused. - Isolated Python checks verified the two no-op hooks, concrete subclass
instantiation, and the absence of generated files after
write(). Host and application dependencies were controlled substitutes, not real HPC setup. - Isolated Rocoto cycle-method checks covered a single cycle, subsequent cycles, subdaily METplus scheduling, and a daily interval. XML fragments were parsed and compared with expected groups/timestamps using controlled timedelta helpers; this did not validate the full Rocoto environment.
- Local module-resolution checks verified the legacy capitalization gap and lowercase package collision under a workflow-root-first search path.
- Source citation paths and line anchors were checked against the checkout.
- No packages were installed, source modules edited, upstream updates fetched, cron entries installed, suites loaded, or forecast jobs submitted. Full ecFlow/Rocoto execution and post-June-29 upstream status remain unverified.
All source links below are pinned to the inspected revision so that this report remains meaningful when published to the wiki.
| Source | Primary evidence |
|---|---|
| Target forecast-only ecFlow class | Constructor and unimplemented hooks |
| Shared workflow base | Common configuration and abstract interface |
| ecFlow base | Initialization forwarding only |
| Workflow setup entry point | CLI selection, factories, unconditional write call |
| Application factory | GFS application registration |
| ecFlow factory | Only active ecFlow application registration |
| Rocoto factory | Corresponding Rocoto registration |
| Application base | Finalization and run options |
| GFS forecast-only application | Selected configurations and task list |
| Rocoto forecast-only cycles | GFS, sequential, and verification cycle rules |
| Rocoto base | XML assembly, output, and cron handling |
| Workflow task traversal | Application tasks to backend task builders |
| Task factory | GFS task builder selection |
| Common Rocoto task support | Environment, resources, and grouping support |
| GFS task builders | Forecast segmentation and readiness/dependency logic |
| Legacy ecFlow entry point | Separate YAML-driven interface |
| Legacy ecFlow setup helper | Definition assembly and saving |
| Legacy ecFlow definition builder | ecFlow objects and script copying |
| Legacy README | Historical design and input documentation |
| Rocoto forecast wrapper | Development J-job invocation |
| Development forecast J-job | Shared forecast payload boundary |
| ecFlow forecast wrapper | Operations-style J-job invocation |
| ecFlow header | ecFlow initialization and failure reporting |
| ecFlow tail | ecFlow completion reporting |
| ecFlow CI case | Declared case and host exclusions |
Historical references: PR #4071, introducing commit, PR #912, and PR #916. The attribution and intent above were read from local Git objects; online pull-request discussion and review threads were not independently retrieved.