GFS Forecast Only ecFlow and Rocoto Report - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

GFS Forecast-Only ecFlow Generator and Its Relationship to Rocoto

Prepared with contributions from Asta.

Report date: 2026-09-08.

Executive Summary

The forecast-only ecFlow module is an architectural scaffold, not an implemented forecast-only ecFlow suite generator. Its constructor delegates to the shared configuration layer, but both get_cycledefs() and write() contain only pass. Calling either method returns None; neither creates cycles, tasks, dependencies, or workflow files.

Rahul Mahajan introduced the file on September 17, 2025, at 15:25:58 -0400, in commit 490a0a45415eaeedfa0d498cb3c1cff2dac7a891, associated with PR #4071, "Make room for generating ecflow suites." The file has no subsequent commits in the inspected checkout's git log --follow history. This is the recorded Git author date, not proof of when the author first drafted the code.

The introducing commit explicitly describes parallel ecFlow and Rocoto backends, shared attributes moved into workflow_suite.py, and preservation of existing Rocoto functionality. The intended relationship is therefore alternative workflow-engine backends sharing experiment configuration, not two schedulers jointly controlling the same tasks.

Scope and Evidence

  • Inspected checkout: supported_repos/global-workflow_develop at commit 6703c6973039aafc08f049fb593675f9feb6a91c (detached HEAD), dated 2026-06-29, with commit subject "Fix documentation errors (#5072)." Upstream changes after that revision were not assessed.
  • The source checkout has local CI-file changes, but none were reported in the workflow files discussed here. No source files were modified.
  • Findings describe this local revision, not necessarily the latest upstream branch or an independently deployed NCO production workflow.
  • EIB MCP graph analysis was requested with tenant gw, branch develop. This session exposes EIB tools through its gateway connection, not a separately selectable eib-mcp-rag-full connection. Direct source and Git history are authoritative for implementation and provenance claims.
  • The MCP graph recognized EcFlowSuite inheritance and the three methods, but returned no callers or callees for the class. That absence is not proof that the class is unused.

Authorship and Original Intent

Evidence Recorded value
Introducing author Rahul Mahajan
Author date 2025-09-17 15:25:58 -0400
Committer GitHub
Commit date 2025-09-17 15:25:58 -0400
Commit 490a0a45415eaeedfa0d498cb3c1cff2dac7a891
Pull request #4071, "Make room for generating ecflow suites"
File size at introduction 49 added lines
Subsequent file-history entries at this revision None

The introducing change touched 38 files. In addition to creating this class, it introduced EcFlowSuite, an ecFlow factory, and WorkflowSuite; renamed the Rocoto base module to rocoto_xml.py; moved older helpers from ecFlow/ to ecflow/; and adjusted workflow setup and CI configuration. The commit message expressly says that current Rocoto functionality is maintained. That scope supports reading this file as an extension point, not as a completed ecFlow implementation.

What the Target File Implements

Element Implementation Consequence
Module and class documentation Present States the intended forecast-only purpose
__init__(app_config, ecflow_config) Calls the parent constructor Uses existing shared initialization
get_cycledefs() pass No cycle definitions; returns None
write() pass No suite output; returns None
ecFlow task and dependency construction Absent from this class No executable task graph is generated here
to_timedelta, timedelta_to_HMS imports Unused in this file Imports do not implement cycle scheduling

There are two meaningful workflow-generation hooks and zero of the two is implemented. This is a precise hook-level measure, not an estimate that all surrounding ecFlow work is 0% complete. Constructor/configuration plumbing does exist, and other ecFlow helper modules must be assessed separately.

Shared Initialization

The inheritance chain is:

WorkflowSuite -> EcFlowSuite -> GFSForecastOnlyEcFlowSuite

EcFlowSuite only forwards initialization to WorkflowSuite. The latter validates workflow_engine as rocoto or ecflow, stores the appropriate configuration dictionary, selects the first application's base configuration, converts GDAS/GFS intervals to timedeltas, resolves host information, and stores ACCOUNT, HOMEglobal, EXPDIR, and PSLOT context.

WorkflowSuite declares get_cycledefs() and write() abstract. The target class overrides both with concrete no-op methods. Consequently, the abstract interface alone does not enforce actual suite generation: after successful configuration initialization, a call to write() can return normally without producing a suite. There is no fail-fast NotImplementedError in these hooks.

How the Current Generators Fit Together

One Entry Point, Two Alternative Backends

The controlling code is the workflow setup entry point. It:

  1. Accepts the experiment directory and a required rocoto or ecflow subcommand.
  2. Reads the experiment configuration, verifies that the requested experiment directory matches EXPDIR, and checks writable working directories.
  3. Uses NET and MODE to select an application configuration through app_config_factory. For gfs_forecast-only, this is GFSForecastOnlyAppConfig.
  4. Builds a workflow configuration dictionary. Both engines receive their name and verbosity; Rocoto additionally receives retry and throttle values.
  5. Selects exactly one engine factory and creates the matching generator.
  6. Calls that generator's write() method without checking for an output file.
flowchart TD
   A[Experiment config files] --> B[setup_workflow.py]
   B --> C[GFSForecastOnlyAppConfig]
   C --> D{Selected engine}
   D -->|rocoto| E[GFSForecastOnlyRocotoXML]
   D -->|ecflow| F[GFSForecastOnlyEcFlowSuite]
   E --> G[RocotoXML: cycles, tasks, dependencies, XML]
   G --> H[Experiment XML and optional cron files]
   H --> I[rocotorun and its state database]
   F --> J[Shared configuration initialization]
   J --> K[write returns None: no suite output]
Loading

The ecFlow factory actively registers only gfs_forecast-only. Other advertised application registrations are comments, not implemented registrations. The Rocoto factory maps the same GFS key to GFSForecastOnlyRocotoXML.

"In tandem" means common inputs and corresponding backend implementations. It does not mean Rocoto first generates XML that this ecFlow class converts, nor that both schedulers run together. There is no XML-to-ecFlow conversion, cross-scheduler state synchronization, or ecFlow-server deployment in the target class or its current setup path.

Shared Application Configuration Is Not a Shared Dependency Graph

AppConfig finalizes run options, task names, and sourced per-run configuration. The GFS forecast-only application selects tasks according to configuration flags. Its conceptual ordering is:

optional fetch -> stage_ic -> optional aerosol/wave initialization -> fcst
   -> selected atmosphere/ocean/ice/wave products and verification
   -> optional archive transfers -> arch_vrfy -> cleanup

This is a task-selection outline, not a strictly serial execution graph. Dependencies determine actual readiness and concurrency. Important options include fetching initial conditions, warm versus cold starts, atmosphere, ocean, ice, waves, inline versus offline UPP, aerosol forecasts, METplus, tracking, sounding products, GEMPAK/AWIPS, and archiving.

Rocoto then calls get_wf_tasks(), which selects GFSTasks through the task factory and asks each selected task for its XML. Tasks supplies environment and resource handling; GFSTasks supplies task-specific dependencies and execution details.

Those task builders already contain Rocoto XML construction calls. They are not a scheduler-independent graph that ecFlow can consume unchanged. The new ecFlow branch does not call them or translate their results. The shared base classes provide configuration reuse, not full task-graph parity.

What Rocoto Already Implements

Cycle Semantics the ecFlow Backend Would Need to Preserve

GFSForecastOnlyRocotoXML.get_cycledefs() implements:

Group Current Rocoto rule Why it matters
gfs SDATE_GFS through EDATE, at INTERVAL_GFS Defines forecast initialization cycles
gfs_seq Starts one GFS interval after SDATE_GFS; omitted if beyond EDATE Supplies cycles after the initial one, including aerosol initialization that needs prior-cycle restarts
metp Present only when DO_METP is enabled Separates verification scheduling from forecast scheduling

For metp, a GFS interval below 24 hours produces a 24-hour verification schedule by replacing the start/end hours with 18. At intervals of 24 hours or more, it reuses the GFS schedule. The implementation normalizes both endpoint hours; it does not clamp the resulting verification times to the original forecast endpoint hours.

For example, with SDATE_GFS=2026030100, EDATE=2026030200, a six-hour interval, and DO_METP=True, the cycle definitions are:

<cycledef group="gfs">202603010000 202603020000 06:00:00</cycledef>
<cycledef group="gfs_seq">202603010600 202603020000 06:00:00</cycledef>
<cycledef group="metp">202603011800 202603021800 24:00:00</cycledef>

An ecFlow implementation would need to express equivalent cycle identities and dependencies using its own suite/family/task and scheduling mechanisms. Copying the XML strings into get_cycledefs() would not accomplish that. The scaffold does not yet establish the ecFlow hook's output representation.

Task Readiness Is More Than a List of Job Names

Examples from GFSTasks show the required semantic detail:

  • stage_ic waits for fetch when remote/local fetching is enabled.
  • The forecast-only forecast waits for staged initial conditions and adds configured wave and aerosol prerequisites.
  • Forecast segments become a serial metatask, with FCST_SEGMENT passed to each segment. The number of GFS segments comes from FCST_SEGMENTS.
  • Aerosol initialization uses gfs_seq and previous-cycle restart data; the forecast dependency includes a first-cycle exception.
  • Offline UPP can begin when per-hour atmospheric, surface, and log files satisfy readiness checks, including file-age thresholds. It need not wait for the entire forecast to finish.
  • Product tasks group forecast hours, account for segment boundaries, and can use file readiness or forecast-metatask completion as alternatives.

These are concrete parity requirements, not evidence that all optional Rocoto configurations have been runtime-validated in this analysis.

Output and Execution Artifacts

RocotoXML assembles the XML during construction: preamble, entities, scheduler/throttle header, cycle definitions, generated tasks, and footer. Its inherited write() then writes the experiment XML and attempts to write the periodic Rocoto driver configuration.

Artifact or operation Rocoto behavior Current forecast-only ecFlow class
Workflow definition Writes EXPDIR/PSLOT.xml Writes nothing
Runtime state Generated driver command references EXPDIR/PSLOT.db; Rocoto manages it at runtime No ecFlow state/server connection
Periodic driver Writes PSLOT.crontab when rocotorun is available; default interval is five minutes None
Slurm cron support Can also write executable PSLOT.scron.sh and validate Rocoto configuration None
Archive-server scheduling Additional server cron file for configured Globus/HPSS archiving None
Installation/launch Writing cron files does not itself install them or run the forecast No suite loading or launch

Example current CLI selection, after preparing a valid experiment and the required host environment, from the workflow setup directory:

./setup_workflow.py /path/to/experiment rocoto
./setup_workflow.py /path/to/experiment ecflow

The second command demonstrates selection syntax only. It is not a working ecFlow generation recipe at this revision. Earlier configuration or host checks can still fail; if they succeed, this backend reaches the no-op writer.

Existing ecFlow Work Outside This Class

Historical Timeline and Attribution

Date Evidence Significance
2022-07-18 Commit aa2542eb4c95827bc1fc7a4a76d4d0f5bc701f74, PR #912 Added the older ecFlow generator to develop
2022-07-19 Commit 5c9639dba419ef51608aa8962b54ae69c8ad8c60, PR #916 Consolidated generator locations but explicitly said the engines remained separate and required separate inputs
2025-09-17 Commit 490a0a45415eaeedfa0d498cb3c1cff2dac7a891, PR #4071 Added the target scaffold and current shared-base/factory architecture
2026-06-29 Inspected checkout revision Target still contains its original two no-op hooks

The older setup script, YAML setup helper, and definition builder credit Kyle Nevins in their source headers. The 2022 integration commits above are Git-authored by Rahul Mahajan. Those are different forms of attribution: Kyle's existing generator work must not be mistaken for authorship of the 2025 target file, or vice versa.

Immediately before PR #4071, the main workflow setup script explicitly raised NotImplementedError when ecFlow was selected. The introducing commit replaced that rejection with factory dispatch. Consequently, the current route is better connected architecturally, but no longer fails explicitly at engine selection to announce that generation is unimplemented.

The Separate YAML Generator

The legacy entry point constructs Ecflowsetup(args, envconfigs), invokes generate_workflow(), then save(). Its input contract includes:

  • A user-supplied ecFlow YAML definition, selected with --ecflow-config.
  • Experiment config.base, selected with --expdir.
  • A definition-output directory, selected with --savedir.
  • ECFgfs and an optional script repository; the helper defaults the latter to ECFgfs/scripts if no override exists.

Implemented helper code loads YAML, substitutes environment/configuration values, expands suite names, constructs families/tasks, applies edits and repeat/time/status attributes, and adds triggers/events after nodes exist. It places the resulting suites in an ecFlow Defs object and saves ecflow_suite.def. It also includes a Defs.check() call in its diagnostic print() method; the ordinary legacy save() path does not itself call that diagnostic method.

The definition builder creates directory trees and can copy an existing task-named or template-named .ecf script into the appropriate suite/family directory. That is script deployment, not automatic synthesis of a modern GFS dependency graph from AppConfig or conversion of Rocoto XML.

Why This Is Not a Drop-In Completion

  1. The new class does not import or invoke Ecflowsetup.
  2. The current ecflow CLI subparser exposes verbosity, not the legacy --ecflow-config and --savedir inputs. There is no adapter between the application configuration and the legacy YAML contract.
  3. The legacy entry point and setup helper retain ecFlow.* imports although PR #4071 moved the directory to lowercase ecflow/. The checkout's local package search finds no uppercase package at the workflow root.
  4. The local lowercase ecflow package also shares its name with the external ecFlow Python bindings. With the workflow directory first on the module search path, the local package wins. Integration must resolve this naming conflict before relying on imports such as from ecflow import Defs.
  5. The legacy entry point also imports Configuration from an old top-level module name, whereas the current setup path imports it from wxflow. Compatibility for that old import was not established here.

These are source-level integration gaps. The legacy generator was not run against an installed ecFlow binding or server, and this report does not certify that its remaining code is otherwise correct.

The legacy README describes its status as of June 2022. Its claim that "this application only works for ecFlow" applies to that older utility, not to the current GFS factory-based interface or the entire global-workflow system.

Shared Job Payloads, Different Scheduler Wrappers

The Rocoto forecast job card loads the development runtime modules and invokes the development forecast J-job. That J-job sources configuration, prepares forecast environment and paths, and invokes FORECASTSH, defaulting to exglobal_forecast.sh under SCRglobal.

The existing ecFlow forecast wrapper invokes ${HOMEgfs}/jobs/JGLOBAL_FORECAST. It uses PBS directives, ecFlow %...% substitutions, and shared ecFlow headers. The header supplies client connection variables, initialization, and failure trapping; the tail reports completion.

Rocoto XML -> Rocoto readiness/batch submission -> development job card
        -> development JGLOBAL_FORECAST -> forecast execution script

ecFlow definition -> ecFlow readiness/batch submission -> .ecf wrapper
            -> deployed JGLOBAL_FORECAST -> forecast execution script

The second line describes the existing ecFlow wrapper architecture, not output generated by the target scaffold. The development and deployed J-job paths differ. The ecFlow template has operations-specific PBS resources, module loading, and environment assumptions, so it is not automatically a portable replacement for the Rocoto job card on every supported host.

For a future side-by-side comparison, use separate experiment identifiers, output/work directories, and scheduler state. Running both engines against the same experiment outputs could duplicate execution or race on products and cleanup. No coordination mechanism was found in the inspected setup code.

Implementation Assessment and Next Steps

Capability Assessment at the inspected revision
Shared experiment parsing and application selection Implemented
Common workflow configuration base Implemented
GFS forecast-only ecFlow factory selection Implemented
New class's cycle generation Not implemented
New class's task/dependency/resource generation Not implemented
New class's definition/script output Not implemented
New class's ecFlow validation/deployment Not implemented
Older YAML-based ecFlow building blocks Substantial code exists, but is not integrated into this class
Rocoto generation and runtime-driver file creation Implemented; not an end-to-end runtime certification
Demonstrated modern ecFlow/Rocoto parity Not established

There is a C48 atmosphere ecFlow CI case, but it skips nine named hosts: WCOSS2, Hera, Ursa, Gaea C6, Orion, Hercules, Derecho, awsepicglobalworkflow, and aws-ec2. Its workflow.engine: ecflow setting demonstrates configuration intent, not successful generation or execution. No CI results demonstrating this backend's correctness were examined.

Recommended future work, not implemented by this documentation task:

  1. Restore an explicit unsupported-backend error until usable output exists, or add a verified output requirement so a no-op cannot look successful.
  2. Specify the initial supported application/host scope and the ecFlow cycle representation before completing the two hooks.
  3. Decide whether to adapt the YAML helper or build a new renderer, resolving legacy imports and external-binding package resolution first.
  4. Preserve a common task/configuration source. If dependency rules are to be reused by both engines, separate those rules from Rocoto XML emission; do not assume the current task builders are already engine-neutral.
  5. Implement matching first/subsequent-cycle behavior, forecast segmentation, file readiness, optional tasks, resources, retries, and concurrency limits.
  6. Produce and validate definitions and .ecf scripts, then separately handle server loading and execution. Definition generation alone is not deployment.
  7. Add focused parity tests and an enabled small-host CI case before claiming readiness: one cycle, multiple cycles, METplus cadence, segmented forecast, disabled optional tasks, missing input files, and abort/retry behavior.

Practical recommendation: use the Rocoto path for forecast-only experiment generation at this revision. Treat the target as the designated location for future ecFlow implementation, not as an available alternate scheduler backend.

Validation and Limitations

  • Git history established the target's introducing author/date and lack of subsequent edits in this checkout. Historical commit contents established the older explicit rejection and the intended backend separation.
  • EIB MCP find_callers_callees, analyze_code_structure, and search_documentation supplied graph and documentation context. The graph's missing importer/caller results were contradicted by the directly inspected factory import; they were not treated as evidence that the class is unused.
  • Isolated Python checks verified the two no-op hooks, concrete subclass instantiation, and the absence of generated files after write(). Host and application dependencies were controlled substitutes, not real HPC setup.
  • Isolated Rocoto cycle-method checks covered a single cycle, subsequent cycles, subdaily METplus scheduling, and a daily interval. XML fragments were parsed and compared with expected groups/timestamps using controlled timedelta helpers; this did not validate the full Rocoto environment.
  • Local module-resolution checks verified the legacy capitalization gap and lowercase package collision under a workflow-root-first search path.
  • Source citation paths and line anchors were checked against the checkout.
  • No packages were installed, source modules edited, upstream updates fetched, cron entries installed, suites loaded, or forecast jobs submitted. Full ecFlow/Rocoto execution and post-June-29 upstream status remain unverified.

Source Index

All source links below are pinned to the inspected revision so that this report remains meaningful when published to the wiki.

Source Primary evidence
Target forecast-only ecFlow class Constructor and unimplemented hooks
Shared workflow base Common configuration and abstract interface
ecFlow base Initialization forwarding only
Workflow setup entry point CLI selection, factories, unconditional write call
Application factory GFS application registration
ecFlow factory Only active ecFlow application registration
Rocoto factory Corresponding Rocoto registration
Application base Finalization and run options
GFS forecast-only application Selected configurations and task list
Rocoto forecast-only cycles GFS, sequential, and verification cycle rules
Rocoto base XML assembly, output, and cron handling
Workflow task traversal Application tasks to backend task builders
Task factory GFS task builder selection
Common Rocoto task support Environment, resources, and grouping support
GFS task builders Forecast segmentation and readiness/dependency logic
Legacy ecFlow entry point Separate YAML-driven interface
Legacy ecFlow setup helper Definition assembly and saving
Legacy ecFlow definition builder ecFlow objects and script copying
Legacy README Historical design and input documentation
Rocoto forecast wrapper Development J-job invocation
Development forecast J-job Shared forecast payload boundary
ecFlow forecast wrapper Operations-style J-job invocation
ecFlow header ecFlow initialization and failure reporting
ecFlow tail ecFlow completion reporting
ecFlow CI case Declared case and host exclusions

Historical references: PR #4071, introducing commit, PR #912, and PR #916. The attribution and intent above were read from local Git objects; online pull-request discussion and review threads were not independently retrieved.

⚠️ **GitHub.com Fallback** ⚠️