Undocumented support of EcFlow in public release of fv3gfs - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

EcFlow Capabilities in the first public release of fv3gfs on GitHub

The EcFlow support is implemented through the CROW (Configurable Workflow) framework located in the fv3gfs/ecf/ecfutils/CROW directory. This framework serves as a meta-scheduler that can generate both Rocoto XML and EcFlow suite definitions from the same YAML configuration.

Architecture Overview

The system uses a multi-layer architecture:

  1. YAML Configuration Layer: Case files define experiment parameters
  2. CROW Meta-scheduler: Translates configurations into workflow definitions
  3. Workflow Execution Layer: The actual EcFlow or Rocoto execution engine

Here's how the architecture works:

┌────────────────────────────┐
│   YAML Configuration       │
│ (cases/*.yaml, user.yaml)  │
└──────────────┬─────────────┘
               │
               ▼
┌────────────────────────────┐
│  CROW Meta-scheduler       │
│  (worktools.py, CROW/)     │
└──────────────┬─────────────┘
               │
      ┌────────┴─────────┐
      │                  │
      ▼                  ▼
┌──────────────┐  ┌──────────────┐
│ Rocoto XML   │  │ EcFlow Suite │
│ Generation   │  │ Generation   │
└──────────────┘  └──────────────┘

Steps to Generate an EcFlow Workflow

To generate an EcFlow workflow from this system, follow these steps:

Step 1: Set up your experiment using a case file

First, navigate to the ecfutils directory and create your experiment directory:

cd fv3gfs/ecf/ecfutils
./setup_case.sh ~/my_experiment Harvey_2017081712_GFS@C384

This uses the Harvey case configuration but you can choose any of the available case files in the cases directory.

Step 2: Customize your workflow

Create a user.yaml file in your experiment directory to override default settings:

cat > ~/my_experiment/user.yaml << 'EOT'
user:
  ECF_HOME: /path/to/ecflow/home
  ECF_PORT: 12345  # Port for ECF server
  ECF_HOST: localhost  # Host running ECF server
  ECF_NAME: fv3gfs_harvey  # Name of the suite
  ACCOUNT: yourAccount  # Your HPC account
  QUEUE: normal  # Submission queue
EOT

Step 3: Generate the EcFlow suite definition and scripts

Use the worktools.py module to generate the EcFlow suite:

cd ~/my_experiment
python3 -c "import sys; sys.path.insert(0, 'fv3gfs/ecf/ecfutils'); import worktools; worktools.generate_ecflow()"

This will create an EcFlow suite definition and all necessary ECF scripts in your experiment directory.

Step 4: Load the workflow into ECF server

Start the ECF server if it's not already running:

ecflow_start -p 12345

Then load your workflow into the ECF server:

ecflow_client --load=ecflow_suite.def
ecflow_client --begin=fv3gfs_harvey  # Use your ECF_NAME here

Step 5: Monitor your workflow

Use the ECF UI to monitor your workflow:

ecflowview &

Example Top-Level Case File

Here's a working example of a top-level case file that you could use to generate an EcFlow workflow:

# FV3GFS_Demo_Workflow.yaml
case:
  fv3_settings:
    CASE: C384  # Resolution
    LEVS: 65    # Vertical levels
    FHMAX: 120  # Max forecast hours
    
  places:
    workflow_file: workflow/public_release_v1.yaml
    ICSDIR: /path/to/initial/conditions
    COMROT: /path/to/com/directory
    EXPDIR: /path/to/experiment

  settings: 
    SDATE: 2025-04-30t00:00:00  # Start date
    EDATE: 2025-05-03t00:00:00  # End date
    IC_CDUMP: gfs              # Initial conditions source
    PSLOT: FV3GFS_ECF_DEMO     # Project slot name
    SCHEDULER: ecflow          # Use ecflow rather than rocoto
    
    # Cycle settings
    gfs_cyc: 4                # Run 4 times per day
    
    # Feature toggles
    run_gsi: Yes              # Run data assimilation
    run_enkf: No              # No ensemble Kalman filter
    run_post: Yes             # Run post-processing
    
  resources:
    NODES_FCST: 20            # Number of nodes for forecast
    NODES_POST: 4             # Number of nodes for post-processing
    job_walltime: 02:30:00    # Default job wall time

Architectural Diagram of the EcFlow Suite

The generated EcFlow suite would have a structure like this:

fv3gfs_harvey                   # Suite
├── setup                       # Family for setup tasks
│   ├── getic                   # Task to get initial conditions
│   └── chgres                  # Task to run chgres (convert ICs)
├── cycles                      # Family containing cycle definitions
│   ├── 20170817                # Family for August 17, 2017 
│   │   ├── 00                  # Family for 00Z cycle
│   │   │   ├── analysis        # Data assimilation steps
│   │   │   ├── forecast        # Forecast steps
│   │   │   │   ├── fcst        # Main forecast task
│   │   │   │   └── post        # Post-processing
│   │   │   └── products        # Product generation
│   │   ├── 06                  # Family for 06Z cycle
│   │   │   └── ...
│   │   ├── 12                  # Family for 12Z cycle - starting point
│   │   │   └── ...
│   │   └── 18                  # Family for 18Z cycle
│   │       └── ...
│   └── 20170818                # Next day's cycles
│       └── ...
└── cleanup                     # Cleanup tasks

The diagram shows how EcFlow organizes the workflow hierarchically, with families for dates and cycles, and tasks for the actual computational steps.

Relationship Between Rocoto and EcFlow in This Framework

The developers of the fv3gfs public release created a framework where the same workflow definition (in YAML) can generate either Rocoto XML or EcFlow suite definitions. This provides flexibility in which workflow manager to use based on site preferences.

The key components enabling this flexibility are:

  1. The CROW framework in fv3gfs/ecf/ecfutils/CROW/
  2. The workflow templates in fv3gfs/ecf/ecfutils/workflow/
  3. The ecflow module in ecflow.py

By abstracting the workflow management system from the a generalized configuration framework gives rise to the flexibility of supporting both Rocoto and EcFlow.

Migration Path

For teams currently using Rocoto but wanting to move to EcFlow, the migration path is straightforward:

  1. Continue using the same case files
  2. Simply change the SCHEDULER setting to ecflow
  3. Add ECF-specific settings in the user.yaml file
  4. Generate and load the EcFlow suite instead of the Rocoto XML

This dual-capability design minimizes the disruption when transitioning between workflow systems and leverages the strengths of both approaches.

Note: This document was generated by Claude Sonnet 3.7 LLM from Anthropic managed by GitHub's Copilot agentic management system by the direction and final review of Terrence McGuinness on May 1, 2025.