This workshop leverages containerized environments and Python CLI tools to ensure consistency and ease of use. There are four tools [two containerized based and two python CLI based] used in combination to carry out the calibration workflow in the NGIAB ecosystem.

💡 Just want to get started? The Quickstart Guide shows you how to use these tools without needing to install them!

The calibration workflow uses two Python CLI tools and two Docker images that work seamlessly together:

sequenceDiagram
    actor Researcher
    Researcher-->>+ngiab-cal(cli): Prepared data
    Note over Researcher,ngiab-cal(cli): From s3, preprocessor, etc
    Researcher-->>ngiab-cal(cli): Calibration params
    Note over ngiab-cal(docker),ngiab(docker): This is a single docker container<br/>ngiab with ngen-cal installed in it
    ngiab-cal(cli)->>+ngiab-cal(docker): Data with calibration config
    loop update model params
        ngiab-cal(docker)->>+ngiab(docker): run ngen calibration
        ngiab(docker)->>-ngiab-cal(docker): simulation results
    end
    ngiab-cal(docker)->>+ngiab(docker): run ngen validation
    ngiab(docker)->>-ngiab-cal(docker): simulation results
    ngiab-cal(docker)->>-ngiab-cal(cli): Calibrated model parameters
    ngiab-cal(cli)-->>-Researcher: Calibrated parameters

What it does: Prepares all the data needed to run a NextGen simulation by creating a complete "run package."

Key Features:

• Subsets hydrofabric data for your area of interest
• Downloads and processes meteorological forcing data
• Creates model configuration files
• Sets up catchment-specific parameters

Data Sources:

• Hydrofabric:
  • Community HF (based on lynker-spatial v2.2) more information
  • lynker-spatial v2.2 CONUS Hydrofabric
• Forcing Data:
  • NWM Retrospective v3
  • AORC 1km gridded data

Example Usage:

uvx ngiab-prep -i gage-10154200 --start 2022-01-01 --end 2022-02-28 -sfr --source aorc -o my_data_package

What it does: Automates the setup and execution of model calibration using the NGIAB folder structure with all necessary configuration files to run a modified version of ngen-cal.

Key Features:

• Creates calibration configurations automatically
• Manages warmup, calibration, and validation periods
• Runs calibration using Docker containers
• Copies calibrated parameters back to model configurations
• Generates performance plots and metrics

Calibration Algorithm: Uses Dynamic Dimensioned Search (DDS) for parameter optimization.

Example Usage:

# Set up calibration
uvx ngiab-cal /path/to/data -g 10154200

# Run calibration with 100 iterations
uvx ngiab-cal /path/to/data -g 10154200 --run -i 100

What it does: Provides a containerized NextGen framework environment with all dependencies pre-installed.

Contains:

• NextGen executable and libraries
• Supporting tools and scripts
• All required system dependencies
• Optimized runtime environment

Docker Hub: awiciroh/ciroh-ngen-image

Usage: Automatically used by the Python CLI tools when you use the --run flag.

What it does: Specialized container for running the calibration algorithms and analysis.

Contains:

• NGEN-Cal calibration software
• Parameter optimization algorithms (DDS)
• Analysis and plotting tools
• Statistical evaluation functions

Docker Hub: awiciroh/ngiab-cal:devcon25

Usage:

• Automatically launched by ngiab-cal when running calibrations
• Manual usage via docker run command

The tools follow a logical sequence:

1. Data Preparation: NGIAB-Data-Preprocess creates a complete data package
2. Calibration Setup: NGIAB-Cal configures the calibration environment
3. Model Execution: NGIAB Docker Image runs NextGen simulations
4. Parameter Optimization: NGEN-Cal Docker Image performs the calibration
5. Results Processing: Generated outputs include plots, metrics, and updated configurations

This workshop requires Docker and UV.

📋 For detailed setup instructions: Pre-Workshop Checklist

The Python packages are automatically downloaded and run using uvx, and Docker images are automatically pulled when needed.

If you want to modify the tools, see Development Setup for:

• Building custom Docker images
• Installing tools locally for development
• Understanding tool interactions
• Creating modified workflows

flowchart TD
    subgraph "Python CLI Tools"
        A[ngiab_data_preprocess]
        B[ngiab-cal]
    end

    subgraph "Docker Images"
        E[NGIAB Image<br/>NextGen Framework]
        F[NGEN-Cal Image<br/>Calibration Tools]
    end

    subgraph "Build Sources"
        C[NGIAB-CloudInfra<br/>Docker Build]
        D[ngen-cal<br/>Calibration Code]
    end

    A -->|"--run command"| E
    B -->|"--run command"| F
    C -->|"builds"| E
    E -->|"base for"| F
    D -->|"builds + installs"| F

    style A fill:#d8a0df,stroke:#333,stroke-width:2px
    style B fill:#a0c4ff,stroke:#333,stroke-width:2px
    style C fill:#a0dfa0,stroke:#333,stroke-width:2px
    style D fill:#ffb6a0,stroke:#333,stroke-width:2px
    style E fill:#f0f0f0,stroke:#333,stroke-width:2px
    style F fill:#f0f0f0,stroke:#333,stroke-width:2px

Having issues? Check the Troubleshooting Guide for solutions.

Ready to use these tools? Head to the Quickstart Guide to start your first calibration!

Back to Home