Development Setup - JoshCu/NGIAB-Calibration-DevCon25 GitHub Wiki

Development Setup

This section is for users who want to modify the code of the calibration tools or NextGen components.

To install and build everything, run the following:

dev_install.sh

How the Tools Interact

Before building these tools from source, it's important to know how they interact to avoid running the incorrect versions of the tools.

All of the commands used in the workshop use uvx which will download and run the version of the python package published to PyPI. To run the local version, replace it with uv run e.g. uv run ngiab-cal

flowchart TD

    subgraph Source Code
    subgraph python code
        A[ngiab_data_preprocess]
        B[ngiab-cal]
    end
    subgraph Local Docker Images
        E["nextgen docker image"]
        F["ngen-cal docker image"]
    end
    subgraph Builds docker images
    C[NextGenInABox<br>NGIAB-Cloudinfra]
    D["ngen-cal<br>(ngiab-cal branch)"]
    end
    end


    A -- "--run command" --> E
    B -- "--run command" --> F


    C -- "builds" --> E
    E -- "Base for" --> F
    D -- "builds image <br>+ installs ngen-cal" --> F


    %% Colors chosen to be visible in both light and dark themes with darker text
    style A fill:#d8a0df,stroke:#000,stroke-width:2px,color:#000
    style B fill:#a0c4ff,stroke:#000,stroke-width:2px,color:#000
    style C fill:#a0dfa0,stroke:#000,stroke-width:2px,color:#000
    style D fill:#ffb6a0,stroke:#000,stroke-width:2px,color:#000
    style E fill:#c9c9c9,stroke:#000,stroke-width:2px,color:#000
    style F fill:#c9c9c9,stroke:#000,stroke-width:2px,color:#000
Loading

Published Versions

Each of these four tools is published publicly and they're designed to interact with each other's published version. Depending on which part you want to modify, you may need to modify multiple components.

GitHub Repo PyPI (pip) DockerHub
https://github.com/CIROH-UA/ngiab_data_preprocess ngiab_data_preprocess
https://github.com/CIROH-UA/ngiab-cal ngiab-cal
https://github.com/CIROH-UA/NGIAB-Cloudinfra awiciroh/ciroh-ngen-image
https://github.com/CIROH-UA/ngen-cal/tree/ngiab_cal awiciroh/ngiab-cal

Which Tools to Modify

Desired Modification Code to Modify Also Needs Modifying
Subsetting, input forcings, model config ngiab_data_preprocess
Default calibration config, calibration CLI tool ngiab-cal-cli
Recompile ngen, models, t-route or add your own model NGIAB-Cloudinfra
Change the calibration process, implement different search algorithms ngen-cal NGIAB-Cloudinfra

*Additionally, for --run to automatically run your images, the relevant python packages shown in the diagram above will need to be modified.

Development Installation

Python Packages

# download the code locally
git submodule init
git submodules update

# install the local code
uv pip install -e tools/NGIAB_data_preprocess
uv pip install -e tools/ngiab-cal

(Optional) Building NextGen in a Box

docker build -t my_ngiab tools/NGIAB-CloudInfra/docker/
# change the ngen-cal base image by editing tools/ngen-cal/Dockerfile or running the command below
sed -i 's/awiciroh\/ciroh-ngen-image/my_ngiab/' tools/ngen-cal/Dockerfile
# Optionally patch the preprocessors --run command (this workshop won't use it)
sed -i 's/awiciroh\/ciroh-ngen-image:latest /my_ngiab /g' tools/NGIAB_data_preprocess/modules/ngiab_data_cli/__main__.py

Build ngen-cal Image

docker build -t ngiab_cal tools/ngen-cal/
# patch ngiab-cal --run to use your image
sed -i 's/DOCKER.*/DOCKER_IMAGE_NAME = "ngiab_cal"/g' tools/ngiab-cal/src/ngiab_cal/__main__.py

Back to Home

⚠️ **GitHub.com Fallback** ⚠️