Development Setup

This section is for users who want to modify the code of the calibration tools or NextGen components.

Prerequisites

This setup requires Docker and UV.

📋 For installation instructions: Pre-Workshop Checklist

Getting Started

Clone the repository and navigate to the directory:

git clone https://github.com/skoriche/NGIAB-Calibration-DevCon25.git
cd NGIAB-Calibration-DevCon25

To install and build everything, run:

./dev_install.sh

How the Tools Interact

Before building these tools from source, it's important to understand how they interact to avoid running the incorrect versions of the tools.

All of the commands used in the workshop use uvx which will download and run the version of the python package published to PyPI. To run the local version, replace it with uv run e.g. uv run ngiab-cal

flowchart TD

    subgraph Source Code
    subgraph python code
        A[ngiab_data_preprocess]
        B[ngiab-cal]
    end
    subgraph Local Docker Images
        E["nextgen docker image"]
        F["ngen-cal docker image"]
    end
    subgraph Builds docker images
    C[NextGenInABox<br>NGIAB-Cloudinfra]
    D["ngen-cal<br>(ngiab-cal branch)"]
    end
    end


    A -- "--run command" --> E
    B -- "--run command" --> F


    C -- "builds" --> E
    E -- "Base for" --> F
    D -- "builds image <br>+ installs ngen-cal" --> F


    %% Colors chosen to be visible in both light and dark themes with darker text
    style A fill:#d8a0df,stroke:#000,stroke-width:2px,color:#000
    style B fill:#a0c4ff,stroke:#000,stroke-width:2px,color:#000
    style C fill:#a0dfa0,stroke:#000,stroke-width:2px,color:#000
    style D fill:#ffb6a0,stroke:#000,stroke-width:2px,color:#000
    style E fill:#c9c9c9,stroke:#000,stroke-width:2px,color:#000
    style F fill:#c9c9c9,stroke:#000,stroke-width:2px,color:#000

Published Versions

Each of these four tools is published publicly and they're designed to interact with each other's published version. Depending on which part you want to modify, you may need to modify multiple components.

GitHub Repo	PyPI (pip)	DockerHub
https://github.com/CIROH-UA/ngiab_data_preprocess	ngiab_data_preprocess
https://github.com/CIROH-UA/ngiab-cal	ngiab-cal
https://github.com/CIROH-UA/NGIAB-Cloudinfra		awiciroh/ciroh-ngen-image
https://github.com/CIROH-UA/ngen-cal/tree/ngiab_cal		awiciroh/ngiab-cal

Which Tools to Modify

Desired Modification	Code to Modify	Also Needs Modifying
Subsetting, input forcings, model config	ngiab_data_preprocess
Default calibration config, calibration CLI tool	ngiab-cal-cli
Recompile ngen, models, t-route or add your own model	NGIAB-Cloudinfra
Change the calibration process, implement different search algorithms	ngen-cal	NGIAB-Cloudinfra

*Additionally, for --run to automatically run your images, the relevant python packages shown in the diagram above will need to be modified.

Development Installation

Python Packages

# download the code locally
git submodule init
git submodule update

# install the local code
uv pip install -e tools/NGIAB_data_preprocess
uv pip install -e tools/ngiab-cal

(Optional) Building NextGen in a Box

docker build -t my_ngiab tools/NGIAB-CloudInfra/docker/
# change the ngen-cal base image by editing tools/ngen-cal/Dockerfile or running the command below
sed -i 's/awiciroh\/ciroh-ngen-image/my_ngiab/' tools/ngen-cal/Dockerfile
# Optionally patch the preprocessors --run command (this workshop won't use it)
sed -i 's/awiciroh\/ciroh-ngen-image:latest /my_ngiab /g' tools/NGIAB_data_preprocess/modules/ngiab_data_cli/__main__.py

Build ngen-cal Image

docker build -t ngiab_cal tools/ngen-cal/
# patch ngiab-cal --run to use your image
sed -i 's/DOCKER.*/DOCKER_IMAGE_NAME = "ngiab_cal"/g' tools/ngiab-cal/src/ngiab_cal/__main__.py

Troubleshooting Development Issues

🔧 Having issues? Check the Troubleshooting Guide for common problems and solutions.

Common Development Issues

• Submodule errors: Run git submodule update --init --recursive
• Docker build fails: Check Docker daemon is running and has sufficient resources
• Python package conflicts: Use a virtual environment or UV's built-in isolation

Back to Home