Environments, Mamba, and Pip

One has to be quite careful in the environment handling.

Overview

At the moment there are three conda/mamba environments:

• exio-bootstrap: For the installation and update system. This never needs to run anything in EXIOBASE itself, so it can remain separate and dedicated to the install/update procedure. Not least, changes can be made to the exio-datamanager environment from code running under exio-bootstrap.

• exio-datamanager: The main environment. The other repos such as macrodb, etc, import from datamanager so these all run under the same environment. This environment also includes editable pip installs for datamanager itself, macrodb, etc, so that these packages can easily see each other. It also includes the pip install for df_file_interchange from PyPi. Make sure to update "atomically", i.e. with mamba env update --file ./environment.yml --prune (bootstrap does this as part of update.py). DO NOT start upgrading or installing packages individually because this is likely to mangle the environment.

• exio-snakemake: This is the dedicated install for the Snakemake pipeline. This should be kept separate. The calls to datamanager, macrodb, etc, are performed by some fancy serialization that Snakemake implements and the target conda enviroment is specified per rule.

Versioning and Warnings

Be careful with the versions. This is largely a note to myself. If the versions in the pyproject.toml for df_file_interchange place an upperbound that is older than what's installed via conda/mamba then pip will overwrite, which causes a massive headache (just took about 2 hours to fix, June 2024).

Make sure to allow conda/mamba to do the install or update of the environment because it manages pip interaction properly.