Creating SVJ virtual environment - jniedzie/SVJanalysis_wiki GitHub Wiki
This page explains how to create a virtual environment to run the framework, including using the Machine Learning Framework.
Important note: If you are a semester project student, you should skip this part and source Florian's virtual env. See instructions at the bottom of the Setup page.
Follow the instructions here to install micromamba.
At PSI T3, because your home has limited storage, you should install conda in /work/<username>/miniforge3.
Usage is just like conda but instead type mamba or micromamba (depending on if you install mamba or micromamba).
We recommend creating a virtual environment with Coffea run Coffea-based code in this repository. conda is already installed on lxplus machines, but this is not the case at t3ui01.psi.ch
If you need to install conda, go to https://conda.io/projects/conda/en/latest/user-guide/install/linux.html#installing-on-linux and follow instructions there.
At PSI T3, because your home has limited storage, you should install conda in /work/<username>/anaconda3.
You can run the following commands or write them to makeSVJEnv.sh and source makeSVJEnv.sh.
name=svj
echo -e "\n=== Create Python 3.11 environment with name ${name} ==="
mamba create --name ${name} python=3.11
echo -e "\n=== Activate ${name} ==="
mamba activate ${name}
# For I/O and array manipulations
echo -e "\n=== Installing xrootd ==="
mamba install -c conda-forge xrootd
echo -e "\n=== Installing coffea ==="
mamba install -c conda-forge coffea"=0.7.22"
# For easier calculations with vectors
echo -e "\n=== Installing vector ==="
mamba install -c conda-forge vector
# Force numpy version
mamba install -c conda-forge numpy==1.23.5
# Force setuptools version
mamba install -c conda-forge setuptools==68.2.2
# For plotting
echo -e "\n=== Installing matplotlib ==="
mamba install -c conda-forge matplotlib
echo -e "\n=== Installing seaborn ==="
mamba install -c anaconda seaborn
echo -e "\n=== Installing plotly ==="
mamba install -c plotly plotly
# For job submission
echo -e "\n=== Installing dask-jobqueue ==="
mamba install dask-jobqueue -c conda-forge
# For the ML framework
echo -e "\n=== Installing scikit-learn ==="
mamba install -c anaconda "scikit-learn>=0.24"
#echo -e "\n=== Installing tensorflow ==="
#echo "Check tensorflow version! Tenserflow 2 needed!"
# commented out because was getting issues, done with pip below
#mamba install -c conda-forge tensorflow
echo -e "\n=== Installing keras ==="
mamba install -c conda-forge keras
echo -e "\n=== Installing PyTorch ==="
mamba install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
mamba install -c conda-forge torchmetrics
mamba install -c conda-forge torchview
echo -e "\n=== Installing SHAP ==="
mamba install -c conda-forge shap
echo -e "\n=== Installing comet ==="
mamba install -c anaconda -c conda-forge -c comet_ml comet_ml
echo -e "\n=== Installing POT ==="
#mamba install -c conda-forge "pot==0.7.0"
mamba install -c conda-forge pot
# Auxiliary package for plotting with keras
echo -e "\n=== Installing pydot ==="
mamba install pydot
# To make gif
echo -e "\n=== Installing imageio ==="
mamba install -c conda-forge imageio
echo -e "\n=== Installing braceexpand ==="
mamba install -c conda-forge braceexpand
# Install ssh-access library paramiko
echo -e "\n=== Installing paramiko ==="
mamba install paramiko
echo -e "\n=== Installing bayesian-optimization ==="
mamba install -c conda-forge bayesian-optimization
echo -e "\n=== Installing optuna ==="
mamba install -c conda-forge optuna
mamba install -c conda-forge python-kaleido
echo -e "\n=== Installing ROOT ==="
mamba install -c conda-forge root
echo -e "\n=== Installing tensorflow ==="
echo "Check tensorflow version! Tenserflow 2 needed!"
pip install tensorflow
echo -e "\n=== Installing energyflow ==="
pip install energyflow
# Install DataFrame Image package
pip install dataframe_image
pip install pytest
pip install torchviz
echo -e "\n=== Installing tensorflow ==="
echo "Check tensorflow version! Tenserflow 2 needed!"
pip install tensorflow
echo -e "\n=== Installing energyflow ==="
pip install energyflow
# Install DataFrame Image package
pip install dataframe_image
pip install pytest
pip install torchviz
pip install pyshaper
# Installing modified version of the MDMM
export OLD_PWD=$PWD
pip install mdmm && cd $(python -m site | grep "site-packages'," | cut -d"'" -f2)/mdmm && git clone [email protected]:fleble/mdmm.git && cp mdmm/mdmm/mdmm.py . && rm -rf mdmm
cd $OLD_PWDImportant: on lxplus conda may complain a bit about non-existing or unaccessible directories - patiently create these directories and change permissions as advised by conda. It should work.
Important (if installing on your machine): if you have a local ROOT installation, there could be a clash with the conda environment. To avoid this, make sure you haven't sourced thisroot.sh from the local ROOT installation when creating the env (i.e., check your .bashrc or equivalent)
You can install the virtual env in your eos home directory, which has more space than your lxplus home.
It may be that installing the virtual env takes forever on lxplus. If so, install your virtual env on the PSI T3, pack it, move it to lxplus with scp and unpack it in your eos/lxplus home. Instructions on how to pack/unpack the venv can be found here.
Activate your virtual environment:
mamba activate svj
It is also possible to do:
source <path_to_the_env>/svj/bin/activate
At the end, you can deactivate the virtual env:
deactivate