PPPL cluster - hiddenSymmetries/simsopt GitHub Wiki

This Wiki page documents the steps to install VMEC2000 and simsopt on PPPL cluster using virtual environments from conda and python venv packages. The approaches shown here are tested, but not the only approaches to install simsopt and VMEC packages.

Conda

VMEC

Download the package

git clone https://github.com/hiddenSymmetries/VMEC2000.git
cd VMEC2000

Conda virtual environment

Load conda module
```
module load anaconda3
```
Create a conda virtual environment with python version 3.9. Let us call it sims_gcc_py39 indicating that we are using python 3.9 and gcc compiler.
```
conda create -n sims_gcc_py39 python=3.9
```
Activate the newly created virtual environment
```
conda activate sims_gcc_py39
```

Dependencies

Intel Compiler

NOTE

Simsopt does not compile with intel compilers on PPPL cluster. Use Intel compiler for VMEC only if you intend to use VMEC without simsopt.

Load all the required compilers and libraries by running
```
module load intel openmpi szip netcdf-c netcdf-fortran
```
The above command also load Intel MKL, which can be found running echo $MKLROOT. This should show MKLROOT as /usr/pppl/intel/2019.u3/compilers_and_libraries/linux/mkl/
We use MKL supplied with a different version of intel compiler to use scalapack and blacs supplied with MKL.
```
export MKLROOT=/usr/pppl/intel/2020.u1/compilers_and_libraries_2020/linux/mkl
```
Copy the relevant cmake config file from cmake/machines folder.
```
cp cmake/machines/pppl_intel.json cmake_config_file.json
```

GCC compilers

Load all the required compilers and libraries by running
```
module load gcc openmpi blacs scalapack szip netcdf-c netcdf-fortran
```
When compared to the corresponding step associated with intel compiler, we are loading default scalapack and blacs libraries from netlib.org compiled with gcc. scalapack and blacs require linear algebra math libraries.
We use Intel MKL supplied with a 2019 version of intel compiler for the math libraries. But we don't have to load intel compilers. We only need to populate the MKLROOT environment variable.
```
export MKLROOT=/usr/pppl/intel/2019.u3/mkl
```
Copy the relevant cmake config file from cmake/machines folder.
```
cp cmake/machines/pppl_gcc.json cmake_config_file.json
```

Compile, and test

First, install python dependencies

pip install --no-cache-dir mpi4py
pip install setuptools wheel scikit-build cmake  ninja numpy f90wrap

Build the python wheel
```
python setup.py bdist_wheel
```
Install the built wheel
```
pip install dist/*.whl
```
Test the installation by loading the installed vmec package
```
python -c "import vmec; print('Success')"
```
If vmec is successfully installed, the above command should print Success.

SIMSOPT

For installing simsopt, follow all the steps related to loading modules and activating the conda virtual environment if not done already. We are going to use gcc to compile simsopt.

If still in VMEC folder, move out of the VMEC folder and download the simsopt package
```
git clone https://github.com/hiddenSymmetries/simsopt.git
cd simsopt
```
Instead of using pip to compile and install simsopt in one step, we use two steps for conda virtual environemnts. First install all the required build dependencies specified in pyproject.toml manually.
```
pip install "setuptools>=45" wheel numpy cmake ninja "setuptools_scm[toml]>=6.0"
```
Then build the simsopt wheel. If simsopt wheels are built before, delete them by deleting the build and dist folders. Before building the wheel, set the environment variable CI to True. This reduces the optimization level because PPPL cluster has heterogenous hardware and we want the compiled code to work on all nodes.
```
export CI=True
python setup.py bdist_wheel
```
The wheel built with the above command resides in dist folder. Make sure there is only one file in dist folder. However it is not a strict requirement. Install the newly built wheel by running
```
pip install dist/simsopt*.whl
```
This command assumes there is only one file with whl extension in the dist folder. If there are multiple whl files, specify the full name of the whl file.

Python venv

Here we use the virtual environment manager venv supplied with python. One drawback of this approach is you can not choose a different version of python other than the installed python. Luckily, different versions of python are available via python modules. For this exercise, we are using python v3.9.4 installed on PPPL cluster and gcc compilers. Python v3.8 and v3.7 are also available on the cluster.

Python Virtual Environment

Load the required python module.
```
module load python/3.9.4
```
NOTE

python refers to system python. When the python module is loaded, only python3 is activated.
Create a virtual environment. Lets call it sims_gcc_py39. We create it under ~/venv folder, where ~ refers to your home directory.
```
python3 -m venv ~/venv/sims_gcc_py39
```
Activate the virtual environment
```
. ~/venv/sims_gcc_py39/bin/activate
```
Now, which python or which python3 should both refer to ~/venv/sims_gcc_py39/bin/ folder.
Upgrade the pip module
```
python -m pip install --upgrade pip
```
If which pip points to ~/venv/sims_gcc_py39/bin/, you could use pip command directly. Otherwise, use pip using the command python -m pip. All the steps below use python -m pip ... for reliability.

VMEC

Download the package

git clone https://github.com/hiddenSymmetries/VMEC2000.git
cd VMEC2000

Load all the required compilers and libraries by running
```
module load gcc openmpi blacs scalapack szip netcdf-c netcdf-fortran
```
We are loading default scalapack and blacs libraries from netlib.org compiled with gcc. scalapack and blacs require linear algebra math libraries.
We use Intel MKL supplied with a 2019 version of intel compiler for the math libraries. But we don't have to load intel compiler module. We only need to populate the MKLROOT environment variable.
```
export MKLROOT=/usr/pppl/intel/2019.u3/mkl
```
Copy the relevant cmake config file from cmake/machines folder.
```
cp cmake/machines/pppl_gcc.json cmake_config_file.json
```
Install mpi4py package manually with fresh compilation.
```
python -m pip install --no-cache-dir mpi4py
```
Here we use only one command to install vmec python extension.
```
python -m pip install .
```
Test the installation by loading the installed vmec package
```
python -c "import vmec; print('Success')"
```
If vmec is successfully installed, the above command should print Success.

SIMSOPT

If still in VMEC folder, move out of the VMEC folder and download the simsopt package
```
cd ..
git clone https://github.com/hiddenSymmetries/simsopt.git
cd simsopt
```
Again we use pip to compile and install simsopt in one step.
```
export CI=True
python -m pip install .
```
Test the installation
```
python -c "import simsopt; print('Success')"
```
If simsopt is successfully installed, the above command should print Success.