CSCS - Ivanderkelen/wiki_tryout GitHub Wiki
Here you'll find info related to CSCS and the Piz Daint cluster.
Some useful links:
- Using Multifactor Authentication (MFA)
- Using Spack on Daint
- Using Python environment for JupyterHub on Daint
Using multifactor authentication
Instructions: https://user.cscs.ch/access/auth/mfa/
💡 With Windows system, there are 2 ways to create the ssh keys: through the SSH service GUI, or through the command line: https://user.cscs.ch/access/auth/mfa/windows/ The command line does not work, we recommend to install Windows Subsystem for Linux (WSL): Ubuntu. Request 'administrator access' to install Ubuntu, and to open 'Turn Windows features On Off' (activate Hyper-V, Virtual Machine Platform, Windows Subsystem for Linux)
Go into your .ssh directory and clone the sshservice-cli
cd .ssh
git clone https://github.com/eth-cscs/sshservice-cli.git
chmod 700 -R sshservice-cli/
The following steps need to be run every time you want to create a new ssh key (max 5 keys per day, but can run as many sessions as we'd like with one key). We recommend to add the following into your .bashrc (you can create aliases):
. ~/.ssh/sshservice-cli/cscs-keygen.sh
eval `ssh-agent -s`
ssh-add -t 1d ~/.ssh/cscs-key
Then log onto ela, then daint:
ssh -A -XY <CSCS username>@ela.cscs.ch
ssh -XY daint
Using Spack package manager on Piz Daint
Spack is a package manager to compile and install packages onto CSCS. It offers the option to install and maintain packages within environments as well.
This page contains an overview of the most usefull commands when using spack on piz daint.
For a more information, check the Spack page of CSCS and official spack documentation.
To install spack, clone the github repository in your $HOME, issue the setup script and load the necessary modules
git clone --branch v0.20.0 https://github.com/spack/spack.git ## pay attention to the version of spack
. spack/share/spack/setup-env.sh
module load daint-gpu spack-configCSCS provide on PizDaint via spack-config module a set of spack configurations, i.e. a set of preset configurations allowing a spack instance to know what is already available on the system (e.g. compilers and packages, like cray-mpich).
Then, you can specify your default installation directory. Note, In your home folder, there should exist spack and .spack, it the later doesn't existed, you should create it.
cd .spack/Add the following lines to the config.yaml file. (create the file if it does not yet exist!)
config:
install_tree: /project/s1207/$USER/spack-install/Now, all spack packages will automaticall be installed under spack-install/ in your user directory on the project folder.
Do this set up before using spack (necessary in every session)
. spack/share/spack/setup-env.sh
module load daint-gpu spack-configLet’s say we want to install NetCDF using spack.
In general, the repository of packages is maintained by the spack community, and in order to get latest updates the user have to update their spack instance by pulling last changes from the GitHub repository.
First, we check which versions of NetCDF (netcdf-c) are available in spack, using spack list netcdf-c or spack info netcdf-c. The latter outputs the following:
$ spack info netcdf-c
CMakePackage: netcdf-c
Description:
NetCDF (network Common Data Form) is a set of software libraries and
machine-independent data formats that support the creation, access, and
sharing of array-oriented scientific data. This is the C distribution.
Homepage: https://www.unidata.ucar.edu/software/netcdf
Preferred version:
4.9.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.9.0.tar.gz
Safe versions:
main [git] https://github.com/Unidata/netcdf-c.git on branch main
4.9.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.9.0.tar.gz
4.8.1 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.8.1.tar.gz
4.8.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.8.0.tar.gz
4.7.4 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.4.tar.gz
4.7.3 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.3.tar.gz
4.7.2 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.2.tar.gz
4.7.1 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.1.tar.gz
4.7.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.0.tar.gz
4.6.3 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.6.3.tar.gz
4.6.2 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.6.2.tar.gz
4.6.1 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.6.1.tar.gz
4.6.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.6.0.tar.gz
4.5.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.5.0.tar.gz
4.4.1.1 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.4.1.1.tar.gz
4.4.1 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.4.1.tar.gz
4.4.0 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.4.0.tar.gz
4.3.3.1 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.3.3.1.tar.gz
4.3.3 https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.3.3.tar.gz
Deprecated versions:
None
Variants:
Name [Default] When Allowed values Description
=========================== ============================== ==================== ===================================================
build_system [autotools] -- cmake, autotools Build systems supported by the package
build_type [RelWithDebInfo] [build_system=cmake] Debug, Release, CMake build type
RelWithDebInfo,
MinSizeRel
dap [off] -- on, off Enable DAP support
fsync [off] -- on, off Enable fsync support
generator [make] [build_system=cmake] ninja,make the build system generator to use
hdf4 [off] -- on, off Enable HDF4 support
ipo [off] [build_system=cmake on, off CMake interprocedural optimization
^[email protected]:]
jna [off] -- on, off Enable JNA support
mpi [on] -- on, off Enable parallel I/O for netcdf-4
optimize [on] -- on, off Enable -O2 for a more optimized lib
parallel-netcdf [off] -- on, off Enable parallel I/O for classic files
pic [on] -- on, off Produce position-independent code (for shared libs)
shared [on] -- on, off Enable shared library
zstd [on] [@4.9.0:] on, off Enable ZStandard compression
Build Dependencies:
autoconf automake cmake curl gmake gnuconfig hdf hdf5 libtool libxml2 m4 mpi ninja parallel-netcdf zlib zstd
Link Dependencies:
curl hdf hdf5 libxml2 mpi parallel-netcdf zlib zstd
Run Dependencies:
None
This suggest netcdf-c version 4.9.0 is the latest (and recommended!) version. We’ll install this for the gcc compiler (version 9.3.0)
spack install [email protected]%[email protected]If all goes to plan, this will install netcdf and its dependencies in the location you specified before.
Have some spa(r)ck-ling fun! ✨
In the following steps, a python environment will be installed. This is usefull to self-install python packages (eg cartopy), for using python on daint. This can with plan python scripts or interactive with the JupyterHub.
For the installation, the Spack package manager is used. If not yet installed, follow the one-time setup described above.
In this example, we will create an environment in which all necessary packages to analyse CTSM/CLM output will be installed.
First, we start by creating a spack environment.
load daint-{mc,gpu} and its related spack-config module
module load daint-mc spack-config
. spack/share/spack/setup-env.sh
Create a folder to work in
mkdir env_ctsm_py
cd env_ctsm_py
Create a file called spack.yaml with the following content:
# spack.yaml
spack:
specs: [python, geos]
view: true
concretizer:
unify: true
modules:
prefix_inspections:
lib64: [LD_LIBRARY_PATH]
packages:
python:
require: '@3.8:'
The geos package is necessary for cartopy to work.
Alternatively, if issues with geos persist (e.g. when not right version is called on when trying to pip install cartopy) then use this:
# spack.yaml
spack:
view: true
concretizer:
unify: true
specs:
- geos
- python
packages:
geos:
require: "@3.3.3:"
python:
require: "@3.8:"
# this is needed to make python able to find GEOS at runtime
modules:
prefix_inspections:
lib64: [LD_LIBRARY_PATH]
Install and activate the spack environment using the spack.yaml file (spack recognizes this file automatically). Note that the installation itself can take a couple of minutes.
spack -e . concretize -f
spack -e . install
spack env activate .
Create the python environment and activate it
python -m venv env_ctsm_py
source env_ctsm_py/bin/activate
Make sure you use a python version > 3.3. If not then first to spack install [email protected] and then python3 -m venv env_ctsm_py. Otherwise, you will get the message that there is no package called "venv". Install the Python packages you need using pip. Here you can add whatever modules you'll need. Below you find a list of modules that are frequently used when analysing netcdfs
pip install cartopy
pip install xarray
pip install geos
pip install geopandas
pip install dask
pip install regionmask
pip install scipy
pip install cf_units
Finally, the spack environment (supporting the python environment) will be linked to JupyterLab kernel
module load jupyter-utils
pip install ipykernel
kernel-create -n env_ctsm_py
💡If you get the following error ImportError: libgeos_c.so.1: cannot open shared object file: No such file or directory when loading cartopy while the spack environment is active, you might need to specify the geos root path in the launcher file as follows:
spack location -i geos #get the root location of geos and add it to the launcher file of the kernel
Once you got this information, you can customise your launcher file.
Open the launcher file of your specific kernel $HOME/.local/share/jupyter/kernels/<kernel-name>/launcher (kernel-name=env_ctsm_py here) and add
export LD_LIBRARY_PATH=<GEOS_ROOT>/lib64:$LD_LIBRARY_PATH
replacing <GEOS_ROOT> with the path you got from previous command
❗ MAKE SURE TO ADD THIS VARIABLE ON TOP OF THE launch SCRIPT RIGHT BELOW THE #!/usr/bin/env bash. This will save you >3 hours. 😓
Launch JupyterHub, and select the kernel env_ctsm_py (top right corner)