Slurm: Conda and Jupyter - nthu-ioa/cluster GitHub Wiki

Python and Slurm

If you are running Python code in a Slurm job, please read the page about multi-threaded programs in Slurm, and following the suggestions under the section about environment variables. Adding the suggested export lines in your sbatch script will prevent python packages like numpy, numexpr and numba from creating many more threads than the number of cores you have requested for your job.

See also:

Jupyter and Slurm

See Jupyter page.

Conda Environments and Slurm

If you want to use a conda environment in your Slurm job, you can source it in your sbatch script (after loading modules, but before starting job steps with srun)

#!/bin/bash
#SBATCH --ntasks=8 
#SBATCH --time=01:00:00
#SBATCH --mem=32G

module purge
module load python
source activate mycondaenvironment

srun python do_something.py

However, this will break if it is submitted from a shell in which a conda environment is already activated (e.g. from a terminal inside a jupyterlab session). The reason for this is complicated (it has to do with how Slurm propagates environment variables).

The following version should always work (if you use the bash shell):

#!/bin/bash
#SBATCH --ntasks=8 
#SBATCH --time=01:00:00
#SBATCH --mem=32G
#SBATCH --export=NONE

source ~/.bash_profile

module purge
module load python
source activate mycondaenvironment

srun python do_something.py

The #SBATCH --export=NONE line stops Slurm propagating any environment variables from the submitting shell. Slurm does not start a login shell, so you have to explicitly source any shell configuration files that you need to set up your environment. For example, I define custom MODULEPATHs in my ~/.bash_profile, so I need to source ~/.bash_profile in the Slurm job script.

What's the difference between source activate and conda activate?

In a normal login session, not much. However, conda activate won't work properly in shell scripts, including slurm job submission scripts. In that case, the easy solution is to use source activate instead.

For the explanation and workaround to make conda activate work in shell scripts, see this github issue: https://github.com/conda/conda/issues/7980