Slurm: Conda and Jupyter - nthu-ioa/cluster GitHub Wiki
Python and Slurm
If you are running Python code in a Slurm job, please read the page about multi-threaded programs in Slurm, and following the suggestions under the section about environment variables. Adding the suggested export
lines in your sbatch
script will prevent python packages like numpy, numexpr and numba from creating many more threads than the number of cores you have requested for your job.
See also:
Jupyter and Slurm
See Jupyter page.
Conda Environments and Slurm
If you want to use a conda environment in your Slurm job, you can source it in your sbatch
script (after loading modules, but before starting job steps with srun
)
#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --time=01:00:00
#SBATCH --mem=32G
module purge
module load python
source activate mycondaenvironment
srun python do_something.py
However, this will break if it is submitted from a shell in which a conda environment is already activated (e.g. from a terminal inside a jupyterlab session). The reason for this is complicated (it has to do with how Slurm propagates environment variables).
The following version should always work (if you use the bash shell):
#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --time=01:00:00
#SBATCH --mem=32G
#SBATCH --export=NONE
source ~/.bash_profile
module purge
module load python
source activate mycondaenvironment
srun python do_something.py
The #SBATCH --export=NONE
line stops Slurm propagating any environment variables from the submitting shell. Slurm does not start a login shell, so you have to explicitly source any shell configuration files that you need to set up your environment. For example, I define custom MODULEPATHs in my ~/.bash_profile
, so I need to source ~/.bash_profile
in the Slurm job script.
source activate
and conda activate
?
What's the difference between In a normal login session, not much. However, conda activate
won't work properly in shell scripts, including slurm job submission scripts. In that case, the easy solution is to use source activate
instead.
For the explanation and workaround to make conda activate
work in shell scripts, see this github issue: https://github.com/conda/conda/issues/7980