SLURM Guide - aimagelab/aimagelab-srv GitHub Wiki

SLURM Workload Manager (or simply SLURM, which stands for "Simple Linux Utility for Resource Management") is an open source and highly scalable job scheduling system.

SLURM has three key functions. Firstly, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time, so they can perform their work. Secondly, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing the queue of pending jobs.

Currently, SLURM is the scheduling system of aimagelab-srv and MARCONI100. A comprehensive documentation can be found on the SchedMD site.

Running applications using SLURM

With SLURM you can specify the tasks that you want to be executed; the system takes care of running these tasks and returns the results to the user. If the resources are full, then SLURM holds your jobs and runs them when they will become available.

With SLURM you create a batch or interactive job which you then submit to the scheduler. A batch job is a file (a shell script under UNIX) containing the set of commands that you want to run. It also contains the directives that specify the characteristics (attributes) of the job, and the resource requirements (e.g. number of processors and CPU time) that your job needs.

Once you create your job, you can reuse it if you wish. Or, you can modify it for subsequent runs.

For example, here is a simple SLURM job script to run a user's application by setting a limit (one hour) to the maximum wall clock time, requesting 1 full node with 36 cores:

#!/bin/bash
#SBATCH --nodes=1                    # 1 node
#SBATCH --ntasks-per-node=36         # 36 tasks per node
#SBATCH --time=1:00:00               # time limits: 1 hour
#SBATCH --error=myJob.err            # standard error file
#SBATCH --output=myJob.out           # standard output file
./my_application

Basic SLURM commands

The main user's commands of SLURM are reported in the table below: please consult the man pages for more information.

Command	Description
sbatch, srun, salloc	Submit a job
squeue	Lists jobs in the queue
sinfo	Prints queue information about nodes and partitions
sbatch batch script	Submits a batch script to the queue
scancel <jobid>	Cancel a job from the queue
scontrol hold <jobid>	Puts a job on hold in the queue.
scontrol release <jobid>	Releases a job from hold
scontrol requeue <jobid>	Requeue a running, suspended or finished Slurm batch job into pending state.
scontrol show job <jobid>	Produce a very detailed report for the job.
sacct -u <user-name>	Show all user job information starting from 00:00:00 of the current day.
sacct -u <user-name> --starttime <YYYY-MM-DD>	Show all user job information starting from a specific date.

Submit a job

> sbatch [opts] job_script
> salloc --nodes=<nodes_no> --ntasks-per-node=<tasks_per_node_no> --account=<account_no> --partition=<name> <command>  (interactive job)

The second command is related to a so-called "Interactive job": with salloc the user allocates a set of resources (nodes). The job is queued and scheduled as any SLURM batch job, but when executed, the standard input, output, and error streams of the job are connected to the terminal session in which salloc is running. When the job begins its execution, all the input to the job is taken from the terminal session. You can use CTRL-D or "exit" to close the session. if you specify a command at the end of your salloc string (like "./myscript"), the job will simply execute the command and close, prompting the standard output and error directly on your working terminal.

Displaying Job Status:

> squeue                           (lists all jobs)
> squeue -u $USER (lists only jobs submitted by you)
> squeue --job <job_id> (only the specified job)
> squeue --job <job_id> -l         (full display of the specified job)
> scontrol show job <job_id>       (detailed informations about your job)

To view a complete list of all options and their descriptions, use man sinfo, or access SchedMD webpage on squeue.

Delete a job:

> scancel <jobID>

SLURM job script

A SLURM job script consists of:

An optional shell specification
SLURM directives
Tasks -- programs or commands

Once ready, the job must be submitted to SLURM:

> sbatch [options] <name of script>

The shell to be used by SLURM is defined in the first line of the job script (mandatory!):

#!/bin/bash (or #!/bin/sh)

The SLURM directives are used to request resources or set attributes. A directive begins with the default string #SBATCH. One or more directives can follow the shell definition in the job script.

The tasks can be programs or commands. This is where the user specifies the application to run.

SLURM directives

The type of resources required for a job must be specified with a SLURM directive:

#SBATCH --<resource_spec>=<value>

where <resource_spec> can be one of the following:

--cpus-per-task=TT             number of threads/cpus per task
--gres=gpu:X                   number of GPUs to be requested

Other SLURM directives

#SBATCH --account=<account_no>          --> name of the project to be accounted to (useful on Cineca HPC systems)
#SBATCH --job-name=<name>               --> job name
#SBATCH --partition=<destination>       --> partition/queue destination (mainly useful on Cineca HPC systems)
#SBATCH --output=<out_file>             --> redirects output file (default, if missing, is slurm-<Pid> containing merged output and error file)
#SBATCH --error=<err_file>              --> redirects error file (as above)

Job submission and sbatch attributes

When the jobscript is ready, you can submit it with the command:

> sbatch <name of script>

The job will be queued by SLURM worload manager and will be executed when the requested resources will become available.