Software: GNU Parallel - nthu-ioa/cluster GitHub Wiki

Gnu Parallel is a powerful (and complex) tool for running multiple instances of independent tasks. The most basic use-case is similar to a Slurm job array. However, very clever things are possible by using both Slurm (to request and manage cluster resources) and Parallel (to distribute tasks to those resources).

If you are struggling to build a complicated loop over tasks and files using Slurm alone, consider Parallel. The learning curve is similar to that of vim.

Parallel is available through module load parallel.

Parallel Website: https://www.gnu.org/software/parallel/ (see Documentation on that page)

A tutorial: https://www.biostars.org/p/63816/