Running R with a batch job with sbatch - EpiModel/EpiModeling GitHub Wiki

In contrast to interactive jobs, batch jobs are set up ahead of time and run in a queue. A typical batch job will be one iteration of a model simulation scenario where the model is simulated and data saved for later exploration interactively. While we use srun for interactive jobs, sbatch is used to ask Slurm for a batch job. Whereas interactive jobs can be free form in terms of their setup, batch jobs for EpiModel projects usually consist of at least three layers:

1. R script to run in batch mode. This is a self-contained R script that loads packages, loads any input data, runs R functions, and saves the output into a data file.
2. A bash shell script that calls the R script. This is a self-contained .sh script that contains any standard Slurm commands for the batch job, loads R for the job, and then runs the R script in layer 1.
3. An sbatch call. This can either be entered interactively into the terminal or chained together with a series of sbatch calls into its own shell script that may be sourced.

Example 1

An example is provided in the sbatchExample directory of this repository. In that case, the sim.R script is the R script, the runSim.sh is the second layer shell script, and the main.sh is top layer call to sbatch over several varying parameter inputs. This call to sbatch also demonstrates array jobs, which are used to create batches of simulations with the same input values (useful for replicating blocks of simulations many times), and also exporting of parameter values that then get pulled into the R script.

Example 2

Another example is provided in the BigNets repository. This shows how you can create a flexible second-layer shell script that can then call any number of R scripts as an argument at the end of the sbatch line in the third-layer shell script.