This page explains the set up on the Stampede2 Cluster at Taxes Advanced Computing Center (TACC).

First time set up

Log in to the Stampede2 using ssh [UserName]@stampede2.tacc.utexas.edu (The login node could be different for different users).

Once you log in to the Stampede2, you can request an interactive node by

idev

Then you can load the essential modules

module load tacc-singularity

• Download the singularity image of the iEBE-MUSIC framework,

mkdir -p $WORK/singularity_repos/
cd $WORK/singularity_repos/
singularity pull docker://chunshen1987/iebe-music:JETSCAPE

• Download the iEBE-MUSIC framework code under your work directory,

cd $WORK
git clone https://github.com/chunshen1987/iEBE-MUSIC -b JETSCAPE

Running Simulations

To generate and run a batch of simulations, you need to start from your work directory,

cd $WORK/iEBE-MUSIC/
./generate_singularity_jobs.py -w $SCRATCH/[runDirectory] -c stampede2 --node_type SKX -n 24 -n_hydro 1 -n_th 4 -par [parameterFile.py] -singularity $WORK/singularity_repos/iebe-music_JETSCAPE.sif -b [bayesParamFile]

Here [runDirectory] needs to be replaced with the information collision system. The file [parameterFile.py] need to be replaced by the actual parameter file. You can find example parameter files in the config/ folder. All the model parameters are listed in the config/parameters_dict_master.py file. The training parameters for Bayesian emulator can be specified using the -b option with a parameter file [bayesParamFile].

The option -n specifies the number of jobs to run. On Stampede2, we recommend to set the number of hydro events to simulation per job -n_hydro to 1 so that we can set the shortest walltime for each batch of jobs. With -n_hydro 1, the option -n effectively sets the number of total events one wants to simulate in the batch.

On Stampede2, the option --node_type has the available options, SKX, KNL, and ICX on Stampede2. When using KNL nodes, we recommend to set the number of openMP threads -n_th 16. For SKX and ICX nodes, -n_th 4.

If one sets the singularity image with a different name, you need to replace the iebe-music_JETSCAPE.sif with the your customized name.

The full help message can be viewed by ./generate_singularity_jobs.py -h,

usage: generate_singularity_jobs.py [-h] [-w] [-c] [--node_type] [-n]
                                    [-n_hydro] [-n_th] [-par] [-singularity]
                                    [-exe] [-b] [-seed]

⚛ Welcome to iEBE-MUSIC package

  -h, --help            show this help message and exit
  -w , --working_folder_name 
                        working folder path (default: playground)
  -c , --cluster_name   name of the cluster (default: local)
  --node_type           name of the queue (work on stampede2) (default: SKX)
  -n , --n_jobs         number of jobs (default: 1)
  -n_hydro , --n_hydro_per_job 
                        number of hydro events per job to run (default: 1)
  -n_th , --n_threads   number of threads used for each job (default: 1)
  -par , --par_dict     user-defined parameter dictionary file (default:
                        parameters_dict_user.py)
  -singularity , --singularity 
                        path of the singularity image (default: iebe-
                        music_latest.sif)
  -exe , --executeScript 
                        job running script (default:
                        Cluster_supports/WSUgrid/run_singularity.sh)
  -b , --bayes_file     parameters from bayesian analysis (default: )
  -seed , --random_seed Random Seed (-1: according to system time) (default: -1)

After running this script, the working directory $SCRATCH/[runDirectory] will be created. You can then submit the job by

cd $SCRATCH/[runDirectory]
sbatch submit_MPI_jobs.script

While the jobs are running, you can type squeue -u $USER to check the progress.

After the simulations finish, the final results will be automatically copied from the $SCRATCH/[runDirectory] to $WORK/RESULTS/[runDirectory].