On the Open Science Grid (OSG), simulations are most efficiently performed through the Singularity container. Please follow the following instructions to set up simulations.

Set up

Log in to the OSG using ssh [UserName]@ap20.uc.osg-htc.org (The login node could differ for different users). The OSG requires one to specify the project.

Once you log in to the OSG,

• Download the iEBE-MUSIC framework code under your home directory,

    git clone https://github.com/chunshen1987/iEBE-MUSIC -b dev

Set up the virtual environment for Python

    bash iEBE-MUSIC/Cluster_supports/OSG/installPythonVenv.sh
    source venv/bin/activate

If the virtual environment is correctly set up, you will see a folder venv created in your $HOME directory. And you will see (venv) in your prompt.

• Download Singularity image in your public folder ($DATA/singularity_repos/),

    cd $DATA
    mkdir -p singularity_repos/
    cd singularity_repos/
    singularity pull docker://chunshen1987/iebe-music:dev

It is a good practice to add a tag to the singularity image file to distinguish different versions of the image files over time.

Note: Please make sure you have a folder log under your $HOME directory (mkdir -p ~/log) to store the job output and error log files. Without this folder set properly, all the jobs on OSG will FAIL.

Running Simulations

To generate and run a batch of simulations, you need to start from your home directory,

    cd ~
    mkdir [iEBEMUSICTestRun]
    cd [iEBEMUSICTestRun]
    cp ../iEBE-MUSIC/config/parameters_dict_user_IPGlasma.py ./  # or copy your own parameter file

You can call your run folder [iEBEMUSICTestRun] any name you think is appropriate. There are a few example parameter files in the iEBE-MUSIC/config/ folder for different running modes. Then you can generate the job submission scripts using the script generate_submission_script.py under the iEBE-MUSIC/Cluster_supports/OSG folder,

    python3 ../iEBE-MUSIC/Cluster_supports/OSG/generate_submission_script.py 10 1 1 $DATA/singularity_repos/iebe-music_dev.sif parameters_dict_user_IPGlasma.py [JobName]

Here you need to give a [JobName] to the job.

The syntax of generate_submission_script.py can be printed out on screen by calling this script without any additional argument,

    python3 ../iEBE-MUSIC/Cluster_supports/OSG/generate_submission_script.py

    Usage: generate_submission_script.py Njobs Nevents_per_job N_threads SingularityImage ParameterFile jobId [bayesFile]

If you have a parameter file for Bayesian Analysis, you can pass it to the script as the last optional argument.

After running this script, two files will be generated under your job folder, run_singularity.sh and singularity.submit. You can then submit the job to OSG by

    condor_submit singularity.submit

While the jobs are running, you can type condor_q to check their progress. If some jobs are held, you can use condor_release -all to return them to the queue.

Collecting Results

After the jobs finish, you can use the following utility scripts to merge individual hdf5 files into one. To collect these hdf5 events, please make sure the h5py is available in the system. One needs to load the Python virtual environment by

    source $HOME/venv/bin/activate

It is recommended to perform this merging results under the $DATA folder as follows,

    cd $DATA
    cp ~/iEBE-MUSIC/Cluster_supports/OSG/collect_results.sh .
    cp ~/iEBE-MUSIC/Cluster_supports/OSG/combine_multiple_hdf5.py .
    ./collect_results.sh ~/[iEBEMUSICTestRun]

Here, [iEBEMUSICTestRun] represents your job-running folder under your home directory. Because you only have 50 GB of disk space under your home directory, you can run the collecting script while the jobs run. The script will collect the finished events. You can rerun the same command later to continue collecting the events together.

After all the simulation events are finished and merged into a single hdf5 file, you can transfer it from OSG to your local computer for final-stage data analysis. You can execute the following command on your LOCAL terminal,

    scp -pr [UserName]@ap20.uc.osg-htc.org:$DATA/[iEBEMUSICTestRun] .

Analyzing simulation data locally

After merging the .hdf5 files into one and transferring it to your PC, the merged h5 file needs to be analyzed. On the OSGConnect terminal, copy average_event_spvn_h5_minimumbias.py to $DATA to transfer it to your PC.

    cd $DATA
    cp -r $HOME/iEBE-MUSIC/analysisKit .

Use your local terminal to transfer the [simulationRun] folder and the average_event_spvn_h5_minimumbias.py file to your PC.

    scp -pr [email protected]:$DATA/analysisKit .
    scp -pr [email protected]:$DATA/[iEBEMUSICTestRun] .

Once you finish downloading the simulation data, you can run fetch_Qnch_from_hdf5_LHC.py under analysisKit folder to fetch charged hadrons Qn vectors from the hdf5 database. This script will produce a more compact Python pickle file, which stores event-by-event Qn vectors for more efficient data analysis later. You can specify the kinematic cuts in fetch_Qnch_from_hdf5_LHC.py.

After you generate the Python pickle file, you can use other analysis scripts for flow analysis.

Alternatively, you can write your own analysis script for the observables you are interested in. You can also submit your analysis script to me to enhance the capability of this analysisKit.