Example Workflow Without Scripts - bregord/emcee-on-calcul-quebec GitHub Wiki

Below is an outline of the process I used to get code working without the use of the filetransfer and initialization scripts. The following assumes that the script using emcee was written using the python2.7

1. Ensure that the program to be run is able to be run with emcee

Full documentation by Dan Foreman-Mackey found here

1 import the emcee utils module, specifically the MPIPool object "from emcee.utils import MPIPool"

2 Initialize the pool object, with loadbalancing

pool = MPIPool(loadbalance=True)
if not pool.is_master():
    pool.wait()
    sys.exit(0)

3 Initialize the sampler, with the pool passed in as an argument. Ensure the lnpostfn and args parameters in your sampler are pickleable

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnpostfn, args=[], pool=pool)

4 Close the pool pool.close()

##2 SSH onto the server, and initialize the environment

$ ssh [email protected]

Then, create a workspace.

$ mkdir workspace
$ cd workspace
$ mkdir data
$ mkdir jobs
$ mkdir programs

Now, we need to use the python modules to get access to the virtual environment features of Python 2.7

$ module load ifort_icc/15.0 
$ module load python/2.7.3-MKL

However, we will also need access to a few other modules in order to correctly install python packages like emcee or scipy or mpi4py.

$ module load iomkl/2015b

iomkl/2015b has a lot of the packages we require, however it also has a version of OpenMPI that is incompatible with python 2.7 and emcee. So, we will unload it and install a version that is.

$ module unload OpenMPI/1.8.8
$ module load openmpi/1.6.3-gcc

Create the virtual env in programs

$ cd programs
$ mkdir env
$ virtualenv env

Now, actitvate the virtual environment
$ source env/bin/activate

And install all the required python packages. It should be noted that a few may already be included.

$ pip install scipy, mpi4py, emcee

Lastly, I added all the module commands to my .bash_profile to ensure a consistent environment every time I logged on. Other servers than the guillimin have a .modules file where this can be done.

##3 Create a submission file The submission file I used is outlined below:

#!/bin/bash
#PBS -l nodes=4:ppn=4
#PBS -l walltime=00:30:00
#PBS -A your-compute-canada-project-id 
#Your id should be of the form abc-123-aa
#PBS -V
#PBS -N JobName
#PBS -l pmem=4000m
#PBS -m bae

# Defining an email address. If you want to send emails to a specific
# email address, specify it with the following option
#PBS -M [email protected]

cd ~/workspace/program

module load ifort_icc/15.0 
module load python/2.7.3-MKL
module load iomkl/2015b
module unload OpenMPI/1.8.8
module load openmpi/1.6.3-gcc

source env/bin/activate
 
#this specifies the program's output
program.py > output.txt

Lines prefixed by #PBS are options given Torque to specify how to run your job.

An example of a submission file with all possible options can be found here

##4 Transfer files

Transferring data:

$ scp data.dat [email protected]:~/workspace/data/

Transfering programs:

$ scp program.py [email protected]:~/workspace/program/

Transferring build script:

$ scp build.sh [email protected]:~/workspace/jobs/

##5 Submit Job ssh onto the server, go to the jobs directory and use the qsub or msub command depending on your server of choice.

It is recommended to run a small test first on the test queue, so change the job submission file so it has the additional #PBS argument #PBS -Q debug, and that its walltime does not exceed 30 minutes.

Then submit.

$ qsub build.sh

And wait for an email indicating your job has completed.

##6 Retrieving and Interpreting Results

Use SCP to retrieve the specific file.

$ scp [email protected]:~/workspace/program/output.txt

By default, each job automatically creates an output file, and an error file, in addition to whatever else you output in your script. These files have the names YOUR_JOB_NAME.o or YOUR_JOB_NAME.e, where each is suffixed by the specific job id generated by the scheduler. The .o file is the output of your file, while .e is the error output. However, you can specify the exact name of the output file as well as the location using the following task manager option in your submission file.

#PBS -o path/output.out