Multiprocessing On A Single Node - bregord/emcee-on-calcul-quebec GitHub Wiki

The default implementation of multiprocessing in emcee doesn't take advantage of any MPI libraries to distribute computations across a cluster.

If there are problems running emcee due to problems with MPI, then the following steps may allow the running of computations on Calcul Quebec systems, albeit these smaller jobs than a cluster would normally allow. This is because your program will only be able to run on a single node.

There are two changes that need to be done in order to run emcee with its default multiprocessing implementation.

Add the threads argument

The first change to make to your script with emcee, is to add the threads argument with the number of desired threads.

 sampler = emcee.EnsembleSampler(nwalkers, ndim, lnpostfn, args=[] threads=x)

In addition, lnpostfn and args need to be pickleable. Information as to what is and isn't pickleable can be found on the python docs here

###Change the submission script The only changes to the submission script inovles restricting it to a single node, and choosing a number of processes that is the same as the number of threads you specified in your python file.

Specifically, just thange the -l argument in the submission script.

 #PBS -l nodes=1:ppn=4

###Further reading More information on multiprocessing without MPI can be found on the emcee site here