MPI Parallelization - pwollstadt/IDTxl GitHub Wiki
IDTxl provides MPI-parallelization for serial CPU estimators.
Requirements
mpi4py>=3.0.0
Usage
To use MPI, simply add a MPI=True
flag in the Estimator settings dictionary (one can optionally provide a n_workers
argument) and start the script using
mpiexec -n 1 -usize <max workers + 1> python
on systems with MPI>=2.0 or
mpiexec -n <max workers + 1> python -m mpi4py.futures
on legacy MPI 1 implementations.
Limitations
Internally, the class MPIEstimator
serves as a decorator for arbitrary serial Estimators (is_parallel()==False
) sending chunks of data to individual Estimator
instances on MPI worker ranks using mpi4py.futures
.
MPIEstimator
does not yet properly work with Estimators
which are already parallel themselves (is_parallel()==True
). Also, GPU support has not yet been implemented