MPI and Hybrid Programming Models, William Gropp

That's the way you solve problems. Find the appropriate tools for each piece

As the scale of these machines continue to grow, it becomes increasingly difficult to use the pure MPI model everywhere.

Memory is expensive. It consumes power and money. And the more of it you have, the more problems you are likely to have.

pthreads works because every time somebody does something in the kernel that makes pthreads fail, it gets fixed.

MPI will exploit whatever hardware is available (that is, additional cores), to improve the performance of communication.

Some Myths about the MPI+OpenMP Model

1. It never works See FEM assembly, others for counter-example
2. It always works See NAS, EarthSim, others, for counter-examples
3. Requires a special thread-safe MPI Requires a level defined in MPI-2
4. Harder to program. Harder than what? Remember that performance is our goal here, no easy

It is important that you really think through what you're doing.

Thread-based models are very sensitive to how you lay out your problem. Very few multicore systems can provide enough bandwidth for every core to scale perfectly.

MPI's Four Levels of Thread Safety

Thread-safety is not only "not free", it's also expensive in terms of performance

• MPI_THREAD_SINGLE - only one thread exists in the application
• MPI_THREAD_FUNNELED - multithreaded, but only the main thread makes MPI calls (the one that called MPI_Init or MPI_Init_thread)
• MPI_THREAD_SERIALIZED - multithreaded, but only one thread at a time makes MPI calls
• MPI_THREAD_MULTIPLE - multithreaded and any thread can make MPI calls at any time (with some restrictions to avoid races)

MPI_THREAD_MULTIPLE

It is the user's responsibility to prevent races when threads in the same application post conflicting MPI calls

Blocking MPI calls will block only the calling thread

A program that calls MPI_Init (instead of MPI_Init_thread) should assume that MPI_THREAD_SINGLE has been requested.

The Current Situation

Most OpenMP programs don't need anything more sophisticated than MPI_THREAD_FUNNELED, which is commonly supported, even if not advertised.

Some Things to Watch for in OpenMP

• No portable way to manage memory affinity
• Memory model can require explicit "memory flush" operations

Challenges for Programming Models

• Parallel programming models need to provide ways to coordinate resource allocation
• They must also provide clean ways to share data
• Remember, parallel programming is about performance
  • You will always get higher programmer productivity with a single-threaded code

MPI and Hybrid Programming Models, William Gropp

That's the way you solve problems. Find the appropriate tools for each piece

As the scale of these machines continue to grow, it becomes increasingly difficult to use the pure MPI model everywhere.

Memory is expensive. It consumes power and money. And the more of it you have, the more problems you are likely to have.

pthreads works because every time somebody does something in the kernel that makes pthreads fail, it gets fixed.

MPI will exploit whatever hardware is available (that is, additional cores), to improve the performance of communication.

Some Myths about the MPI+OpenMP Model

1. It never works See FEM assembly, others for counter-example
2. It always works See NAS, EarthSim, others, for counter-examples
3. Requires a special thread-safe MPI Requires a level defined in MPI-2
4. Harder to program. Harder than what? Remember that performance is our goal here, no easy

It is important that you really think through what you're doing.

Thread-based models are very sensitive to how you lay out your problem. Very few multicore systems can provide enough bandwidth for every core to scale perfectly.

MPI's Four Levels of Thread Safety

Thread-safety is not only "not free", it's also expensive in terms of performance

• MPI_THREAD_SINGLE - only one thread exists in the application
• MPI_THREAD_FUNNELED - multithreaded, but only the main thread makes MPI calls (the one that called MPI_Init or MPI_Init_thread)
• MPI_THREAD_SERIALIZED - multithreaded, but only one thread at a time makes MPI calls
• MPI_THREAD_MULTIPLE - multithreaded and any thread can make MPI calls at any time (with some restrictions to avoid races)

MPI_THREAD_MULTIPLE

It is the user's responsibility to prevent races when threads in the same application post conflicting MPI calls

Blocking MPI calls will block only the calling thread

A program that calls MPI_Init (instead of MPI_Init_thread) should assume that MPI_THREAD_SINGLE has been requested.

The Current Situation

Most OpenMP programs don't need anything more sophisticated than MPI_THREAD_FUNNELED, which is commonly supported, even if not advertised.

Some Things to Watch for in OpenMP

• No portable way to manage memory affinity
• Memory model can require explicit "memory flush" operations

Challenges for Programming Models

• Parallel programming models need to provide ways to coordinate resource allocation
• They must also provide clean ways to share data
• Remember, parallel programming is about performance
  • You will always get higher programmer productivity with a single-threaded code

Some nice references

"Threads cannot be implemented as a library" "You Don't Know Jack about Shared Variables or Memory Models"