Why MPI shared memory?

One main target are applications with “replicated data”: You may have some data in your MPI application that is not distributed, because every process needs this data to combine it with data in its subdomain. In those cases, you must deal with the fact that this replicated data has to be located everywhere and therefore basically fills up your memory. If you now have your replicated data only once per shared memory node as an alternative, then you can significantly reduce the total memory consumption. Having 4 cores on a node may not seem a lot, but if you have 128 cores on your shared memory node, then you could reduce the total memory consumption for this replicated data by a factor of 128.

And there are some applications that have a serious scaling problem

precisely because of such replicated data: you can’t get them to run on more processes because in principle these replicated data will overflow memory. Therefore, this is a prime application for the MPI-3.0 shared memory interface.