Learn more about this course.

Restrictions for MPI shared memory

In this article we discuss restrictions for MPI shared memory, for example limited size or number of MPI shared memory windows.

The following caution is discussed with members of the MPI forum:

Caution: On some systems

the number of shared memory windows, and

the total size of shared memory windows may be limited.

Some operating systems may provide options, e.g.,

at job launch, or

MPI process start,

to enlarge restricting defaults.

In effect, the number of bytes you can get for shared memory windows and the number of shared memory windows you can get at all, can be limited. On some operating systems there are methods for you to extend these limitations. The first limitation you have is that the number of window handles is limited, for example, to around 2043. Despite this fact, in principle, every MPI library has a method to increase this number by an environment variable before you call mpirun. So, if you need 7.000 window handles this should be possible. The second limitation you have is that only a part of the memory may be available for shared memory:

If MPI shared memory support is based on POSIX shared memory:

Shared memory windows are located in memory-mapped /dev/shm or /run/shm

Default: 25% or 50% of the physical memory,

Root may change size, e.g., with:
mount -o remount,size=6G /dev/shm

In most cases, if you have a POSIX shared memory system, your available shared memory is mapped to /dev/shm or /run/shm. A system administrator can change this size at any time with a remount. Typically, your administrator will give you 25 or 50% of the memory and the rest won’t work. In current supercomputing system it may be possible to increase this percentage up to 90%.