Skip main navigation

Restrictions for MPI shared memory

In this article we discuss restrictions for MPI shared memory, for example limited size or number of MPI shared memory windows.
© HLRS

The following caution is discussed with members of the MPI forum:

Caution: On some systems
  • the number of shared memory windows, and
  • the total size of shared memory windows may be limited.
Some operating systems may provide options, e.g.,
  • at job launch, or
  • MPI process start,
to enlarge restricting defaults.
In effect, the number of bytes you can get for shared memory windows and the number of shared memory windows you can get at all, can be limited. On some operating systems there are methods for you to extend these limitations. The first limitation you have is that the number of window handles is limited, for example, to around 2043. Despite this fact, in principle, every MPI library has a method to increase this number by an environment variable before you call mpirun. So, if you need 7.000 window handles this should be possible. The second limitation you have is that only a part of the memory may be available for shared memory:
If MPI shared memory support is based on POSIX shared memory:
  • Shared memory windows are located in memory-mapped /dev/shm or /run/shm
  • Default: 25% or 50% of the physical memory,
  • Root may change size, e.g., with:
    mount -o remount,size=6G /dev/shm

In most cases, if you have a POSIX shared memory system, your available shared memory is mapped to /dev/shm or /run/shm. A system administrator can change this size at any time with a remount. Typically, your administrator will give you 25 or 50% of the memory and the rest won’t work. In current supercomputing system it may be possible to increase this percentage up to 90%.

© HLRS
This article is from the free online

One-Sided Communication and the MPI Shared Memory Interface

Created by
FutureLearn - Learning For Life

Reach your personal and professional goals

Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.

Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.

Start Learning now