Introduction to MPI
- Communication between processes
- sending and receiving messages between two processes
- sending and receiving messages between several processes
- Synchronization between processes
- Communicator creation and manipulation
- Advanced features (e.g. user defined datatypes, one-sided communication and parallel I/O)
Execution modelA MPI program is launched as a set of independent, identical processes. Each process executes exactly the same program code and instructions. The processes can reside in different CPU cores, compute nodes or even in different computers. In principle, all that is needed is a network connection between the processes, but in practice, a high-bandwidth, low-latency interconnect is needed for good performance. A special MPI launcher is needed to start a MPI program. Different systems use different MPI launchers (depending on the MPI implementation and/or queueing system in place), but common ones include:
srun. In the virtual machine used in the course we use
MPI rankWhen a MPI program is launched, each process is assigned an unique ID number
called rank. It is possible to query the rank within a program and to
perform different tasks based on it. In fact this is the basis on which all
logic for parallelism is built on in a MPI program.
if (rank == 0):
# do something
elif (rank == 1):
# do something else
# all other processes do something different
Want to keep
Partnership for Advanced Computing in Europe (PRACE) online course,
Python in High Performance Computing
processes that participate in communication. When a MPI routine is called, the
communication will involve some or all of the processes in a communicator. All processes start in a global communicator called
MPI.COMM_WORLDin mpi4py), but the user can also define their own custom
communicators as needed. In C and Fortran, all MPI routines expect a communicator as one of the
arguments. In Python, most MPI routines are implemented as methods of a
Data modelSince all MPI processes are completely independent, this means also a
complete separation of data. Each process has its own separate memory space,
i.e. all variables and data structures are local to the process. To exchange
information, processes need to explicitly send and receive messages.
Getting startedTo get started with MPI, it is a good idea to look into two basic methods of
a communicator object:
Get_rank(). These two routines can be used to find out two pieces of information that
virtually all MPI programs need, i.e. the total number of MPI processes
executing the program and the unique ID (rank) assigned to this
process. To be more precise, they will give you the number of processes in the
communicator and the rank within this communicator. A single process can
belong to multiple communicators and have a different rank in each one.
Simple example: Hello world
from mpi4py import MPI
comm = MPI.COMM_WORLD # communicator object containing all processes
size = comm.Get_size()
rank = comm.Get_rank()
print("I am rank %d in group of %d processes" % (rank, size))
Running the example programAs discussed earlier, MPI program is started with a special
launcher. When working with Python, the executable that is started is the
Python interpreter. In the example below, we start four Python
interpreters which all run then the same hello.py program. The
communication between the programs is faciliated with the MPI function calls.
$ mpirun -np 4 python3 hello.py
I am rank 2 in group of 4 processes
I am rank 0 in group of 4 processes
I am rank 3 in group of 4 processes
I am rank 1 in group of 4 processes
Python in High Performance Computing
Our purpose is to transform access to education.
We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.
We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.