Skip main navigation

How can I use collective communication to move data into a single task?

You can use collective communication to collect data from all tasks and move them into a single task, i.e. move data from many to one
© CC-BY-NC-SA 4.0 by CSC - IT Center for Science Ltd.

You can use collective communication to collect data from all tasks and move them into a single task, i.e. how to move data from many to one.

1 Gather

Gather collects an equal amount of data from all the processes in a communicator to one process. One can think of it as an inverse scatter operation that allows one, for example, to collect partial results from the tasks.

Segments A, B, etc may contain multiple elements just like in scatter.

An example of gathering a list of single values (rank) from each process as well as a numpy array of multiple elements (data) that are then stored in a larger receive array (buffer):

from mpi4py import MPI
from numpy import arange, zeros

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

data = arange(10, dtype=float) * (rank + 1)
buffer = zeros(size * 10, float)

n = comm.gather(rank, root=0) # returns the value
comm.Gather(data, buffer, root=0) # in-place modification

2 Reduce

Reduce gathers data from all the processes in a communicator and applies an operation on the data before storing the result in a single process. Essentially, it is just like gather, but with an additional operation applied to the gathered data.

The operation that is applied on the data can be anything from a sum to a logical XOR and anything in between. MPI supports a wide variety of operations that can be used, including e.g. maximum value (MPI.MAX), minimum value
(MPI.MIN), summation (MPI.SUM), and product (MPI.PROD).

An example of reduction using MPI.SUM to calculate the total sum of all received values:

from mpi4py import MPI
from numpy import arange, zeros

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

data = arange(10 * size, dtype=float) * (rank + 1)
buffer = zeros(size * 10, float)

n = comm.reduce(rank, op=MPI.SUM, root=0) # returns the value

comm.Reduce(data, buffer, op=MPI.SUM, root=0) # in-place modification

Other common collective operations

  • Scatterv (each process receives different amount of data)
  • Gatherv (each process sends different amount of data)
© CC-BY-NC-SA 4.0 by CSC - IT Center for Science Ltd.
This article is from the free online

Python in High Performance Computing

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education