Introduction to Parallel Programming

Syllabus

• Week 1

  Introduction to parallel programming

  • Intro to parallel programming

    Here you will find out what we have in store for you over the next few weeks and what are some different programming models.

  • OpenMP overview

    We briefly discuss OpenMP.

  • MPI overview

    We briefly discuss MPI.

  • Accelerators overview

    We briefly discuss accelerators.

• Week 2

  OpenMP

  • Getting started with OpenMP

    Over the course of this week, we will delve deeper into the concepts of OpenMP to provide us with pragmatic knowledge that will enable us to manoeuvre OpenMP more efficiently.

  • Worksharing directives

    Work share? No way.

  • Data environment and combined constructs

    Private and shared variables, Reduction clause, combined parallel and worksharing directives

  • Tasking

    Let's get to know the terms like tasking model and task's data environment while tackling some exercises.

• Week 3

  Message Passing Interface

  • Communicators and communication in MPI

    In this section we will learn about various ways of communication in MPI and the imperative we knowledge we need to use them.

  • Types of communication in MPI

    We will explore the various ways to classify different types of communication in MPI and learn how these different methods can be optimally used.

  • Collective communications

    In this section we will see some more advanced level of communications i.e communications in which more processes are involved.

  • Advanced Collective operations

    So far in the 'basic collective communication' we have encountered broadcast, scatter and gather. Now we can move on to more 'advanced collective communication' where we will cover routines MPI_Reduce and MPI_Allreduce.

• Week 4

  MPI Continued

  • Advanced communication in MPI

    In this subsection we will learn about some more advanced communication in MPI such as Non Blocking communications and One sided communication.

  • MPI + OpenMP

    In this section we will build up upon the introduction of OpenMP we did in the first two weeks and we will see how to include it into MPI.

  • User derived datatypes

    So far we have learnt communication of messages that were a continuous sequence of elements with mostly basic datatypes. In this section we will learn to communicate strided data and communicating various combined datatypes.

  • Parallel File I/O

    In this section we will learn about parallel I/O principles and how to employ them efficiently!

• Week 5

  Beyond OpenMP and MPI - GPU parallelisation

  • GPUs: Introduction and Architecture

    In this section we will introduce you to GPUs and their architecture. You will have a look at characteristics of professional and consumer-grade GPUs and compare them to the GPU available for you in a notebook session.

  • GPUs: Execution and programming models, Introduction to CUDA and OpenCL

    In this section we introduce you to GPU programming and execution models of CUDA and OpenCL.

  • CUDA and OpenCL step-by-step

    In this section you will get acquainted with CUDA and OpenCL step-by-step through the vector addition example on GPU.

  • Advanced examples on GPUs: Numerical integration

    In this section you will deepen your knowledge of CUDA and OpenCL programming with a more complex example of numerical computation: Riemann sum with trapeziums on GPU. Also OpenMP GPU off-loading is shortly presented.

  • Improving performance of GPU codes

    In the last section of this week you will learn how to improve GPU codes. Sum reduction for improving Riemann sum codes, as well as tools for profiling GPU codes, are discussed. Python scripts for GPUs are given as bonus material.

Who will you learn with?