• Partnership for Advanced Computing in Europe (PRACE)

Introduction to Parallel Programming

Discover parallel programming fundamentals in this beginner programming course, covering OpenMP, MPI, OpenCL, and CUDA.

487 enrolled on this course

A pen pointing to a parallel code example

Get an introduction to parallel programming using C and similar languages

This course gives beginner programmers an introduction to parallel programming.

Parallel programming describes the breaking down of a larger problem into smaller steps. Instructions are delivered to multiple processors, which will execute necessary calculations in parallel – hence the name.

On this course, you will explore the fundamentals of parallel programming using C, C++, Python, Fortran, Cuda/OpenCL or similar programming languages.

Learn to use OpenMP and MPI for parallel programming

On this course, you will:

  • Learn how to use OpenMP (Open Multi-Processing) for incremental parallelisation within a single multicore shared memory paradigm
  • Learn how to use the MPI (Message Passing Interface) message-passing standard to control communication between two processes, subroutines, or functions within a program
  • Get an introduction to the OpenMPI library project, used by TOP500 supercomputers, which aims to create a single world-class open source MPI implementation based on best practice
  • Learn how to express numerical problems in parallel programming paradigms, gaining an awareness of potential design and performance pitfalls in heterogeneous architectures Get an introduction to hybrid and accelerated paradigms (Cuda, OpenCL)

This course is delivered by the LECAD Laboratory at the University of Ljubljana, as part of PRACE (Partnership for Advanced Computing in Europe). The LECAD Laboratory emphasises the teaching of parallel programming alongside leading European HPC (high performance computing) centres.

Download video: standard or HD

Skip to 0 minutes and 8 seconds Traditionally, computers were only focused on serial computing. These programs were divided into distinct instructions that were executed one by one by a processor. In parallel programming, however, a single program is broken up into independent parts that are each executed by their own processor, and the results obtained from the individual parts are then merged. This course gives beginner programmers an introduction to parallel and heterogeneous programming. It covers parallel programming fundamentals with OpenMP, MPI, CUDA and OpenCL. It also provides you with a set of interactive practical examples using different platforms and programming languages, like C, Python and Fortran. You learn how to express numerical problems in parallel programming paradigms, gaining an awareness of potential design and performance pitfalls in heterogeneous architectures.

Skip to 1 minute and 1 second Join us and discover the world of parallel programming.

Syllabus

  • Week 1

    Introduction to parallel programming

    • Intro to parallel programming

      Here you will find out what we have in store for you over the next few weeks and what are some different programming models.

    • OpenMP overview

      We briefly discuss OpenMP.

    • MPI overview

      We briefly discuss MPI.

    • Accelerators overview

      We briefly discuss accelerators.

  • Week 2

    OpenMP

    • Getting started with OpenMP

      Over the course of this week, we will delve deeper into the concepts of OpenMP to provide us with pragmatic knowledge that will enable us to manoeuvre OpenMP more efficiently.

    • Worksharing directives

      Work share? No way.

    • Data environment and combined constructs

      Private and shared variables, Reduction clause, combined parallel and worksharing directives

    • Tasking

      Let's get to know the terms like tasking model and task's data environment while tackling some exercises.

  • Week 3

    Message Passing Interface

    • Communicators and communication in MPI

      In this section we will learn about various ways of communication in MPI and the imperative we knowledge we need to use them.

    • Types of communication in MPI

      We will explore the various ways to classify different types of communication in MPI and learn how these different methods can be optimally used.

    • Collective communications

      In this section we will see some more advanced level of communications i.e communications in which more processes are involved.

    • Advanced Collective operations

      So far in the 'basic collective communication' we have encountered broadcast, scatter and gather. Now we can move on to more 'advanced collective communication' where we will cover routines MPI_Reduce and MPI_Allreduce.

  • Week 4

    MPI Continued

    • Advanced communication in MPI

      In this subsection we will learn about some more advanced communication in MPI such as Non Blocking communications and One sided communication.

    • MPI + OpenMP

      In this section we will build up upon the introduction of OpenMP we did in the first two weeks and we will see how to include it into MPI.

    • User derived datatypes

      So far we have learnt communication of messages that were a continuous sequence of elements with mostly basic datatypes. In this section we will learn to communicate strided data and communicating various combined datatypes.

    • Parallel File I/O

      In this section we will learn about parallel I/O principles and how to employ them efficiently!

  • Week 5

    Beyond OpenMP and MPI - GPU parallelisation

    • GPUs: Introduction and Architecture

      In this section we will introduce you to GPUs and their architecture. You will have a look at characteristics of professional and consumer-grade GPUs and compare them to the GPU available for you in a notebook session.

    • GPUs: Execution and programming models, Introduction to CUDA and OpenCL

      In this section we introduce you to GPU programming and execution models of CUDA and OpenCL.

    • CUDA and OpenCL step-by-step

      In this section you will get acquainted with CUDA and OpenCL step-by-step through the vector addition example on GPU.

    • Advanced examples on GPUs: Numerical integration

      In this section you will deepen your knowledge of CUDA and OpenCL programming with a more complex example of numerical computation: Riemann sum with trapeziums on GPU. Also OpenMP GPU off-loading is shortly presented.

    • Improving performance of GPU codes

      In the last section of this week you will learn how to improve GPU codes. Sum reduction for improving Riemann sum codes, as well as tools for profiling GPU codes, are discussed. Python scripts for GPUs are given as bonus material.

When would you like to start?

Start straight away and join a global classroom of learners. If the course hasn’t started yet you’ll see the future date listed below.

  • Available now

Learning on this course

On every step of the course you can meet other learners, share your ideas and join in with active discussions in the comments.

What will you achieve?

By the end of the course, you‘ll be able to...

  • Explore parallel programming fundamentals, covering OpenMP and MPI
  • Experiment how to use OpenMP for incremental parallelisation within a single multicore shared memory paradigm
  • Experiment how to use MPI message-passing standard to control communication between two processes, subroutines, or functions within a program
  • Solve numerical problems in parallel programming paradigms, gaining an awareness of potential design and performance pitfalls in heterogeneous architectures
  • Explore hybrid and accelerated paradigms (Cuda, OpenCL)
  • Hands-on practice with interactive examples using different platforms and programming languages

Who is the course for?

This course is part of an introduction to programming from PRACE, targeting beginner programmers. It follows on from the Supercomputing course.

This course is hands-on. Some knowledge of C, C++, Fortran, Python, or similar programming languages would be useful.

Who will you learn with?

Leon Kos is a 25+ years
veteran of using Linux desktop on a daily basis to build digital
relationships for research, teaching, and getting the job done by programming.

Matic Brank is a researcher from the Faculty of Mechanical Engineering at the University of Ljubljana. His area of research are numerical simulations in heat transfer and plasma physics.

A programmer with a keen interest in machine learning applications (from small to big data) currently researching CPU, GPU and hybrid platforms for parallel applications in plasma physics.

Kim Badovinac is a programmer and a Master's degree student of Computer science at University of Ljubljana.

Who developed the course?

Partnership for Advanced Computing in Europe (PRACE)

The Partnership for Advanced Computing in Europe (PRACE) is an international non-profit association with its seat in Brussels.

Recommended

Ways to learn

Buy this course

Subscribe to Unlimited

Join with limited access

Choose the best way to learn for you!

$74

One-off payment

$27.99/month

Automatically renews

Free

Unlock your expertiseAccess thousands of coursesTry this course - with limits
Access to this courseticktickLimited to 5 weeks
Access to ALL eligible coursescrosstickcross
Learn at your own paceticktickcross
Discuss your learning in commentstickticktick
Tests to check your learningticktickcross
Certificate when you're eligiblePrinted and digitalDigital onlycross
Buy this course

Ways to learn

Subscribe to Unlimited to access this course and thousands of other short courses or buy just this course for extra benefits. Or, join with limited access.

Subscribe to Unlimited

$27.99/month

Automatically renews

Access thousands of courses

  • Access to this course
  • Access to ALL eligible short courses
  • Learn at your own pace
  • Discuss your learning in comments
  • Tests to boost your learning
  • Digital certificate when you're eligible

Buy this course

$74

One-off payment

Unlock your expertise

  • Access to this course
  • Learn at your own pace
  • Discuss your learning in comments
  • Tests to boost your learning
  • Printed and digital certificate when you’re eligible

Join with limited access

Free

Try before you buy

  • Limited access to course content for 5 weeks

Find out more about certificates, Unlimited or buying a course (Upgrades)

Learning on FutureLearn

Your learning, your rules

  • Courses are split into weeks, activities, and steps to help you keep track of your learning
  • Learn through a mix of bite-sized videos, long- and short-form articles, audio, and practical activities
  • Stay motivated by using the Progress page to keep track of your step completion and assessment scores

Join a global classroom

  • Experience the power of social learning, and get inspired by an international network of learners
  • Share ideas with your peers and course educators on every step of the course
  • Join the conversation by reading, @ing, liking, bookmarking, and replying to comments from others

Map your progress

  • As you work through the course, use notifications and the Progress page to guide your learning
  • Whenever you’re ready, mark each step as complete, you’re in control
  • Complete 90% of course steps and all of the assessments to earn your certificate

Want to know more about learning on FutureLearn? Using FutureLearn

Do you know someone who'd love this course? Tell them about it...

You can use the hashtag #FLparallelprogramming to talk about this course on social media.