• FutureLearn logo

Introduction to Data Engineering with Microsoft Azure 2

Further your knowledge of Microsoft Azure services and improve your data engineering skills.

A woman with glasses sits gazing at her computer screen surrounded by code, whilst holding a pencil.

Introduction to Data Engineering with Microsoft Azure 2

  • 6 weeks

  • 4 hours per week

  • Digital certificate when eligible

  • Introductory level

Find out more about how to join this course

Prepare for the DP-203: Data Engineering on Microsoft Azure exam

Microsoft logo This course has been created in partnership with Microsoft.

Building on your learning from Introduction to Data Engineering with Microsoft Azure 1, this course will develop your understanding of data engineering processes in Microsoft Azure, further preparing you to take the DP-203 exam and kickstart your career in data engineering.

Explore data services within Microsoft Azure

Using Azure data services and tools, you’ll be able to implement, develop, and optimise data storage, processing and security operations within your organisation.

You’ll be introduced to tools including Azure Synapse, Databricks and Azure Data Lake Storage, learning how each can improve and streamline your processes.

Design hybrid transactional and analytical processing (HTAP) patterns

As businesses continue to move to digital processes, they recognise the value of making faster, well-informed decisions and the impact this can have on gaining a competitive advantage.

You’ll be guided through HTAP architecture and learn how to design HTAP using Azure Synapse Analytics.

With this knowledge, you’ll be able to run analytics in near-real-time, giving you the ability to respond to opportunities at speed.

Discover data operations in Azure Databricks

Azure Databricks, a cloud-based big data and machine learning platform, empowers developers by simplifying enterprise-grade data application production.

You’ll identify the advantages of Azure Databricks over other Big Data platforms, and learn how to spend more time building apps and less time managing infrastructure.

You’ll finish this course understanding how Microsoft Azure can be used to optimise data engineering operations. Having completed both courses, you’ll be equipped to take the DP-203 exam and develop a career as a data professional.

Syllabus

  • Week 1

    Work with Hybrid Transactional and Analytical Processing Solutions using Azure Synapse Analytics

    • Plan hybrid transactional and analytical processing using Azure Synapse Analytics

      In this activity, you will learn about planning hybrid transactional and analytical processing using Azure Synapse Analytics.

    • Configure Azure Synapse Link with Azure Cosmos DB

      During this week, you will learn how to configure Azure Synapse link with Azure Cosmos DB.

    • Query Azure Cosmos DB with Apache Spark for Azure Synapse Analytics

      In this activity, you will learn how to query Azure Cosmos DB with Apache Spark for Azure Synapse Analytics.

    • Query Azure Cosmos DB with SQL Serverless for Azure Synapse Analytics

      In this activity, you will learn about querying Azure Cosmos DB with SQL Serverless for Azure Synapse Analytics.

  • Week 2

    Data engineering with Azure Databricks Part 1

    • Describe Azure Databricks

      In this activity, you will describe Azure Databricks.

    • Spark architecture fundamentals

      In this activity, you will learn about spark architecture fundamentals.

    • Read and write data in Azure Databricks

      During this week, you will learn about reading and writing data in Azure Databricks.

    • Work with DataFrames in Azure Databricks

      During this week, you will learn how to work with DataFrames in Azure Databricks.

    • Work with DataFrames columns in Azure Databricks

      During this week, you will learn how to work with DataFrames columns in Azure Databricks.

  • Week 3

    Data engineering with Azure Databricks Part 2

    • Describe lazy evaluation and other performance features in Azure Databricks

      During this week, you will learn how to describe lazy evaluation and other performance features in Azure Databricks.

    • Work with DataFrames advanced methods in Azure Databricks

      In this activity, you will learn how to work with DataFrames advanced methods in Azure Databricks.

    • Describe platform architecture, security, and data protection in Azure Databricks

      During this week, you will learn how to describe platform architecture, security, and data protection in Azure Databricks.

    • Build and query a Delta Lake

      Learn how to use Delta Lake to create, append, and upsert data to Apache Spark tables, taking advantage of built-in reliability and optimizations.

    • Process streaming data with Azure Databricks structured streaming

      Learn how Structured Streaming helps you process streaming data in real time, and how you can aggregate data over windows of time.

  • Week 4

    Data engineering with Azure Databricks Part 3

    • Describe Azure Databricks Delta Lake architecture

      Use Delta Lakes as an optimization layer on top of blob storage to ensure reliability and low latency within unified Streaming and batch data pipelines.

    • Create production workloads on Azure Databricks with Azure Data Factory

      Azure Data Factory helps you create workflows that orchestrate data movement and transformation at scale. Integrate Azure Databricks into your production pipelines by calling notebooks and libraries.

    • Implement CI/CD with Azure DevOps

      CI/CID isn't just for developers. Learn how to put Azure Databricks notebooks under version control in an Azure DevOps repo and build deployment pipelines to manage your release process.

    • Integrate Azure Databricks with Azure Synapse

      Azure Databricks is just one of many powerful data services in Azure. Learn how to integrate with Azure Synapse Analytics as part of your data architecture.

    • Describe Azure Databricks best practices

      Learn best practices for workspace administration, security, tools, integration, databricks runtime, HA/DR, and clusters in Azure Databricks.

  • Week 5

    Large-Scale Data Processing with Azure Data Lake Storage Gen2

    • Introduction to Azure Data Lake storage

      Learn how Azure Data Lake Storage provides a cloud storage service that is highly available, secure, durable, scalable, and redundant and brings new efficiencies to processing big data analytics workloads.

    • Upload data to Azure Data Lake Storage

      Learn various ways to upload data to Data Lake Storage Gen 2. Upload data through the Azure portal, Azure Storage Explorer, or .NET. Or copy the data in Azure Data Factory.

    • Secure your Azure Storage account

      Learn how Azure Storage provides multilayered security to protect your data. Find out how to use access keys, to secure networks, and to use Advanced Threat Protection to proactively monitor your system.

  • Week 6

    Implement a Data Streaming Solution with Azure Streaming Analytics

    • Work with data streams by using Azure Stream Analytics

      Explore how Azure Stream Analytics integrates with your applications or Internet of Things (IoT) devices to gain insights with real-time streaming data. Learn how to consume and analyze data streams and derive actionable results.

    • Enable reliable messaging for Big Data applications using Azure Event Hubs

      Connect sending and receiving applications with Event Hubs so you can handle extremely high loads without losing data.

    • Ingest data streams with Azure Stream Analytics

      Learn how to create Azure Stream Analytics jobs to process input data, transform it with a query, and return results.

When would you like to start?

Start straight away and join a global classroom of learners. If the course hasn’t started yet you’ll see the future date listed below.

  • Available now

Learning on this course

On every step of the course you can meet other learners, share your ideas and join in with active discussions in the comments.

What will you achieve?

By the end of the course, you‘ll be able to...

  • Explain Hybrid Transactional and Analytical Processing Solutions using Azure Synapse Analytics
  • Investigate Data engineering with Azure Databricks
  • Explore Large-Scale Data Processing with Azure Data Lake Storage Gen2
  • Apply a Data Streaming Solution with Azure Streaming Analytics

Who is the course for?

This course is designed for data professionals preparing for the DP 203: Data Engineering on Microsoft Azure exam.

Before taking this course, learners should take Introduction to Data Engineering with Microsoft Azure 1 to ensure they have covered all topics required for the DP 203 exam.

It’s recommended that you already have a solid understanding of data processing languages, as well as parallel processing and data architecture patterns before taking the exam.

Who developed the course?

FutureLearn

FutureLearn is a leading social learning platform and has been providing high quality online courses for learners around the world over the last ten years.

Ways to learn

Buy this course

Subscribe & save

Limited access

Choose the best way to learn for you!

$134/one-off payment

$27.99

For the first two months. Automatically renews

Free

Fulfill your current learning needDevelop skills to further your careerSample the course materials
Access to this courseticktick

Access expires 9 May 2024

Access to 1,000+ coursescrosstickcross
Learn at your own paceticktickcross
Discuss your learning in commentstickticktick
Certificate when you're eligiblePrinted and digitalDigital onlycross
Continue & Upgrade

Cancel for free anytime

Ways to learn

Choose the best way to learn for you!

Subscribe & save

$27.99

For the first two months. Automatically renews

Develop skills to further your career

  • Access to this course
  • Access to 1,000+ courses
  • Learn at your own pace
  • Discuss your learning in comments
  • Digital certificate when you're eligible

Cancel for free anytime

Buy this course

$134/one-off payment

Fulfill your current learning need

  • Access to this course
  • Learn at your own pace
  • Discuss your learning in comments
  • Printed and digital certificate when you’re eligible

Limited access

Free

Sample the course materials

  • Access expires 9 May 2024

Find out more about certificates, Unlimited or buying a course (Upgrades)

T&Cs apply.

Learning on FutureLearn

Your learning, your rules

  • Courses are split into weeks, activities, and steps to help you keep track of your learning
  • Learn through a mix of bite-sized videos, long- and short-form articles, audio, and practical activities
  • Stay motivated by using the Progress page to keep track of your step completion and assessment scores

Join a global classroom

  • Experience the power of social learning, and get inspired by an international network of learners
  • Share ideas with your peers and course educators on every step of the course
  • Join the conversation by reading, @ing, liking, bookmarking, and replying to comments from others

Map your progress

  • As you work through the course, use notifications and the Progress page to guide your learning
  • Whenever you’re ready, mark each step as complete, you’re in control
  • Complete 90% of course steps and all of the assessments to earn your certificate

Want to know more about learning on FutureLearn? Using FutureLearn

Do you know someone who'd love this course? Tell them about it...