Skip main navigation

Batch Processing in Azure

In this step, we'll explore some tools and considerations for batch processing data.

In this step, we’ll explore some tools and considerations for batch processing data, which is often necessary when working on large historical stores such as a data warehouse.

The purpose of the historical store is to track and establish trends, and often this process can be augmented through additional data from elsewhere. In the scenario in the video, Graeme uses the log file from the company’s website to relate the website’s traffic to the historical data stored within the data warehouse.

Azure Data Factory

Azure Data Factory is Azure’s cloud Extract, Transform, Load (ETL) service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring, monitoring, and management–all within the same window.

Azure Data Lake Analytics

Azure Data Lake Analytics allows us to easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python, and .NET. With no infrastructure to manage, you can process data on-demand, scale instantly, and only pay per job.

Analytics Output

There are a few ways to present the output of your analytics. Two options presented are to combine the results in an Excel Pivot Table, or to upload to Power BI, allowing you to add visualisations. This is particularly useful for presenting information generated from data in a succinct way to the relevant stakeholders on a project.

Learn More about SQL Server stored procedures, which allow you to store regular SQL code and call upon it simply when needed.

In the next step, we’ll look at real-time processing of data in the Azure environment, which is beneficial for data in motion.

This article is from the free online

Microsoft Future Ready: Fundamentals of Big Data

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education