Data Engineering · Azure Data Engineering
Build Data Pipelines on Microsoft Azure
Learn how to design, build and operate data pipelines on Microsoft Azure, from ingestion with Azure Data Factory to large-scale processing with Azure Databricks and PySpark.
- Azure Data Factory pipelines
- Azure Databricks and PySpark
- Azure Data Lake Storage
- Orchestration, monitoring and optimisation
Course overview
About This Course
Azure Data Engineering covers the Microsoft Azure services used to ingest, store, process and serve data, and how to combine them into reliable pipelines.
Many organisations run their data platforms on Azure. Data engineers who can build pipelines with Azure Data Factory, process data at scale with Databricks and organise data in a data lake are in demand across industries.
You will learn how the Azure data platform fits together, how to build and orchestrate pipelines, how to transform data with PySpark, how to model data for analytics, and how to monitor and tune your workloads.
This course follows SQL and Python for Data Engineering in our Data Engineering learning paths. AWS Data Engineering covers the equivalent skills on Amazon Web Services.
Who it is for
Who Should Take This Course?
-
Learners with SQL and Python
You have the foundations and want to apply them on a cloud data platform.
-
ETL and BI developers
You build data integration or reporting solutions and want to move them to Azure.
-
Azure professionals moving into data
You work with Azure and want to specialise in its data services.
-
Aspiring data engineers
You want hands-on experience with a widely used cloud data platform.
Learning outcomes
What You Will Learn
By working through the course, you will build towards being able to:
Describe the main Azure data services and how they fit into a modern data platform.
Organise raw, cleaned and curated data in Azure Data Lake Storage.
Build Azure Data Factory pipelines with linked services, datasets, activities and triggers.
Transform large datasets with PySpark in Azure Databricks.
Design data models suited to analytics and reporting.
Orchestrate, schedule and monitor pipelines, and handle failures.
Apply performance and cost optimisation practices to data workloads.
Curriculum
Course Curriculum
The course is organised into modules. Session order and depth may vary by training arrangement.
-
Module 1: Azure data platform fundamentals
The Azure services that make up a modern data platform and how they connect.
Topics covered include
Practical activity: Sketch an Azure data platform architecture for a sample business.
-
Module 2: Azure Data Lake Storage
Store and organise data at every stage of processing.
Topics covered include
Practical activity: Design a data lake folder structure for raw, cleaned and curated data.
-
Module 3: Azure Data Factory
Ingest and move data with managed pipelines.
Topics covered include
Practical activity: Build a pipeline that copies sample data from a source into the data lake.
-
Module 4: Azure Databricks and PySpark
Process large datasets with Spark on Azure.
Topics covered include
Practical activity: Transform raw sample data into a cleaned Delta table with PySpark.
-
Module 5: Data modelling for analytics
Shape data so it is easy and fast to analyse.
Topics covered include
Practical activity: Model a sample sales dataset as a star schema.
-
Module 6: Orchestration, monitoring and optimisation
Run data pipelines reliably and efficiently in production.
Topics covered include
Practical activity: Add monitoring and incremental loading to a sample end-to-end pipeline.
Practical learning
Hands-On Learning
Practical work builds a sample data platform step by step. Examples of activities include:
Hands-on work on Azure needs access to an Azure environment. Whether an environment is provided, or you use your own account, will be explained when you enquire.
- Designing a data lake structure
- Building Azure Data Factory ingestion pipelines
- Transforming data with PySpark notebooks
- Monitoring and optimising a sample pipeline
Before you start
Prerequisites and Learning Requirements
-
Recommended background
Working knowledge of SQL and basic Python. Our SQL and Python for Data Engineering courses cover these.
-
Technical prerequisites
Familiarity with basic cloud concepts is helpful but not required.
-
Software and tools
A computer with a reliable internet connection for live online sessions. Access to an Azure environment is needed for hands-on work; the options will be confirmed when you enquire.
-
Prior experience
Not intended as a first course for complete beginners in data.
Learning Format
IT Champion Hub offers live, instructor-led online training and workshops. Formats, duration and schedules vary by course and are not the same for every program.
- Live online sessions led by an instructor
- Practical activities alongside explanations
- Upcoming batches and schedules shared on enquiry
Enquire to find out the formats and upcoming batches available for this course. Enquire about this course
FAQ
Frequently Asked Questions
Do I need an Azure account?
Hands-on work needs access to an Azure environment. We'll explain the options, and any costs involved, when you enquire.
What should I know before starting?
You should be comfortable writing SQL queries and basic Python. Our SQL and Python for Data Engineering courses are the recommended preparation.
Is this an Azure certification course?
No. The course focuses on practical Azure data engineering skills. It is not an official Microsoft certification course.
Are there practical exercises?
Yes. You will build and run sample pipelines with Azure Data Factory and Azure Databricks throughout the course.
How can I enquire about upcoming training?
Select Enquire About This Course. The enquiry form will note that you are asking about Azure Data Engineering, and we'll get back to you with upcoming batches and available schedules.
Ready to Explore This Learning Path?
Have questions about the curriculum, prerequisites, or training arrangements? Get in touch with IT Champion Hub to discuss your learning goals.