Azure Data Factory – Data Engineering On Azure And Fabric

What you'll learn
Azure Data Factory
Microsoft Fabric Data Factory
Azure Storage Accounts
Azure SQL Databases and Upsert Operations
Overview and Background of Azure Big Data Analytics Solutions
Integrating Data From Various File Formats
Authentication and Access: Service Principals, Managed Identities and Azure Key Vault
Data Pipelines, Copy Activity, Data Flows, Control Flow and Transformation Activities
Implementing a Modern Data Lake
Control Flow, Parameters and Variables
Real World Application of Azure Data Factory
Azure DevOps Source Control with Data Factory
Continuous Integration and Continuous Deployment (CI/CD)
Requirements
Microsoft Azure Subscription
Basic SQL
Description
I am pleased to present this course on one of the most in demand data engineering tools around... Azure Data Factory!As the demand for cloud-based data integration services continues to skyrocket, there is a huge demand for professionals with knowledge of services like Azure Data Factory and Microsoft Fabric. By learning these applications, users can enhance their skills and increase their job prospects in the field of data engineering and analytics.In this course you will primarily be using Azure Data Factory on Microsoft Azure and Microsoft Fabric in addition to other services such as Azure Blob Storage, Azure Data Lake Storage Gen 2 and Azure SQL Database.The course is packed with lectures, code-along videos and a dedicated course project. As an added benefit you will also have lifetime access to all of the lectures.This course will cover the following topics:Azure Storage Solutions such as Azure Blob Storage and Azure Data Lake Gen2 StorageThe basics of Azure Data Factory including the core components such as Linked Services, Datasets, Activities, Data Flows, Pipelines and Integration RuntimesIntegrating data from various file formats such as CSV, JSON and ParquetThe Copy Activity in Azure Data FactoryData Flows, Control Flow and Transformation Activities in Azure Data FactoryOrchestrating Data Integration WorkflowsHow to create Schedules and Triggers to execute your pipelinesHow to use Parameters and Variables with your Linked Services, Datasets and PipelinesHow to use Azure Data Factory with SQL DatabasesAuthentication and Access including Managed Identities, Service Principals and Azure Key VaultThe Data Factory Experience in Microsoft FabricAzure DevOps Source ControlContinuous Integration and Continuous Deployment
Overview
Section 1: Course Overview and Set Up
Lecture 1 Course Overview
Lecture 2 Azure Big Data Analytics
Lecture 3 Azure Account Set Up
Lecture 4 Azure UI Overview and Creating Your Subscription
Lecture 5 Services and Regions
Lecture 6 Naming Conventions and Creating the Course Resource Group
Lecture 7 Cost Management and Pricing
Lecture 8 Links and Resources
Section 2: Azure Storage
Lecture 9 Azure Storage Solutions Overview
Lecture 10 Classification of Data
Lecture 11 ADF Supported File Formats
Lecture 12 Customer Orders Dataset Download and Overview
Lecture 13 Blueprint for the Data Lake
Lecture 14 Storage Account Creation
Lecture 15 Container Creation and File Upload
Lecture 16 Links and Resources
Section 3: ADF - Overview and Set Up
Lecture 17 Azure Data Factory Overview
Lecture 18 ADF Resource Creation
Lecture 19 ADF User Interface and Components Overview
Lecture 20 Links and Resources
Section 4: Data Movement
Lecture 21 Copy Activity Overview
Lecture 22 Section Objective
Lecture 23 Copy Activity - Order Items Part 1 (Linked Services and Source/Sink Datasets)
Lecture 24 Copy Activity - Order Items Part 2 (Copy Activity and Pipeline)
Lecture 25 Copy Activity - Customers JSON
Lecture 26 Copy Activity - Orders Parquet
Lecture 27 Copy Activity - Stores and Products JSON
Lecture 28 Organising Data Factory Objects in Folders
Lecture 29 Chaining Activities
Lecture 30 Working with Multiple Files as a Source and Copy Behaviour
Lecture 31 Links and Resources
Section 5: Data Transformation
Lecture 32 Data Flow Overview
Lecture 33 Section Objective
Lecture 34 Data Flow Demo and User Interface
Lecture 35 Data Flow Debug (Pricing)
Lecture 36 Data Flow Debug (Row Limit)
Lecture 37 Source and Sink Partitioning
Lecture 38 Select Transformation
Lecture 39 Cast Transformation
Lecture 40 Import Projection
Lecture 41 Derived Column Transformation
Lecture 42 Raw to Cleansed Data Flow Requirement
Lecture 43 Raw to Cleansed Data Flow Walkthrough
Lecture 44 Partitioned Datasets
Lecture 45 Filter and Sort Transformations
Lecture 46 Aggregate Transformation
Lecture 47 Join Transformation
Lecture 48 Conditional Split and Union Transformations
Lecture 49 Cleansed to Structured Data Flow Requirement
Lecture 50 Cleansed to Structured Data Flow Walkthrough
Lecture 51 Structured to Analytics Data Flow Requirement
Lecture 52 Structured to Analytics Data Flow Walkthrough
Lecture 53 Links and Resources
Section 6: Scheduling and Chaining Pipelines for Execution
Lecture 54 Execute Pipeline Activity
Lecture 55 Data Flow Debug Session vs Activity Runtime
Lecture 56 Schedule Triggers
Lecture 57 Storage Events Trigger
Lecture 58 Tumbling Window Triggers
Lecture 59 Links and Resources
Section 7: Control Flow, Parameters and Variables
Lecture 60 Data Lake Preparation
Lecture 61 Delete Activity
Lecture 62 Get Metadata Activity
Lecture 63 Lookup Activity
Lecture 64 If Condition and Switch Activities
Lecture 65 Introduction to Parameters in ADF
Lecture 66 Introduction to Variables in ADF
Lecture 67 ForEach Activity
Lecture 68 Metadata Based Approach (Example)
Lecture 69 Course Project Activity: Dynamic Datasets and Folder Structure
Lecture 70 Links and Resources
Section 8: Azure SQL Database for Structured Data
Lecture 71 Section Overview
Lecture 72 Creating our Azure SQL Resource
Lecture 73 Creating our Schema and Tables
Lecture 74 Adding a Unique Identifier Column to Structured Orders Data
Lecture 75 Creating Linked Service and Datasets for SQL
Lecture 76 Important Note Regarding the Dynamic Datasets
Lecture 77 Copy Activity with SQL Sink
Lecture 78 Data Flow with SQL Sink
Lecture 79 Course Project Activity: Storing our Structured Data in the SQL Database
Lecture 80 Links and Resources
Section 9: Tumbling Window Triggers
Lecture 81 Tumbling Window Triggers Overview
Lecture 82 Ingest Data for Specific Time Intervals
Lecture 83 Tumbling Window Trigger Dependencies
Lecture 84 Links and Resources
Section 10: Authentication and Access
Lecture 85 Overview
Lecture 86 Azure Key Vault
Lecture 87 System Assigned Managed Identities
Lecture 88 User Assigned Managed Identities
Lecture 89 Service Principals
Lecture 90 Fine Grained Access Control for Storage Accounts
Lecture 91 Links and Resources
Section 11: Data Factory in Microsoft Fabric
Lecture 92 Section Overview
Lecture 93 What is Microsoft Fabric?
Lecture 94 What is OneLake?
Lecture 95 What are Fabric Capacities?
Lecture 96 Set Up a Fabric Trial with a Business Email
Lecture 97 Set Up a Fabric Trial with a Personal Email
Lecture 98 Overview of Fabric UI and Project Workspace Creation
Lecture 99 Data Pipeline and Dataflow Gen2 in Data Factory
Lecture 100 Demo 1: Copy Data Between External ADLS Containers
Lecture 101 Overview of Data Lakehouses in Microsoft Fabric
Lecture 102 Demo 2: Copy Data from Azure to Fabric
Lecture 103 Demo 3: Dataflow Gen2
Lecture 104 (Optional) Semantic Modelling and Power BI Reporting
Lecture 105 Section Debrief
Lecture 106 Links and Resources
Section 12: CI/CD: Source Control with Azure DevOps Git
Lecture 107 Section Overview
Lecture 108 Overview of CI/CD in Azure Data Factory
Lecture 109 Azure Resource Set Up
Lecture 110 Azure DevOps Overview and Set Up
Lecture 111 Azure DevOps Parallelism Request
Lecture 112 Initialising and Configuring Git Repo in ADF
Lecture 113 ADF Git Workflow Demo 1: Feature Development
Lecture 114 ADF Git Workflow Demo 2: Merge Conflicts
Lecture 115 Section Debrief
Lecture 116 Links and Resources
Section 13: CI/CD: Automated Build and Deploying to Different Environments
Lecture 117 Section Overview
Lecture 118 Reminder to Complete Parallelism Request
Lecture 119 Automated Build Pipeline
Lecture 120 Enabling Release Pipelines in Azure DevOps
Lecture 121 Release Pipeline: Deploy to Dev
Lecture 122 ARM Template Deployment Limitations: Overview
Lecture 123 ARM Template Deployment Limitations: Adding Pre and Post Deployment Tasks
Lecture 124 Test and Prod Set Up in Azure
Lecture 125 Release Pipeline: Deploy to Test
Lecture 126 Using Pipeline Variables
Lecture 127 Release Pipeline: Deploy to Production
Lecture 128 End to End Walkthrough PART 1: CI/CD with ADF and Azure Data Lake Gen2
Lecture 129 End to End Walkthrough PART 2: CI/CD with ADF and Azure Data Lake Gen2
Lecture 130 Deleting Build and Release Pipelines
Lecture 131 Links and Resources
Section 14: Continue learning with me!
Lecture 132 BONUS: Check out my other courses
Anyone interested in Azure, Microsoft Fabric or Cloud Data Engineering,Anyone interested in working with Big Data,Anyone working with cloud data
Homepage

https://www.udemy.com/course/build-data-engineering-pipelines-with-azure-data-factory/