Real Time Project on Retail Data Using PySpark ,SQL, Delta/Delta Live Table, Unity Catalogue, Auto Loader and Streaming

What you'll learn

Medallion Architecture , Dimensional Data Modelling Design , DeltalakeHouse Design , Spark Core Architecture , Unity Catalogue Setup , Spark Cluster Setup
PySpark Dataframe Reader , Writer , Transformation Functions , Action Functions , DateTime Functions , Aggregation Functions , Dataframe Joins , Complex Data
Spark SQL External Tables , Managed Tables , Delta Lake Tables , Create Table As Script(CTAS) , Temp Views , Table Joins , Data Transformation Functions
Four Reusable Ingestion Pipelines To Ingest Source Data From Web(HTTP) Service , Database Tables , API Source Systems , Incremental Loading & Job Scheduling
Seven Data Transformation Pipelines to process source data in Silver & Gold Layers and Build Reporting Database And Datalake With Change Data Capture
Spark Streaming Reader & Writer Configuration To Process Real Time Steaming Data , CHECKPOINTLOCATION setup for automated Incremental Loading in Streaming Data
Delta Live Tables - Materialised Views , Streaming Tables setup , Delta Live Table Pipeline Configuration , Data Quality Checks , AUTOLOADER and APPLY CHANGES
Monitoring And Logging Setup To Monitor Production Job Runs, Setup Alerts for Job Failure and Extended Logging of Job Runs and Service Metrics
Security Settings in Azure using Microsoft Entra ID , IAM Role Based Access Control(RBAC) And Databricks Workspace Admin Settings
Configure Github Repository , Git Repos Folders in Databricks Workspace , Ways of Working with Git branches , Merging Code & PULL requests
Setup Production Environment , CI/CD Pipeline to automate Code Deployment Using GitHub Actions

Requirements

None , Course Includes all of the Basic Python Skils and SQL skills necessary to develop the code

Description

By Completing this course you will be equipped with below Data Engineer Roles & Responsibilities in the real time project• Designing and Developing Databricks(PySpark) Notebooks to Ingest the data from Web(HTTP) Services• Designing and Developing Databricks(PySpark) Notebooks to Ingest the data from SQL Databases• Designing and Developing Databricks(PySpark) Notebooks to Ingest the data from API source Systems• Designing and Developing Spark SQL External and Managed Tables• Developed Databricks Spark SQL Reusable Notebooks To Create Delta Lake Tables• Developed Databricks SQL Code to populate Reporting Dimension tables• Developed Databricks SQL Code to populate Reporting Fact Table• Designing and Developing Databricks(PySpark ) Notebooks to Process and Flatten Semi Structured JSON Data • Designing and Developing Databricks(PySpark ) Notebooks to Integrate Data and load into Datalake Gold Layer• Designing and Developing Databricks(PySpark) Notebooks to Process Semi Structured JSON Data in DataLake Silver Layer• Designing and Developing Databricks(SQL) Notebooks to Integrate Data and load into Datalake Gold Layer• Designing and Configuring Unity Catalogue for Better Access Control & Connecting to External Data Stores• Developed Databricks Jobs for Scheduling the Data Ingestion and Transformation Notebooks• Designing and Configuring Delta Live Tables in all layers for seamless Data Integration• Setup Azure Monitor and Log Analytics for Automated Monitoring of Job Runs and Stored Extended Log Details• Setup Azure Key Vault and Configure Key Vault Backed Secret Scopes in Databricks Workspace• Configuring GitHub Repository and creating Git Repo Folders in Databricks Workspace• Designing and Configuring CI/CD Pipelines to release the code into multiple environment

Who this course is for

Anyone Interested in Learn and Apply For Data Engineering Jobs
Homepage:

https://www.udemy.com/course/azure-databricks-data-engineering-with-real-time-project/