Apache Airflow Bootcamp Hands–On Workflow Automation

Step-by-Step Guide to Building and Managing Robust Workflows with Apache Airflow
What you'll learn:
Understand what Apache Airflow is, its purpose and pros and cons of using Airflow
Step-by-step guide to installing Airflow
Launch and navigate the Airflow Web UI and learn about various views: DAG, Grid, Graph, Calendar, Task Duration, Code, Variable and Gantt View
Understand what a DAG is and how to create a DAG definition file and different methods for DAG creation
Learn about DAG Run, default_arguments, and DAG arguments and Master scheduling concepts such as depends_on_past, wait_for_downstream, catchup, and backfill
Use the Airflow CLI for various operations and access a handy cheatsheet for quick reference
Understand tasks, task instances and Learn the lifecycle of a task
Master different operators including BashOperator, PostgresOperator, PythonOperator, SqliteOperator, and EmailOperator
Implement sensors like FileSensor, SQLSensor, TimeDeltaSensor, and TimeSensor
Apply branching logic with BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, and ShortCircuitOperator
Manage DAG dependencies and use TaskGroups ,Utilize TriggerDagRunOperator , ExternalTaskSensor and use hooks such as PostgresHook and SHook
Manage resources with pools and task priorities
Learn about different types of executors: SequentialExecutor and LocalExecutor and learn the Transition from SequentialExecutor to LocalExecutor
Explore the Airflow metadata database and Manage roles and create users with different roles including admin, public, user, and operator roles
Set and manage task-level and DAG-level SLAs and handle SLA misses
Address issues like zombie tasks, SIGTERM, and SIGKILL errors
Requirements:
Knowledge of Python
Rest we will cover to learn Airflow from scratch
Familiarity with command-line interfaces.
Understanding of database concepts is a plus but not required
Description:
Hello and welcome to the Master Apache Airflow: Guide to Workflow Automation with Practical Examples! Throughout my career, I've built and managed countless workflows using Apache Airflow, and I'm excited to share my knowledge with you.This course is designed to take you from a complete beginner to a confident user of Apache Airflow. We'll cover everything from installation to advanced features, and you'll get hands-on experience through practical examples and real-world projectsWhat's included in the course ?Introduction to AirflowUnderstanding the purpose and benefits of using Apache Airflow.Pros and cons of adopting Airflow in your projects.Airflow ArchitectureA detailed look into the components that make up Airflow.Key terminology used in Airflow.Configuration and InstallationThe role and configuration of the airflow.cfg file.Step-by-step guide to installing Airflow.Airflow Web UI ViewsLaunching and navigating the Airflow Web UI.DAG ViewGrid ViewGraph ViewCalendar ViewTask Duration ViewCode ViewVariable ViewGantt ViewDAGs (Directed Acyclic Graphs)What is a DAG?Creating a DAG definition file.Different methods for DAG creation.Understanding DAG Run, default_arguments, and DAG arguments.Using parameters in DAGs and passing parameters through TriggerDagRunOperator.Scheduling concepts including depends_on_past, wait_for_downstream, catchup, and backfill.Airflow CLI and CheatsheetUtilizing the Airflow CLI for various operations.Handy cheatsheet for quick reference.Tasks in AirflowWhat are tasks and task instances?The lifecycle of a task. Operators in AirflowDetailed exploration of operators including BashOperator, PostgresOperator, PythonOperator, SqliteOperator, and EmailOperator.SensorsUsing sensors like FileSensor, SQLSensor, TimeDeltaSensor, and TimeSensor.BranchingImplementing branching logic with BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, and ShortCircuitOperator.DAG Dependencies and TaskGroupsManaging DAG dependencies and using TaskGroups.Using TriggerDagRunOperator and ExternalTaskSensor.HooksUnderstanding and using hooks such as PostgresHook and SHook.Resource ManagementManaging resources with pools and task priorities.Executors in AirflowDifferent types of executors: SequentialExecutor and LocalExecutor.Transitioning from SequentialExecutor to LocalExecutor.Airflow Metadata Database and RolesUnderstanding the Airflow metadata database.Managing roles: creating users with different roles, including admin, public, user, and operator roles.Creating custom roles and modifying existing ones.SLA (Service Level Agreement)Setting and managing task-level and DAG-level SLAs.Handling SLA misses.Advanced ConceptsUsing XComs for inter-task communication.Configuring .airflowignore file.Implementing TriggerRule and setting up task dependencies.Retrieving context parameters and using callback functions.Dealing with zombie tasks, SIGTERM, and SIGKILL errors.I believe that mastering workflow automation with Airflow can open up incredible opportunities in the field of data engineering. I've seen firsthand how it can transform the way we handle data, and I can't wait to see what you'll achieve with these skills.So, whether you're looking to advance your career, work on more efficient data pipelines, or just curious about Airflow, you're in the right place. Let's dive in and start creating some amazing workflows together. Are you ready? Let's get started!I wish you a great success!
Who this course is for:
Data Engineers: Data engineers who are responsible for building and managing data pipelines can greatly benefit from learning Apache Airflow.
Data Scientists: Data scientists who work with large datasets and perform data analysis can leverage Apache Airflow to automate repetitive tasks, such as data preprocessing, model training, and evaluation
DevOps Engineers: DevOps engineers who are responsible for managing and automating infrastructure can use Apache Airflow to automate deployment processes, monitor system health, and trigger actions based on predefined conditions
Software Developers: Software developers who build and maintain software applications can use Apache Airflow to automate various tasks, such as data ingestion, data processing, and workflow orchestration
Homepage

https://www.udemy.com/course/apache-airflow-bootcamp-hands-on-workflow-automation/