Data Engineering on GCP with Medallion Real Time Project

Source: https://www.udemy.com/course/data-engineering-on-gcp-with-medallion-real-time-project/

What you’ll learn

Build a complete, end-to-end Data Engineering project on Google Cloud (GCP) from scratch.
Architect a modern “Medallion Lakehouse” using GCS for the lake and BigQuery for the warehouse.
Automate data workflows and build complex DAGs with Apache Airflow (Cloud Composer).
Containerize Python (Pandas) data transformation scripts using Docker .
Run scalable, serverless data processing jobs on Google Cloud Run .
Process terabyte-scale data with Apache Spark (PySpark) on Dataproc clusters .
Build a “Customer 360” view by joining multiple data sources for BI dashboards .
Create event-driven, real-time pipelines that trigger on new files using Eventarc .
Secure your entire data platform using Service Accounts and Cloud IAM roles.
Master professional developer workflows with Git, VS Code, PyEnv, and venv .
Connect a Data Lake (GCS) to a Data Warehouse (BigQuery) using External Tables .
Use AI (Gemini) to accelerate development by generating Python code, SQL, and DAGs .

Requirements

Basic understanding of Python programming: You should be familiar with variables, functions, lists and dictionaries.
Basic knowledge of SQL: You should know how to write simple SELECT statements and understand what a JOIN is
A computer (Windows, Mac, or Linux): The course provides setup instructions for all major operating systems.
A Google Cloud (GCP) account: We will be building our entire project on GCP. A free-tier account is sufficient to follow along.
A strong desire to learn data engineering! This is a hands-on, project-based course, so be ready to build.