Source: https://www.udemy.com/course/data-engineering-on-gcp-with-medallion-real-time-project/What you’ll learn
- Build a complete, end-to-end Data Engineering project on Google Cloud (GCP) from scratch.
- Architect a modern “Medallion Lakehouse” using GCS for the lake and BigQuery for the warehouse.
- Automate data workflows and build complex DAGs with Apache Airflow (Cloud Composer).
- Containerize Python (Pandas) data transformation scripts using Docker .
- Run scalable, serverless data processing jobs on Google Cloud Run .
- Process terabyte-scale data with Apache Spark (PySpark) on Dataproc clusters .
- Build a “Customer 360” view by joining multiple data sources for BI dashboards .
- Create event-driven, real-time pipelines that trigger on new files using Eventarc .
- Secure your entire data platform using Service Accounts and Cloud IAM roles.
- Master professional developer workflows with Git, VS Code, PyEnv, and venv .
- Connect a Data Lake (GCS) to a Data Warehouse (BigQuery) using External Tables .
- Use AI (Gemini) to accelerate development by generating Python code, SQL, and DAGs .
Requirements
- Basic understanding of Python programming: You should be familiar with variables, functions, lists and dictionaries.
- Basic knowledge of SQL: You should know how to write simple SELECT statements and understand what a JOIN is
- A computer (Windows, Mac, or Linux): The course provides setup instructions for all major operating systems.
- A Google Cloud (GCP) account: We will be building our entire project on GCP. A free-tier account is sufficient to follow along.
- A strong desire to learn data engineering! This is a hands-on, project-based course, so be ready to build.
Download Links
Password: cms.ddpanda.org
