Data Pipeline Development Design, build, and maintain scalable batch and real-time data pipelines using ETL/ELT patterns. Ingest and integrate structured, semi-structured, and unstructured data from diverse sources, including APIs, databases, files, and event streams.Implement incremental loading, change data capture (CDC), and idempotent pipeline design.
Implement CDC pipelines using Datastream to replicate data from operational databases into BigQuery. Build batch and streaming pipelines with Apache Beam on Dataflow, using Pub/Sub for event ingestion. Page 1 of 4 Data Modeling and Architecture Design and optimize data models (dimensional modeling, star/snowflake schemas, Data Vault where relevant) for analytics and reporting.
Build and manage data warehouses and lakehouses following the Medallion architecture (Bronze, Silver, Gold layers), ideally on BigQuery and Cloud Storage. Design BigQuery datasets using partitioning, clustering, materialized views, and authorized views for performance and secure sharing. Work with modern table formats such as Delta Lake, Apache Iceberg (including BigLake), or Apache Hudi....