Data Pipeline Development Design, build, and maintain scalable batch and real-time data pipelines using ETL/ELT patterns. Ingest and integrate structured, semi-structured, and unstructured data from diverse sources, including APIs, databases, files, and event streams.Implement incremental loading, change data capture (CDC), and idempotent pipeline design.
Data Governance & Quality: Implement data quality monitoring frameworks, enforce privacy compliance (PDPA), and maintain data lifecycle policies including retention and archiving.
Data Preparation: Cleaning and preprocessing data to make it suitable for analysis, implementing data transformation processes to extract data for the creation and loading of Data Products. Provide strategic recommendations that enhance productivity and business performance
Integrate data from multiple sources into centralised platforms for analytics and reporting, including API integration with internal and external applications and Kafka for real-time streaming
Data Preparation: Cleaning and preprocessing data to make it suitable for analysis, implementing data transformation processes to extract data for the creation and loading of Data Products. Provide strategic recommendations that enhance productivity and business performance
What You'll Do: Design, build, and maintain robust data pipelines️ Collaborate with cross-functional teams to integrate data sources Optimize database performance and ensure data integrity Implement ETL processes for efficient data flow Ensure security and compliance of all managed datasets Support analytics and BI teams with reliable data solutions