Most people are surprised a palm oil company hires data engineers. The work is closer to IoT platform engineering than to agriculture - and an industry being instrumented for the first time throws up genuinely interesting data problems.
Build the data infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL including Google, Microsoft & AWS cloud data/serverless technologies.
Data Quality Assurance: Collaborate with all the data owners to implement processes that ensure data reliability, efficiency, and quality, including data cleaning and validation. Implement and maintain a robust Row-Level Security.
Data Quality Assurance: Collaborate with all the data owners to implement processes that ensure data reliability, efficiency, and quality, including data cleaning and validation. Implement and maintain a robust Row-Level Security.
What You'll Do: Design, build, and maintain robust data pipelines️ Collaborate with cross-functional teams to integrate data sources Optimize database performance and ensure data integrity Implement ETL processes for efficient data flow Ensure security and compliance of all managed datasets Support analytics and BI teams with reliable data solutions
Data Pipeline Development Design, build, and maintain scalable batch and real-time data pipelines using ETL/ELT patterns. Ingest and integrate structured, semi-structured, and unstructured data from diverse sources, including APIs, databases, files, and event streams.Implement incremental loading, change data capture (CDC), and idempotent pipeline design.
Data Governance & Quality: Implement data quality monitoring frameworks, enforce privacy compliance (PDPA), and maintain data lifecycle policies including retention and archiving.
Data Preparation: Cleaning and preprocessing data to make it suitable for analysis, implementing data transformation processes to extract data for the creation and loading of Data Products. Provide strategic recommendations that enhance productivity and business performance
Integrate data from multiple sources into centralised platforms for analytics and reporting, including API integration with internal and external applications and Kafka for real-time streaming