Role Overview
This role designs, builds and deploys a scalable, reliable data lakehouse platform on AWS. The Data Engineer will architect end-to-end data pipelines, establish data quality frameworks, and deliver production-ready solutions that form the foundation of the organisation's data infrastructure.
Key Responsibilities
Data Pipeline & Architecture Design
- Design and implement scalable ETL/ELT pipelines using AWS services including Glue, Step Functions, Lambda and S3.
- Architect and build data lakehouse solutions leveraging Apache Iceberg or S3 tables — incorporating schema evolution, partition evolution and ACID transactions.
- Optimise pipelines for performance, cost and reliability at scale.
Data Quality & Governance
- Define, implement and maintain automated data quality validation frameworks, with metrics and monitoring to ensure data accuracy, completeness and consistency.
- Enforce data governance standards and ensure compliance across the data platform.
Application Development & Deployment
- Write production-quality code and deploy solutions on AWS cloud infrastructure.
- Apply DevOps practices — CI/CD pipelines and infrastructure-as-code tools such as Terraform or CloudFormation — for repeatable, auditable deployments.
Collaboration & Handover
- Work closely with cross-functional teams and communicate technical designs clearly to non-technical stakeholders.
- Produce thorough technical documentation to support the handover of solutions to the Day 2 operations team.
Qualifications & Experience
- Degree in Computer Science, Data Engineering, Information Systems or a related field.
- 3–5 years of experience in data engineering, ETL/ELT development or data platform roles.
- Strong hands-on experience with AWS services, and proficiency in Apache Iceberg or similar open table formats.
- Experience in a Government Commercial Cloud (GCC) environment is strongly preferred.
- AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are an advantage.
Skills & Competencies
- Data Engineering & Architecture — design and build scalable, reliable data pipelines using modern data lakehouse architectures including Apache Iceberg and S3 tables.
- Data Quality & Governance — define, measure and monitor data quality metrics, implement automated validation frameworks, and ensure compliance with governance standards.
- Application Development — write production-quality code, optimise performance, and deploy solutions on cloud platforms.
- Technical Depth — deep understanding of Apache Iceberg features and S3 table optimisation techniques.
- Collaboration & Communication — work effectively in cross-functional teams and communicate technical concepts clearly to non-technical stakeholders.
Pay: $7,000.00 - $9,000.00 per month
Work Location: In person