- 60 PAYA LEBAR ROAD Central Region (Singapore) Singapore
Working Location
Job Description
Responsibilities
Develop 20 data table ingestion pipelines end-to-end (source extraction → S3 landing → Glue ETL → Lake Formation curated zone)
Implement ETL transformations per data mapping specifications
Build data quality validation rules (Great Expectations / custom Glue checks) — completeness, schema conformance, referential integrity
Configure error handling and dead-letter patterns for failed ingestion records
Register all datasets in AWS Glue Data Catalog with standardised metadata tags (owner, classification, freshness SLA)
Write and maintain IaC (CDK/Terraform) for pipeline resources — Glue jobs, crawlers, S3 buckets, IAM roles
Execute unit testing (per-transform logic) and integration testing (end-to-end flow with sample data)
Support UAT with Agency A data owners — validate output tables match expected schema and row counts
Document pipeline configurations, runbooks, and data flow diagrams for handover
Participate in daily stand-ups, sprint demos, and code reviews
What We Are Looking For
Required skills: AWS Glue, Lake Formation, S3, Athena, Python/PySpark, IaC (Terraform/CDK), SQL
Nice to Have
Great Expectations, SHIP-HATS CI/CD, data cataloging, prior public sector / Government Agency experience
Important Information
Never provide your bank or credit card details when applying for jobs. Do not transfer any money or complete unrelated online surveys. If you see something suspicious, Report this Job ad.