- Kuala Lumpur Federal Territory Malaysia
全职 Lead Software Engineer 工作, 薪水, Lavu Tech Solutions Federal Territory 公司招聘中 - Ricebowl
分享
保存
工作地点
职位描述
岗位职责
Key Responsibilities
Migrate data pipelines from existing data acquisition framework to the new GDP data acquisition framework
Configure, develop and deliver data ingestion scripts for loading data into T1 data layer
Develop and manage ETL/ELT workflows, ensuring data quality, integrity, and reliability.
Integrate and automate data quality checks and validation processes within data pipelines.
Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
Work with modern data lake and warehouse technologies such as Iceberg
Implement real time data streaming solutions using Kafka.
Orchestrate complex workflows using Airflow.
Integrate with data catalog and governance tools such as Datahub and Ranger.
Collaborate with cross functional teams to understand business requirements and deliver data solutions.
Ensure security, compliance, and best practices in data management and governance.
Key Skills Required
Strong proficiency in Linux, Python, and Shell scripting.
Hands on experience with Docker, Kubernetes, and container orchestration.
Hands on experience with Minio and Azure Data Lake Storage (ADLS) using S3 protocols.
Experience with Apache Iceberg, Kafka, Airflow, Datahub, Trino, and Ranger.
Proficiency in Java for data engineering tasks.
Solid understanding of data modeling, data warehousing, and big data technologies.
Migrate data pipelines from existing data acquisition framework to the new GDP data acquisition framework
Configure, develop and deliver data ingestion scripts for loading data into T1 data layer
Develop and manage ETL/ELT workflows, ensuring data quality, integrity, and reliability.
Integrate and automate data quality checks and validation processes within data pipelines.
Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
Work with modern data lake and warehouse technologies such as Iceberg
Implement real time data streaming solutions using Kafka.
Orchestrate complex workflows using Airflow.
Integrate with data catalog and governance tools such as Datahub and Ranger.
Collaborate with cross functional teams to understand business requirements and deliver data solutions.
Ensure security, compliance, and best practices in data management and governance.
Key Skills Required
Strong proficiency in Linux, Python, and Shell scripting.
Hands on experience with Docker, Kubernetes, and container orchestration.
Hands on experience with Minio and Azure Data Lake Storage (ADLS) using S3 protocols.
Experience with Apache Iceberg, Kafka, Airflow, Datahub, Trino, and Ranger.
Proficiency in Java for data engineering tasks.
Solid understanding of data modeling, data warehousing, and big data technologies.
重要安全守则
申请工作时，切勿提供您的银行或信用卡详细资料。不要转账或完成无关的在线调查问卷。如果您发现可疑内容，请举报此招聘广告。
了解更多
