全职 Senior Data Engineer 工作, 薪水, RiDiK (a Subsidiary Of CLPS. Nasdaq: CLPS) Federal Territory 公司招聘中 - Ricebowl

Job Title: Senior Data Engineer

Experience: 7+ Years

Location: Malaysia

Job Description:

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and platforms On Premise and Cloud Ecosystem (Microsoft Azure Preferrable), with hands-on expertise in modern data engineering tools and frameworks.

Key Responsibilities:

• · Migrate data pipelines from existing data acquisition framework to the new GDP data acquisition framework
• · Configure, develop and deliver data ingestion scripts for loading data into T1 data layer
• · Develop and manage ETL/ELT workflows, ensuring data quality, integrity, and reliability.
• · Integrate and automate data quality checks and validation processes within data pipelines.
• · Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
• · Work with modern data lake and warehouse technologies such as Iceberg
• · Implement real-time data streaming solutions using Kafka.
• · Orchestrate complex workflows using Airflow.
• · Integrate with data catalog and governance tools such as Datahub and Ranger.
• · Collaborate with cross-functional teams to understand business requirements and deliver data solutions.
• · Ensure security, compliance, and best practices in data management and governance.

Key Skills Required:

• · Strong proficiency in Linux, Python, and Shell scripting.
• · Hands-on experience with Docker, Kubernetes, and container orchestration.
• · Hands-on experience with Minio and Azure Data Lake Storage (ADLS) using S3 protocols.
• · Experience with Apache Iceberg, Kafka, Airflow, Datahub, Trino, and Ranger.
• · Proficiency in Java for data engineering tasks.
• · Solid understanding of data modeling, data warehousing, and big data technologies.

Prior Experience:

• · Extensive background in building and maintaining data pipelines/ETL processes.
• · Experience in implementing and integrating data quality frameworks.
• · Extensive experience in working on migration projects moving from legacy data pipelines to modernized tech stack