全职 Manager, Data Platform Engineer 工作, 薪水, Alliance Bank WP Kuala Lumpur 公司招聘中 - Ricebowl

Key Responsibilities

Platform Design & Engineering

·       Design and implement scalable, resilient, and secure data platforms (data lakes, lakehouse, data warehouses).

·       Build and maintain distributed data systems that support batch, streaming, and real-time processing.

·       Develop reusable frameworks and platform services to standardize data ingestion, processing, and access.

Data Pipeline Development

·       Build and maintain robust ETL/ELT pipelines using modern orchestration tools.

·       Ensure data quality, lineage, observability, and governance are embedded within pipelines.

·       Optimize data workflows for performance, cost efficiency, and reliability.

Cloud & Infrastructure Management

·       Deploy, manage, and optimize data platforms on cloud providers (Azure, AWS, GCP).

·       Implement Infrastructure as Code (IaC) using tools like Terraform, ARM/Bicep, or CloudFormation.

·       Monitor and manage platform performance, availability, and scalability.

Data Governance & Security

·       Implement data governance frameworks, including cataloging, classification, and lineage tracking.

·       Ensure compliance with data security and privacy standards (e.g., GDPR, PDPA).

·       Manage access control, encryption, and auditing mechanisms.

DevOps & Automation

·       Build CI/CD pipelines for data platform components.

·       Automate deployments, monitoring, and alerting.

·       Apply SRE principles to improve platform reliability and availability.

Collaboration & Enablement

·       Partner with data engineers, data scientists, and business stakeholders to deliver data solutions.

·       Provide platform best practices and guidelines to engineering teams.

·       Support self-service data capabilities for analytics and AI use cases.

Requirement

Education & Experience

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

• 5+ years of experience in data engineering, platform engineering, or related roles.

• Experience with AI/ML data pipelines and feature stores.

• Knowledge of data security frameworks and zero-trust architecture.

• Certification in cloud platforms (e.g., Azure Data Engineer, AWS Certified Data Analytics).

• Familiarity with FinOps practices for data platform cost optimization.

Technical Skills

• Strong programming skills (Python, Java, Scala, or similar).

• Experience with distributed data processing frameworks (Spark, Flink, or equivalent).

• Proficiency in SQL and data modeling techniques.

Cloud & Data Technologies

• Hands-on experience with cloud data services:

o Azure: Data Factory, Synapse, Databricks, ADLS, HDInsight

o AWS: S3, Glue, Redshift, EMR

o GCP: BigQuery, Dataflow, Composer, Managed Spark, Hadoop

o Specialized Big Data: Snowflake, Databricks, Cloudera, Oracle Big Data Services

• Experience with modern data architectures (Lakehouse, Data Mesh, Data Fabric).

Data Pipeline & Orchestration

• Tools such as Airflow, Azure Data Factory, Prefect, or Dagster, Docket, Git.

• Experience with streaming platforms (Kafka, Redpanda Event Hubs, Kinesis).

DevOps & Infrastructure

• Familiarity with containerization (Docker) and orchestration (Kubernetes).

• Experience with CI/CD tools (Azure DevOps, GitHub Actions, Jenkins).

• Infrastructure as Code (Terraform preferred).

Data Governance, Security & Observability

• Experience with tools like Collibra, Purview, DataHub, Prometheus, Grafana, OpenLineage/ Apache Ranger

• Understanding data quality frameworks and monitoring tools.