全职 Manager, Data Engineer 工作, 薪水, Alliance Bank WP Kuala Lumpur 公司招聘中 - Ricebowl

Key Responsibilities

1.Data Pipeline Development & Integration

• Design, build, and maintain scalable, reusable ETL/ELT pipelines for batch and real-time data processing

• Integrate data from multiple internal and external sources (databases, APIs, streaming platforms, third-party systems)

• Ensure efficient data ingestion, transformation, and loading processes

2. Data Modeling & Transformation

• Develop and maintain logical and physical data models (e.g., dimensional models, star/snowflake schemas)

• Transform raw data into structured, analytics-ready datasets

• Optimize data structures for performance, scalability, and usability

3. Data Quality & Reliability

• Implement data validation, cleansing, and enrichment processes

• Establish data quality frameworks, including rules, checks, and monitoring

• Troubleshoot and resolve data inconsistencies, anomalies, and failures

• Ensure high availability and reliability of data pipelines

4. Data Platform Utilization & Optimization

• Work with cloud data platforms (e.g., Azure, AWS, GCP) and data services (e.g., Synapse, BigQuery, Redshift)

• Optimize storage, query performance, and compute efficiency

• Monitor pipeline performance and continuously improve throughput and latency

5. Collaboration & Data Enablement

• Collaborate with data analysts, data scientists, architects, and business stakeholders to understand data requirements

• Provide clean, well-structured datasets for reporting, dashboards, and machine learning use cases

• Support self-service analytics by enabling easy data access and documentation

6.Data Governance & Security

• Implement and adhere to data governance policies, standards, and best practices

• Ensure compliance with data privacy, security, and regulatory requirements

• Manage data access controls, lineage, and auditability

7. Automation & DevOps Practices

• Automate data workflows using orchestration tools (e.g., Airflow, Azure Data Factory)

• Implement CI/CD pipelines for data engineering processes

• Apply version control and infrastructure-as-code practices

• Promote reusable components and standardized frameworks

8. Monitoring, Logging & Observability

• Set up monitoring, alerting, and logging frameworks for pipelines and datasets

• Track data lineage, pipeline health, and system performance

• Proactively identify and resolve production issues

9. Documentation & Knowledge Sharing

• Maintain clear documentation for data pipelines, data models, and architecture

• Define data definitions, metadata, and data catalogs

• Share knowledge and best practices across teams

10. Continuous Improvement & Innovation

• Stay current with emerging data engineering tools, technologies, and patterns

• Recommend and implement improvements in architecture, performance, and cost efficiency

• Contribute to evolving enterprise data strategy and standards

Requirement

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or related field

• Experience with real-time data streaming and processing

• Familiarity with containerization tools such as Docker and Kubernetes

• Knowledge of data modeling techniques (dimensional, star schema)

• Experience with CI/CD pipelines for data workflows

• Understanding of machine learning workflows and data preparation

• Strong programming skills in Python, Java, or Scala

• Experience with SQL and NoSQL databases

• Hands-on experience with data pipeline tools (e.g., Apache Airflow, Talend, Informatica)

• Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka)

• Experience with cloud platforms such as AWS, Azure, or Google Cloud

• Understanding of data warehousing concepts (e.g., Snowflake, Redshift, BigQuery)

• Knowledge of version control systems (e.g., Git)