全职 Data Engineer 工作, 薪水 up to MYR 14,000, Ikigaienablers Federal Territory 公司招聘中 - Ricebowl

Data Engineer – Databricks & PySpark

Location: Kuala Lumpur, Malaysia
Work Location: On-site – Menara AIA, 99 Jalan Ampang, Kuala Lumpur
Experience: 5+ Years

Job Overview

We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, PySpark, Delta Lake, and enterprise data platforms to join our team in Kuala Lumpur, Malaysia.

The ideal candidate will have extensive experience developing scalable data pipelines, building enterprise Data Warehouse and Lakehouse solutions, implementing data governance using Unity Catalog, and optimizing large-scale data processing workloads.

Candidates should also have experience with CI/CD automation, GitHub, data security, and modern data engineering practices. Experience with Azure Databricks is preferred.

This position requires strong technical ownership, problem-solving skills, stakeholder management capabilities, and a willingness to participate in production support and on-call rotations.

Key Responsibilities

1. Data Engineering & Platform Development

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT solutions for batch and near-real-time data processing.
• Design and maintain enterprise Data Warehouse and Lakehouse architectures supporting reporting, analytics, and AI/ML use cases.
• Develop and maintain data models using Delta Lake and Delta Tables.
• Implement dimensional data models, including Fact and Dimension tables, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD Type 1 and Type 2).
• Ensure data quality, reliability, scalability, and performance across the data platform.
• Establish best practices for code management, testing, deployment, and operational monitoring.

2. Databricks Platform & Data Governance

• Implement and manage Unity Catalog for centralized data governance, discovery, and security.
• Implement Role-Based Access Control (RBAC), Attribute-Based Access Control (ABAC), and fine-grained data permissions.
• Manage data lineage, auditing, and governance frameworks.
• Configure and manage Delta Sharing for secure internal and external data collaboration.
• Support the administration and adoption of Databricks Genie, including Genie Spaces, natural-language data interaction, and security controls.
• Ensure compliance with enterprise data security and governance requirements.

3. Performance Optimization

• Perform advanced PySpark performance tuning and troubleshooting.
• Optimize SQL queries, cluster utilization, partitioning strategies, and workload management.
• Identify performance bottlenecks and implement improvements in platform efficiency and cost optimization.
• Optimize Data Warehouse and Lakehouse workloads for high-performance analytics and reporting.

4. DevOps & Automation

• Design and implement CI/CD pipelines for Databricks solutions.
• Integrate Databricks development workflows with GitHub, GitHub Actions, and enterprise DevOps processes.
• Automate deployment, testing, code validation, and release management.
• Establish infrastructure automation and data engineering best practices.

5. Stakeholder Management

• Collaborate with business users, data consumers, architects, analysts, and technology leadership to understand requirements.
• Translate business requirements into scalable technical solutions, data models, and platform capabilities.
• Communicate technical concepts effectively to business and technical stakeholders.
• Work independently, manage priorities, and ensure timely delivery.
• Provide technical guidance and mentorship to junior team members where required.

6. Production Support

• Participate in a rotating production support and on-call roster.
• Troubleshoot production incidents and resolve issues within established SLAs.
• Conduct root cause analysis and implement preventive measures.
• Maintain platform stability, reliability, and operational excellence.

Required Skills & QualificationsEssential Technical Requirements

• 5+ years of experience in Data Engineering, Data Warehousing, or Big Data technologies.
• Minimum 4+ years of hands-on Databricks experience in enterprise environments.
• Strong hands-on experience with Databricks and PySpark development.
• Experience with Delta Lake and Delta Tables.
• Hands-on experience with Unity Catalog, data governance, and access controls.
• Experience designing and implementing enterprise Data Warehouse and Lakehouse solutions.
• Strong knowledge of SQL and advanced query optimization.
• Experience with ETL/ELT pipeline development and optimization.
• Strong understanding of dimensional data modelling, including:
• Fact and Dimension tables
• Star Schema and Snowflake Schema
• Slowly Changing Dimensions (SCD Type 1 and Type 2)
• Experience implementing CI/CD pipelines and Git-based development workflows.
• Experience with GitHub and GitHub Actions or equivalent CI/CD tools.
• Experience working with at least one major cloud platform: Azure, AWS, or GCP.
• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.

Additional Technical Requirements

• Experience with Delta Sharing.
• Knowledge of Databricks Workflows.
• Experience implementing RBAC and ABAC within Unity Catalog.
• Experience with Databricks Genie and Genie Spaces.
• Knowledge of Lakehouse architecture and Medallion design patterns.
• Experience with data observability, monitoring, and production troubleshooting.
• Knowledge of Infrastructure as Code, preferably Terraform.
• Familiarity with Agile development methodologies.
• Experience integrating data platforms with Power BI, Tableau, or other analytics tools.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification.
• Experience implementing enterprise data governance frameworks.
• Experience supporting large-scale Lakehouse and Data Warehouse architectures.
• Experience working in regulated industries with strict security and compliance requirements.
• Knowledge of Data Mesh and modern data platform architectures.
• Azure Databricks experience.

Required Soft Skills

• Strong ownership, accountability, and commitment to delivery.
• Ability to work independently with minimal supervision.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong communication and stakeholder management abilities.
• Ability to manage multiple priorities in a fast-paced environment.
• Collaborative, proactive, and customer-focused approach.
• Willingness to participate in production support and on-call rotations.

What We Are Looking For

The successful candidate will serve as a key technical contributor to the data engineering team, driving Databricks development, data governance, platform optimization, and the delivery of reliable and scalable enterprise data solutions.

Candidates should demonstrate practical, hands-on experience rather than theoretical knowledge alone, particularly in Databricks, PySpark, Delta Lake, Unity Catalog, and enterprise data platform development.

Interested candidates with relevant Databricks and Data Engineering experience are encouraged to apply.

Pay: RM4,200.00 - RM14,000.00 per month

Benefits:

• Additional leave
• Health insurance
• Maternity leave
• Opportunities for promotion
• Parental leave
• Professional development

Application Question(s):

• Currently in Malaysia
• Willing to apply for 1 year extended contract through agency payroll
• Budget for the given role would be RM 14000
• Mention your current, expected salary and notice period
• Do you have a minimum of 5 years of professional experience in Data Engineering, Data Warehousing, or Big Data technologies?
• Do you have at least 4 years of hands-on experience with Databricks in an enterprise environment?
• Do you have hands-on experience with Delta Lake, Delta Tables, and Unity Catalog?

Work Location: In person