jobs in Unison Group

全职 Data Engineer (Quantexa) 工作, 薪水, Unison Group 公司招聘中 - Ricebowl

Data Engineer (Quantexa)

Unison Group

全职

Singapore

分享
保存

工作地点

  • Singapore

职位描述

岗位职责

  • Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives
  • Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions
  • Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
  • Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
  • Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
  • Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
  • Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
  • Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
  • Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
  • Ensure data quality and integrity throughout the data processing lifecycle
  • Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques

Requirements

  • Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field
  • Must be Quantexa certified data engineer / data architect and proficient with the software
  • Proven experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments
  • Proficiency in Scala programming language and familiarity with functional programming concepts
  • Experience with Quantexa tool is highly preferred
  • In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
  • Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)
  • Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes
  • Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java
  • Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
  • Experience with Graphana, Prometheus, Splunk will be an added benefit
  • Experience integrating and working with Elasticsearch for data indexing and search applications
  • Experience with Quantexa tool is highly preferred
  • In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
  • Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)

重要安全守则

申请工作时，切勿提供您的银行或信用卡详细资料。不要转账或完成无关的在线调查问卷。如果您发现可疑内容，请举报此招聘广告。

了解更多