全职 Data Engineer 工作, 薪水, Optimum Solutions Pte Ltd 公司招聘中 - Ricebowl

• Design, develop and deploy data engineering solutions, including data tables, views, data marts and related components, across data pipelines, data warehouses, operational data stores, data lakes and data virtualisation platforms.
• Perform data extraction, cleansing, transformation and data flow activities. Web scraping may also form part of the data extraction scope.
• Design, build, launch and maintain efficient, reliable and scalable batch and real-time data pipelines using appropriate data processing frameworks.
• Integrate and consolidate data from multiple silos in a scalable manner while ensuring compliance with relevant requirements.
• Ensure that established quality processes are followed throughout the data product lifecycle, including development, deployment, validation, change management and documentation, with particular focus on production environment standards.
• Ensure that proper data governance practices are followed and that solutions comply with security policies and regulatory requirements.
• Collaborate with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers and Data Analysts to develop scalable, data-driven products.
• Develop backend APIs and work with databases to support applications.
• Work within an Agile environment that follows Continuous Integration and Continuous Delivery practices.
• Collaborate closely with fellow developers through pair programming and code review activities.
• Work with business stakeholders to gather and analyse requirements for data engineering solutions, ensuring that technical implementations are aligned with business objectives.
• Support business users in designing and developing front-end reports, dashboards and interactive visualisations to strengthen the organisation’s data analysis capabilities.
• Provide ongoing operational maintenance and technical support for deployed data engineering products and systems.
• Manage incidents and service requests, ensuring that issues are resolved promptly and within the agreed Service Level Agreements.
• Participate in regular project audits and provide training support to team members and stakeholders when required.

Qualifications

• Proficient in general data cleansing and transformation using tools and technologies such as SQL, VQL, pandas and R to maintain data accuracy and consistency.
• Proficient in building ETL pipelines using technologies such as SQL Server Integration Services, AWS Database Migration Service, Python, AWS Lambda, ECS container tasks, EventBridge, AWS Glue and Spring.
• Proficient in database design and experienced with different database and storage technologies, including SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostGIS, MySQL, SQLite, VoltDB and Cassandra.
• Experience with cloud technologies and environments such as GPC and GCC, including AWS, Azure and Google Cloud.
• Experience and a strong interest in data engineering within big data environments using cloud platforms such as GPC and GCC, particularly AWS.
• Experience building production-grade data pipelines and ETL or ELT data integration solutions.
• Knowledge of system design, data structures and algorithms.
• Familiarity with data modelling, data access and data storage infrastructure, including Data Marts, Data Lakes, Data Virtualisation and Data Warehouses, to support efficient data storage and retrieval.
• Familiarity with REST APIs, web requests and general web protocols.
• Familiarity with big data frameworks and technologies such as Hadoop, Spark, Kafka and RabbitMQ.
• Familiarity with the W3C Document Object Model and customised web scraping using tools such as BeautifulSoup, CasperJS, PhantomJS, Selenium and Node.js.
• Familiarity with data governance policies, access controls and security best practices.
• Comfortable working with at least one scripting language, such as SQL or Python.
• Comfortable working in both Windows and Linux development environments.
• Interested in serving as the bridge between engineering and analytics teams.
• Strong communication skills and the ability to work closely with stakeholders, technical leads and fellow team members.