全职 Senior Data Infrastructure Engineer (Spatial Data Platform) 工作, 薪水, Dada Consultants 公司招聘中 - Ricebowl

Our client is looking for a hands-on Senior Data Infrastructure Engineer to build, scale, and operate the petabyte-scale data lakehouse behind our spatial data products.

Every field and lab capture—robotics, egocentric headsets, multi-camera video, depth, IMU—lands in this platform, gets versioned, and powers downstream 3D reconstructions (Gaussian Splats, SLAM) and ML training clusters.

This is a true engineering role. We need someone who has personally built and run a PB-scale platform in production, not just used one.

Key Responsibilities

• PB-Scale Lakehouse: Design object storage layouts (S3), partitioning strategies, catalog architectures, and open table formats (Iceberg/Delta).
• Data Versioning & Ingest: Build reproducible dataset lineage (LakeFS) and high-throughput ingest pipelines (validation, checksumming, deduplication, bronze $\rightarrow$ gold promotion).
• Distributed CPU/GPU Compute: Operate large-scale processing (Spark, Ray, Flink) for frame extraction, sync/calibration, SLAM/3DGS batch jobs, and auto-labeling.
• ML Training Integration: Keep GPU clusters fed using sharded training formats (WebDataset, Mosaic Streaming), caching, and optimized cross-region transfers.
• Curation & Querying: Build indexing, search, and high-performance SQL query layers (Trino, DuckDB, Spark SQL) so researchers can slice data without copying it.
• Reliability & Infrastructure: Maintain IaC (Terraform) on AWS, handle pipeline backfills/recovery, optimize storage/egress costs, and participate in on-call rotation.

Qualifications

• Experience: 6+ years in data infrastructure or distributed systems engineering.
• Scale: Proven track record setting up and operating a live petabyte-scale data platform (able to speak concretely to throughput, object limits, and failure modes).
• Tech Stack:
• Languages: Python + at least one of Go, Rust, C++, Java, or Scala.
• Table Formats & Versioning: Iceberg/Delta Lake, Catalogs (Glue/Nessie), LakeFS or content-addressed storage.
• Processing & Orchestration: Spark, Ray, or Flink; Airflow, Argo, or Prefect.
• Cloud/IaC: Strong AWS (S3, EKS/ECS, IAM) and Terraform experience.

Nice to Have

• Experience with unstructured/multimodal spatial data (video, LiDAR, point clouds, ROS bags/MCAP).
• GPU-accelerated pipelines (NVIDIA DALI, NVDEC, RAPIDS) or ML data loaders (WebDataset, Petastorm).
• Query engines (Trino, DuckDB, ClickHouse) and embedding/vector search.