全职 Research Engineer, Training Systems - Experimentation - Monee 工作, 薪水, Monee 公司招聘中 - Ricebowl

About The Team

We are looking for a Research Engineer who bridges model research and engineering to build reliable, efficient, and reproducible experimentation and training foundations for behavioral sequence pretraining and graph learning, reducing the time needed to reach trustworthy research conclusions.

You will work with algorithm experts, researchers, and application owners on real training workloads, addressing bottlenecks in data loading, graph sampling, computation, memory, experiment execution, and evaluation. You will improve experiment turnaround, resource use, and batch representation generation while preserving model quality and data correctness, and build observability tools that support model diagnosis.

Job Description

• Reproducible experimentation: Build configurable workflows for training, checkpointing, recovery, evaluation, and result recording. Track code, data, graph snapshots, sampling, and randomness, reducing duplicate scripts and manual work.
• Performance baselines and diagnosis: Profile loading, sampling, transfer, forward and backward passes, synchronization, and output writing. Establish benchmarks for throughput, memory, failure rate, turnaround, and GPU-hours required to reach a target quality.
• Sequence training efficiency: Improve data sharding, prefetching, length bucketing, padding, batching, mixed precision, and checkpointing. Evaluate attention backends, compilation, or multi-GPU approaches against measured bottlenecks while checking numerical and training correctness.
• Graph learning engineering: Optimize graph storage, neighbor sampling, feature fetching, caching, CPU–GPU transfer, and training pipelines. Support repeatable evaluation on fixed snapshots and validate sampling or graph-construction changes with researchers.
• Evaluation and diagnostic tools: Connect training, embedding generation, downstream integration, and reporting. Expose losses, gradients, numerical anomalies, and representation statistics to help researchers diagnose issues, while preserving point-in-time correctness of features and relationships.
• Resource efficiency and experiment scheduling: Estimate capacity and compute budgets from measured throughput, memory, and concurrency needs. Compare scaling a single run with running independent experiments in parallel and improve recovery and execution efficiency.
• Inference and engineering delivery: Optimize batch representation or score generation, sharding, outputs, and versioning. Support reliable integration with application and platform teams and maintain benchmarks, tools, runbooks, and reusable code.
  
  

What You Will Work On

• Build research engineering capabilities around real sequence and graph learning problems and directly improve experiment speed and trustworthiness.
• Investigate bottlenecks across models, data, and systems and validate improvements through quality and cost measurements.
• Work with researchers and application colleagues to build engineering foundations the team can reuse over time.
  
  

Requirements

• Bachelor's degree or above in Computer Science, Software Engineering, AI, or a related field, with strong software engineering foundations and demonstrable research engineering, ML engineering, or systems projects.
• Proficiency in Python and PyTorch or comparable frameworks, with an understanding of training loops, automatic differentiation, batches, optimizers, checkpoints, and inference, and the ability to debug model code independently.
• In-depth practical experience in at least one of training systems, data pipelines, performance diagnosis, distributed systems, or experimentation infrastructure, with clear evidence of bottlenecks, trade-offs, and improvements.
• Foundational performance analysis skills across GPUs, CPUs, memory, and I/O, with the ability to use profilers and runtime metrics to identify issues rather than relying solely on trial and error.
• Attention to model quality, experimental comparability, numerical correctness, data versions, and reproducibility, with the ability to design fair quality and cost comparisons for engineering changes.
• Strong practices in code organization, testing, logging, recovery, and documentation, with the ability to work through open-ended problems with researchers and deliver iteratively.
• Ability to read technical material, write engineering documentation, and collaborate across teams.
  
  

Good to have

• Experience with Transformer pretraining or fine-tuning, GNNs, or large-scale behavioral data.
• Experience with DGL or PyG, graph sampling, sparse computation, feature stores, or large-graph training optimization.
• Experience with multi-GPU training, DDP or FSDP, mixed precision, or distributed data processing, with an understanding of scaling efficiency and cost trade-offs.
• Experience with CUDA or Triton, kernel optimization, compilation, experiment orchestration, evaluation platforms, or observability tooling.