Research Engineer - Distributed Training

About the job

Responsibilities

• Build and optimize distributed training infrastructure for pre-training and large-scale RL workloads, contributing to the prime-rl framework.
• Improve end-to-end training efficiency across compute, memory, networking, and scheduling layers.
• Design and implement low-level performance optimizations, including kernels, communication paths, and runtime improvements.
• Work on distributed training systems involving data, tensor, and pipeline parallel workloads.
• Shape the architecture of the RL training stack, including async rollout and post-training systems.
• Contribute to open-source libraries and internal infrastructure for frontier-scale model training.
• Collaborate with researchers and infrastructure engineers to translate bottlenecks into systems improvements.
• Stay current with training systems, inference systems, compiler/runtime tooling, and hardware-aware optimization techniques.

Requirements

• Strong systems engineering experience in AI/ML infrastructure, particularly large-scale model training or inference.
• Deep familiarity with PyTorch and distributed training frameworks (e.g., PyTorch Distributed, DeepSpeed, FSDP, Megatron, vLLM, Ray).
• Experience optimizing training performance across kernels, memory movement, communication overhead, or parallelization strategy.
• Hands-on experience with large-scale training techniques like data parallelism, tensor parallelism, and pipeline parallelism.
• Strong understanding of GPU architecture, profiling, and performance debugging.
• Ability to identify stack bottlenecks and drive improvements from first principles.
• Comfort working in a fast-paced environment with ambiguous problems and high ownership.

Benefits

• Cash Compensation Range of $150-350k
• Equity incentives
• Flexible work arrangements (remote or in-person)
• Visa sponsorship
• Relocation assistance
• Quarterly team off-sites
• Hackathons
• Conferences and learning opportunities