Training Performance Engineer
$295k - $500k • San Francisco • FullTime
Posted 1y ago
About the job
As a Training Performance Engineer, you will be instrumental in enhancing the efficiency of our distributed training stack. Your responsibilities will include analyzing large-scale training runs, pinpointing utilization gaps, and devising optimizations to maximize throughput and uptime. This role requires a strong grasp of systems and practical performance engineering, involving the analysis of GPU kernel performance, collective communication throughput, investigation of I/O bottlenecks, and model sharding for massive-scale training. You will ensure our clusters operate at peak performance, enabling OpenAI to train larger, more capable models within existing compute budgets.
Responsibilities
- Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
- Optimize GPU utilization and throughput for large-scale distributed model training.
- Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
- Implement model graph transforms to improve end-to-end throughput.
- Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
- Partner with researchers to ensure new model architectures scale efficiently during pre-training.
- Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.
Requirements
- Strong programming skills in Python and C++.
- Experience running distributed training jobs on multi-GPU systems or HPC clusters.
- Ability to debug complex distributed systems and measure efficiency rigorously.
- Familiarity with frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
- Comfort collaborating across teams and translating raw profiling data into practical engineering improvements.
- Rust or CUDA experience is a plus.
- Familiarity with NCCL, MPI, or UCX communication libraries is a plus.
- Experience with large-scale data loading and checkpointing systems is a plus.
- Prior work on training runtime, distributed scheduling, or ML compiler optimization is a plus.