Software Engineer, Model Inference
$266k - $500k • Remote • San Francisco • FullTime
Posted 1y ago
About the job
The Inference team at OpenAI is responsible for bringing the company's most capable AI models to the world through its products, empowering consumers, enterprises, and developers. This role focuses on optimizing these large-scale AI models for high-volume, low-latency, and high-availability production and research environments. The engineer will work to improve the performance, latency, throughput, and efficiency of the model inference stack, and build tools to identify and resolve bottlenecks. The team aims to accelerate research progression and enable users to achieve unprecedented capabilities with AI.
Responsibilities
- Collaborate with machine learning researchers, engineers, and product managers to deploy new technologies into production.
- Enable advanced research through engineering efforts.
- Introduce new techniques, tools, and architecture to enhance the performance, latency, throughput, and efficiency of the model inference stack.
- Develop tools for visibility into bottlenecks and instability, and implement solutions for high-priority issues.
- Optimize code and Azure VM fleets to maximize hardware utilization (FLOPs and GPU RAM).
Requirements
- Understanding of modern ML architectures and their inference performance optimization.
- Ability to own problems end-to-end and acquire necessary knowledge.
- At least 5 years of professional software engineering experience.
- Familiarity with PyTorch, NVIDIA GPUs, and optimization stacks (e.g., NCCL, CUDA).
- Familiarity with HPC technologies such as InfiniBand, MPI, NVLink.
- Experience architecting, building, observing, and debugging production distributed systems.
- Experience rebuilding or refactoring production systems due to scale increases.
- Self-directed with the ability to identify and prioritize important problems.
- Humble attitude, eagerness to help colleagues, and commitment to team success.