Helix AI Engineer, Training Performance
$200k - $400k • San Jose, CA
Posted 9d ago
About the job
Figure is an AI robotics company developing autonomous general-purpose humanoid robots with the goal of shipping human-level intelligence. The Helix team is seeking an experienced AI Training Performance Engineer to elevate model training. This role focuses on enhancing distributed training frameworks for large-scale model training, optimizing GPU kernels, evaluating different accelerator types, and co-designing models for maximum hardware utilization.
Responsibilities
- Optimize training performance for 100B+ parameter models across 100k+ GPUs.
- Collaborate on accelerator choice, cluster topology, scheduling, and hardware procurement.
- Write and optimize custom kernels using Triton/CUDA.
- Build tooling and dashboards for continuous performance monitoring and regression analysis.
- Optimize data loading and preprocessing pipelines to prevent I/O bottlenecks.
- Improve checkpointing, fault tolerance, and elastic restart for quick job recovery.
- Partner with researchers to co-design performant model architectures and training recipes.
- Extend kernel compilers (e.g., Triton, Gluon) for faster iteration and custom accelerator support.
- Build and extend agentic systems for automatic kernel generation and benchmarking.
- Evaluate emerging accelerator architectures for training workloads and lead proof-of-concept ports.
- Explore different model/data parallelisms to determine optimal configurations.
Requirements
- Bachelor's or Master's degree in Computer Science, Computer/Electrical Engineering, or related field.
- 3+ years in AI performance engineering, with experience leading large-scale performance improvement projects.
- Deep understanding of GPU architecture and performance characteristics.
- Proficiency with profiling tools (Nsight Systems/Compute, PyTorch Profiler, HTA) and translating traces into optimizations.
- Solid grasp of collective communication (NCCL) and modern networking concepts (RDMA, NVLink, InfiniBand/RoCE).
- Strong Python and CUDA/C++ skills, with ability to modify framework internals.
- Experience debugging performance regressions and instability at scale.
- Experience defining and using hardware-efficiency metrics (MFU/HFU) to drive optimization.