Research Engineer, Infrastructure, Training Systems
$350k - $475k • San Francisco • FullTime
Posted 2mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Thinking Machines is seeking an infrastructure research engineer to design and build the core systems that enable scalable, efficient training of large models for deployment and research. The goal is to make experimentation and training fast and reliable, allowing research teams to focus on science rather than system bottlenecks. This role is ideal for individuals with deep systems and performance expertise combined with a curiosity for machine learning at scale, who will own the training stack end-to-end to ensure every GPU cycle drives scientific progress.
Responsibilities
- Design, implement, and optimize distributed training systems that scale across thousands of GPUs and nodes for large-scale training workloads.
- Develop high-performance optimizations to maximize throughput and efficiency.
- Develop reusable frameworks and libraries to improve training reproducibility, reliability, and scalability for new model architectures.
- Establish standards for reliability, maintainability, and security, ensuring systems are robust under rapid iteration.
- Collaborate with researchers and engineers to build scalable infrastructure.
- Publish and share learnings through internal documentation, open-source libraries, or technical reports that advance the field of scalable AI infrastructure.
Requirements
- Bachelor’s degree or equivalent experience in computer science, electrical engineering, statistics, machine learning, physics, robotics, or similar.
- Strong engineering skills, ability to contribute performant, maintainable code and debug in complex codebases.
- Understanding of deep learning frameworks (e.g., PyTorch, JAX) and their underlying system architectures.
- Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
- A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.
- Past experience working on distributed training for large models to make them stable, reliable, and performant.
- Track record of improving research productivity through infrastructure design or process improvements.
- Contributions to open-source ML infrastructure such as PyTorch, XLA, Megatron-LM, or DeepSpeed.
Benefits
- Generous health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support