Research Engineer, ML Infrastructure
Remote • San Francisco • FullTime
Posted 2mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
We are an applied AI lab building end-to-end software agents, known for creating Devin, the first AI software engineer. Our team comprises top talent from leading AI companies. This role is crucial for advancing AI research by ensuring the underlying infrastructure is fast, reliable, and scalable. You will own and develop the core systems that researchers depend on, including distributed training infrastructure, experiment orchestration, data pipelines, and tooling. This is a hands-on position where you will collaborate closely with researchers, understand their scientific needs, and build systems capable of handling large-scale training jobs across thousands of GPUs. We value a deep understanding of both research and engineering, expecting infrastructure engineers to grasp the 'why' behind the research.
Responsibilities
- Build and manage systems for reliable, large-scale distributed training across GPU clusters, including job launching, checkpointing, fault tolerance, and monitoring.
- Own the infrastructure for running hundreds of thousands of concurrent coding agent rollouts in VM sandboxes, including environment design and distributed systems for RL training.
- Profile and optimize training throughput by identifying and resolving bottlenecks in data loading, communication, memory, and compute efficiency.
- Design and maintain systems for researchers to launch, track, and analyze experiments, reducing friction in the research loop.
- Develop high-throughput, reliable data pipelines for training and evaluation, ensuring data quality, reproducibility, and efficiency.
- Diagnose and resolve training failures across GPUs, networking, numerics, and data, building systems that fail gracefully and recover quickly.
- Implement and optimize parallelism strategies (data, tensor, pipeline, sequence) for large model training.
- Proactively build infrastructure to anticipate future research needs and prevent constraints.
- Operate infrastructure running across thousands of GPUs, ensuring compute is not a constraint.
Requirements
- Deep experience building and operating distributed training systems for large models.
- Comfortable owning infrastructure end-to-end, from cluster level to training loop.
- Strong systems engineering fundamentals, including distributed systems, networking, and storage.
- Ability to reason about performance across the full hardware-software stack.
- Proficiency in Python and C++.
- Experience with PyTorch or equivalent deep learning frameworks at a systems level.
- Hands-on experience with GPU performance profiling, memory optimization, and compute efficiency.
- Experience implementing or optimizing parallelism strategies for large model training.
- Track record of building tooling and abstractions that accelerate research workflows.
- Strong debugging instincts for complex, distributed systems with non-deterministic failures.
- Sufficient ML knowledge to engage substantively with researchers on training, architectures, and infrastructure needs.