Post-Training Research Engineer
$200k - $275k • Remote • San Francisco • FullTime
Posted 6mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Baseten is seeking an engineer with strong machine learning experience and solid foundations in math and computer science to join our growing Post-Training team. This role is crucial for supporting the success of our customers' custom-trained models, which represent the majority of traffic on our platform. You will be responsible for building in-house tooling to efficiently train a wide spectrum of model architectures using various techniques at scale. This involves deep dives into technical topics and working across the full stack, from systems-level concepts like Kubernetes and storage to PyTorch distributed tensor computation and GPU kernels.
Responsibilities
- Build in-house tooling to support the training of diverse model architectures and techniques at scale.
- Work across the full technology stack, including systems-level concepts and distributed computing.
- Optimize the performance of distributed GPU programs.
- Collaborate with researchers and derive specifications by asking important questions.
- Execute on technical challenges related to model training and optimization.
Requirements
- Deep understanding of modern ML techniques and tools for training transformers.
- Advanced experience in a tensor/array computation library (e.g., PyTorch, TensorFlow, Jax).
- Detailed understanding of transformer training parallelism strategies (data, sharded data, tensor, pipeline, context parallelism).
- Experience profiling and improving the performance of distributed GPU programs.
- Ability to perform roofline analysis on transformer training setups.
- Willingness to dive into complex problems and work collaboratively.
- Familiarity with HPC and distributed computing platforms (e.g., Slurm, Ray, Kubernetes, Dask).
- Familiarity with cluster networking technology (e.g., Infiniband, RoCE, GPUDirect).
- Solid fundamentals in operating systems concepts (processes, files, kernel drivers, containerization, networking protocols).
- Creativity and a willingness to question assumptions and tooling choices.
Benefits
- Competitive compensation
- Meaningful equity
- 100% coverage of medical, dental, and vision insurance for employee and dependents (U.S. only)
- Flexible PTO policy
- Company-wide Winter Break
- Paid parental leave
- Fertility and family-building stipend
- Company-facilitated 401(k) (U.S. only)
- Exposure to a variety of ML startups for learning and networking opportunities.