Member of Technical Staff - Pre-Training Infra
San Francisco, CA • FullTime
Posted 6mo ago
Job Location
San Francisco, CA
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Reflection is a research lab dedicated to making intelligence open and accessible. We build open models that empower individuals to control their intelligence and shape the future of AI. As a Member of Technical Staff - Pre-Training Infra, you will be instrumental in building and scaling the distributed training systems that power frontier model pre-training. This role involves close collaboration with research teams to design and operate large-scale training runs for foundation models, and developing the infrastructure necessary for efficient training across thousands of GPUs.
Responsibilities
- Build and scale distributed training systems for frontier model pre-training.
- Design and operate large-scale training runs for foundation models in collaboration with research teams.
- Develop infrastructure for efficient training across thousands of GPUs using distributed training frameworks.
- Optimize training throughput, stability, and efficiency for large model training workloads.
- Translate experimental ideas into scalable, production-ready training systems with pre-training researchers.
- Improve performance of distributed training workloads by optimizing communication, memory usage, and GPU utilization.
- Build and maintain training pipelines for large-scale datasets, checkpointing, and experiment iteration.
- Debug and resolve performance bottlenecks in distributed training stacks, including model parallelism, GPU communication, and training runtime systems.
- Contribute to systems enabling rapid experimentation and iteration on new training techniques.
Requirements
- Experience building or operating distributed training systems for large machine learning models.
- Strong experience with modern distributed training frameworks (e.g., Megatron, DeepSpeed).
- Familiarity with large-scale model parallelism strategies (data, tensor, pipeline, or expert parallelism).
- Experience optimizing training throughput and GPU utilization in large distributed environments.
- Familiarity with GPU communication libraries (e.g., NCCL) and performance tuning for distributed workloads.
- Experience collaborating with ML researchers to productionize experimental training workflows.
- Strong debugging skills across GPU compute, distributed training systems, and large-scale ML pipelines.
- Experience with large datasets and training pipelines for foundation model pre-training.
Benefits
- Top-tier compensation (salary and equity)
- Stock options
- Comprehensive medical, dental, vision, and life insurance
- Annual wellness allowance
- Provided lunch and dinner in the office
- 22 weeks paid parental leave
- Unlimited paid time off (U.S.)
- 30 days paid time off (U.K.)
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations