Member of Technical Staff, Pre-training Systems
$275k - $550k • San Francisco • FullTime
Posted 7mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
As a Research Engineer on the Pre-training Systems team, you will design and operate the distributed infrastructure that trains Magic’s long-context models at scale. This role focuses on large-scale model training across massive GPU clusters, working at the boundary between deep learning and distributed systems to ensure training runs are performant, reliable, and reproducible under extreme scale. You will own the systems that make large-scale pre-training stable and fast, addressing challenges like sustained memory pressure, communication overhead across thousands of devices, long-running jobs that must survive failures, and efficient sequence packing under hardware constraints.
Responsibilities
- Scale distributed training across large GPU clusters using data, tensor, and pipeline parallelism.
- Optimize communication patterns and gradient synchronization.
- Improve checkpointing, fault tolerance, and job recovery systems.
- Profile and eliminate performance bottlenecks across compute, networking, and storage.
- Improve experiment reproducibility and orchestration workflows.
- Increase hardware utilization and training throughput.
- Collaborate with Kernels and Research to align model architecture with systems realities.
Requirements
- Strong software engineering and distributed systems fundamentals.
- Experience training large models in multi-node GPU environments.
- Deep understanding of parallelism strategies and performance trade-offs.
- Experience debugging cross-layer issues in production ML systems.
- Strong ownership mindset and ability to operate critical infrastructure.
- Track record of improving performance or reliability of large-scale systems.
Benefits
- Equity is a significant part of total compensation, in addition to salary.
- 401(k) plan with 6% salary matching.
- Generous health, dental and vision insurance for you and your dependents.
- Unlimited paid time off.
- Visa sponsorship and relocation stipend to bring you to SF, if possible.