Site Reliability Engineer, Post Training
$300k - $350k • San Francisco • FullTime
Posted 1mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Thinking Machines is seeking a Site Reliability Engineer (SRE) to ensure the speed, reliability, and ease of iteration for our post-training and reinforcement learning (RL) systems. This role is akin to production engineering or site reliability, focused on model training infrastructure. You will be responsible for the health of training runs, clusters, and pipelines, working closely with research teams to debug issues in real-time, strengthen infrastructure, and develop tools and automation that allow researchers to focus on scientific discovery rather than job management. This position offers significant ownership, requiring you to be the point person for critical issues during active model runs and to implement solutions preventing future occurrences.
Responsibilities
- Own the reliability, performance, and uptime of large-scale post-training and RL training jobs.
- Partner with research teams during active model runs to unblock training and accelerate iteration.
- Debug failures across the full stack, including accelerators, networking, storage, schedulers, and training frameworks, driving issues to root cause.
- Build monitoring, alerting, and automated recovery systems for self-healing or fast-failing runs.
- Improve checkpointing, fault tolerance, and job scheduling to minimize compute loss from hardware failures.
- Develop internal tools to reduce toil and enhance cluster utilization for post-training and RL workloads.
- Participate in an on-call rotation to support production model runs.
- Write postmortems and implement permanent infrastructure fixes for recurring failure patterns.
Requirements
- 4+ years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems.
- Proven track record debugging complex failures across distributed systems (networking, hardware, kernel, scheduler).
- Strong software engineering skills in Python and/or Go/C++.
- Solid understanding of Linux systems internals and networking fundamentals.
- Experience operating GPU or TPU training clusters at scale.
- Familiarity with post-training and RL techniques and their specific infrastructure challenges.
- Experience with distributed training frameworks (e.g., PyTorch, Ray) and job schedulers (e.g., Slurm, Kubernetes).
- Experience with high-performance networking (e.g., InfiniBand, RDMA, NCCL).
- Experience building observability tooling for ML training.
- Ability to thrive in fast-changing, research-driven environments.
Benefits
- Generous health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support