Software Engineer - Training Infrastructure
$165k - $330k • Remote • San Francisco • FullTime
Posted 1y ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Baseten powers mission-critical inference for leading AI companies, enabling them to bring cutting-edge models into production. We are seeking a Software Engineer for our Training Infrastructure team to architect and lead the development of our ML training platform. This role will involve making key technical decisions for the infrastructure that supports developers in deploying, scaling, and monitoring their workloads with high performance and reliability, with ownership over scheduling, storage, networking, reliability, and observability within the training stack.
Responsibilities
- Design and architect scalable infrastructure systems for the ML training platform, including scheduling, storage, and networking.
- Partner with developers and research engineers to translate complex training requirements into technical solutions.
- Design and architect a global training scheduler.
- Design and architect reinforcement learning systems and continuous learning pipelines.
- Drive long-term improvements to enhance system reliability and development velocity.
- Collaborate with SRE and Capacity teams to optimize training infrastructure.
- Make critical architectural decisions balancing performance and system reliability.
- Lead technical discussions and mentor junior engineers on infrastructure best practices.
- Contribute to long-term technical strategy and the infrastructure roadmap.
Requirements
- Bachelor’s degree or higher in Computer Science or a related field.
- 5+ years of experience.
- Proficiency in Go.
- Deep expertise with Kubernetes in production environments.
- Advanced understanding of distributed systems concepts and performance tuning.
- Proven experience designing observability systems.
- Experience with ML/AI workloads and MLOps platforms.
- Experience with distributed storage systems.
- Experience with major cloud providers (AWS, GCP) and neo-cloud providers (Crusoe, DigitalOcean, Nebius).
- Experience with workload orchestration platforms like Temporal or Airflow.
- Familiarity or experience with the open source training stack and frameworks (NCCL, PyTorch, Megatron, NemoRL, VeRL, Axolotl, HF Trainer) and distributed training techniques (FSDP, DeepSpeed).
- Experience developing AI products, tooling, or agents.
Benefits
- Competitive compensation, including meaningful equity.
- 100% coverage of medical, dental, and vision insurance for employee and dependents (U.S. only).
- Flexible PTO policy including company wide Winter Break.
- Paid parental leave.
- Fertility and family-building stipend through Carrot.
- Company-facilitated 401(k) (U.S. only).
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.