Software Engineer, Inference
$300k - $400k • San Francisco • FullTime
Posted 16d ago
About the job
We are seeking a Software Engineer, Inference to be responsible for the reliability, scale, and efficiency of the systems that serve our AI models to users. This production-facing systems role is central to the company, acting as a bridge between cutting-edge inference techniques and the practicalities of serving live traffic, including rollouts, capacity management, and incident response. You will ensure that performance gains from our research and inference teams reach production safely and remain operational, powering our live serving platform and related products.
Responsibilities
- Operate and scale production inference systems serving live traffic.
- Manage the rollout process for new models and inference optimizations.
- Build and improve observability, alerting, and capacity planning.
- Collaborate with inference and research teams to productionize new serving techniques.
- Lead incident response for production inference issues, including root cause analysis.
- Design for graceful degradation, failover, and redundancy in serving infrastructure.
- Manage capacity and cost tradeoffs for serving infrastructure.
Requirements
- Experience operating large-scale, latency-sensitive production systems.
- Proficiency in Python and Go or another systems language.
- Experience with observability, monitoring, and incident response for production services.
- Strong understanding of distributed systems and their failure modes at scale.
- Experience running production inference for large language models or other large-scale ML systems.
- Experience with deployment and rollout systems (e.g., canarying, blue/green deploys).
- Experience with capacity planning and cost optimization for GPU or TPU infrastructure.
- Familiarity with inference-specific techniques like batching, caching, or quantization.
- Comfortable being on-call and leading incident response.
- Comfortable working with high autonomy in a fast-changing, early-stage environment.
Benefits
- Generous health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support