Staff + Senior Software Engineer, Inference Infrastructure
$320k - $485k • San Francisco, CA | New York City, NY | Seattle, WA
Posted 1mo ago
Job Location
San Francisco, CA | New York City, NY | Seattle, WA
Tech Stack
Remote Work Policy
On-site
Categories
AI Infrastructure Engineer
About the job
Our Inference team is responsible for building and maintaining the critical systems that serve Claude to millions of users worldwide. We bring Claude to life by serving our models via the industry’s largest compute-agnostic inference deployments. We are responsible for the entire stack from intelligent request routing to fleet-wide orchestration across diverse AI accelerators. The team has a dual mandate: maximizing compute efficiency to reliably serve our explosive customer growth, while enabling breakthrough research by giving our scientists the high-performance inference infrastructure they need to develop next-generation models. We tackle complex, distributed systems challenges across multiple accelerator families and emerging AI hardware running in multiple cloud platforms.
Responsibilities
- Design, build, and maintain distributed systems for serving AI models globally.
- Develop resilient, adaptive systems for real-time event handling.
- Create intelligent request routing, load balancing, and traffic management systems for large-scale accelerator fleets.
- Maximize compute efficiency through autoscaling and orchestration of diverse workloads.
- Build and operate production deployment pipelines for new model releases.
- Provide high-performance inference infrastructure to support next-generation model development.
- Integrate new AI accelerator platforms and support inference for novel model architectures.
Requirements
- Significant software engineering experience, particularly with distributed systems.
- Results-oriented with a bias towards flexibility and impact.
- Willingness to learn and adapt beyond initial job scope.
- Desire to learn about machine learning systems and infrastructure.
- Ability to thrive in environments where technical excellence drives business and research outcomes.
- Experience with high-performance, large-scale distributed systems.
- Experience implementing and deploying machine learning systems at scale.
- Experience with load balancing, request routing, or traffic management systems.
- Familiarity with LLM inference optimization, batching, and caching strategies.
- Experience with Kubernetes and cloud infrastructure (AWS, GCP, Azure).
- Proficiency in Python or Rust.
Benefits
- Annual Salary: $320,000 - $485,000 USD
- Visa sponsorship available