Engineering Manager, Inference Infrastructure
$405k - $625k • Remote • San Francisco, CA | New York City, NY | Seattle, WA
Posted 1mo ago
Job Location
San Francisco, CA | New York City, NY | Seattle, WA
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
Anthropic is building reliable, interpretable, and steerable AI systems to be safe and beneficial for users and society. This role leads a team of engineers focused on the critical control plane for Anthropic's inference fleet. This involves making key decisions about request routing and capacity allocation to meet throughput, reliability, and latency goals. The team designs placement and load-balancing algorithms, builds quantitative models for demand and performance, and optimizes latency across various system boundaries. The Engineering Manager will guide a team of ML platform, infrastructure, and distributed systems engineers, ensuring the efficiency, reliability, and evolution of the entire request-to-model path.
Responsibilities
- Own the technical roadmap for inference fleet coordination, including traffic routing, capacity management, caching, demand response, and control plane synchronization.
- Partner with product, inference engine, performance, and capacity teams to identify and implement improvements in throughput, latency, utilization, and cost.
- Establish and promote quantitative modeling practices for measuring impact and predicting outcomes before deployment.
- Define technical strategy for evolving the control plane across diverse hardware, cloud providers, and serving surfaces.
- Manage the group's operational backbone, including on-call rotations, incident response, and postmortem reviews, to ensure system stability.
- Facilitate clear communication and collaboration at the intersection of API surfaces, inference engines, capacity planning, and cloud deployment teams.
- Develop and retain existing team members, and hire new talent with a high technical bar.
- Coach engineers through shifting priorities and roadmap changes.
- Shape team structure and define problem area boundaries as the scope of the group expands.
- Provide direct support and unblock initiatives or synthesize design debates when critical.
Requirements
- Engineering management experience leading teams on critical-path production infrastructure at scale.
- Deep systems background in areas like load balancing, scheduling, cluster orchestration, autoscaling, distributed state, or high-performance networking.
- Ability to make architectural decisions for large-scale fleet coordination and evaluate candidates with kernel/framework-level expertise.
- Experience shipping performance or efficiency improvements in large-scale systems, with quantifiable impact analysis.
- Experience running production infrastructure with operational responsibilities including on-call, incident response, and capacity management.
- Results-oriented approach, comfortable balancing competing priorities like throughput, latency, cost, stability, and velocity.
- Ability to build strong cross-team relationships in a 'seam' role.
- Curiosity about machine learning systems and a desire to learn about transformer inference and its system implications.