Performance Engineer, Inference Systems
San Francisco, CA | New York City, NY | Seattle, WA
Posted 16d ago
About the job
Anthropic's inference fleet serves Claude to millions of users across its own products and major cloud platforms. The stack supporting this is complex, involving accelerator kernels, model servers, distributed routing, autoscaling, and capacity management, where each layer's performance impacts the others. The Inference System Dynamics team is tasked with understanding this entire system and ensuring high performance across throughput, latency, reliability, and correctness. This involves measuring fleet performance against theoretical limits, investigating performance gaps, and owning correctness checks to ensure Claude's outputs are accurate across various hardware and serving configurations. The team focuses on identifying high-leverage optimization opportunities across components and collaborating with owning teams to implement them, treating correctness as an integral part of performance.
Responsibilities
- Conduct cross-layer performance investigations for throughput, latency, and reliability, quantifying performance gaps and identifying root causes.
- Own and enhance the correctness evaluation pipeline to validate model output quality across different hardware, numerics, and serving configurations.
- Develop observability tools, dashboards, and models to visualize and understand performance metrics like throughput, latency, cost, and reliability.
- Collaborate with kernel, serving, routing, autoscaling, and capacity teams to implement high-impact optimizations.
- Prioritize optimization opportunities based on impact and effort, focusing on the most critical areas.
Requirements
- Hands-on performance engineering experience, including profiling, roofline analysis, and root-cause investigation in complex production systems.
- Proficiency in Python for reading, instrumenting, and contributing to large production codebases.
- Solid data analysis skills (e.g., SQL, pandas) to derive clear findings from telemetry data.
- Ability to clearly communicate quantitative results in writing to influence priorities.
- Genuine interest in correctness as an engineering discipline, including numerics and regression detection.