Performance Engineer, Inference Engine
San Francisco, CA | New York City, NY
Posted 4d ago
About the job
Anthropic is seeking a Performance Engineer for its Inference Engine team. The inference engine is a critical in-house software component that manages the entire token path, from batching requests to coordinating forward passes across accelerator platforms. This role involves building and optimizing this system at scale to enhance throughput, cost-efficiency, reliability, and latency. You will leverage your deep understanding of hardware and bandwidth metrics to model and solve performance bottlenecks. This is a highly technical and impactful position for engineers who thrive on accelerator programming, high-performance host-device coordination, and large-scale distributed systems.
Responsibilities
- Build and optimize the inference engine for Anthropic's scale.
- Improve throughput, cost, reliability, and latency across all accelerator and cloud platforms.
- Model performance bottlenecks by analyzing time and byte consumption.
- Ensure high device utilization by minimizing overheads.
- Implement strategies for reusing computations, such as caching model states.
- Develop observability tools to identify performance gaps and model potential improvements.
- Ensure model quality and maintain intelligence across platforms.
- Collaborate with safety teams to integrate safety systems into the inference engine.
Requirements
- A working mental model of LLM inference, including prefill and decode on accelerators and host operations.
- Demonstrated ability to quickly learn and make significant contributions in complex systems.
- Strong systems programming skills in languages like Rust or C++, with a focus on code quality and testing.
- Analytical approach to performance, involving observation, profiling, hypothesis testing, and measurement.
- Low ego, willingness to ask fundamental questions, and openness to feedback.
- Enjoyment of pair programming and consideration for the societal impact of AI.
- Bachelor's degree or equivalent combination of education, training, and experience.
Benefits
- Annual compensation range: $350,000 - $850,000 USD
- Visa sponsorship available for some roles.