Evals Infrastructure Tech Lead / Manager
San Francisco, CA
Posted 8d ago
About the job
Anthropic is seeking an experienced tech lead to join its Evals Infrastructure team. This role involves building and managing the systems that measure the capabilities of AI models, ensuring these evaluations are fast, reliable, and trustworthy at scale. You will work at the intersection of inference, research, and infrastructure engineering, orchestrating large-scale distributed systems for frontier model evaluations, developing researcher harnesses, and ensuring the reproducibility and interpretability of results. Your work will directly influence the development of AI systems and their alignment with human interests.
Responsibilities
- Lead the team building distributed systems for scheduling, orchestrating, and executing frontier model evals.
- Own eval throughput and cost, including compute allocation, queueing, and caching.
- Build and scale harnesses for researchers to define, run, and iterate on evals.
- Ensure eval results are trustworthy through determinism, reproducibility, and uncertainty quantification.
- Integrate eval signals into dashboards and review processes for decision-making.
- Contribute as an engineer while managing the team, prioritizing work, and coaching reports.
Requirements
- Experience leading end-to-end technical projects on large-scale distributed systems.
- 1+ years of experience managing engineers or tech-lead experience with direct reports.
- Proficiency in Python and Rust.
- Experience building high-throughput, fault-tolerant systems on cloud or on-prem accelerator fleets.
- Focus on measurement quality beyond just pipeline uptime.
- Ability to communicate effectively with researchers and translate research needs into infrastructure.
- Deep interest in advanced AI and commitment to safe development.
Benefits
- Annual compensation range: $500,000 - $850,000 USD
- Visa sponsorship available