Research Lead, Training Insights
Remote • Remote-Friendly (Travel Required) | San Francisco, CA; San Francisco, CA | New York City, NY
Posted 16d ago
Job Location
Remote-Friendly (Travel Required) | San Francisco, CA; San Francisco, CA | New York City, NY
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Research Engineer
About the job
As a Research Lead on the Training Insights team, you will develop the strategy for, and lead execution on, how we measure and characterize model capabilities across training and deployment. This is a hands-on leadership role where you will drive original research into new evaluation methodologies while leading a small team of researchers and research engineers. Your work will span the full lifecycle of model development, from researching and building new long-horizon evaluations to developing novel approaches for measuring emerging capabilities and deepening our understanding of how those capabilities develop. You will also take a cross-organizational view, working across various teams to map the landscape of model evaluations and identify critical gaps. This role carries significant visibility and impact, helping to shape the evaluation narrative for model releases and contributing directly to how Anthropic communicates about its models. Done well, you will change how the industry measures and understands model capabilities, significantly furthering our safety mission.
Responsibilities
- Build new novel and long-horizon evaluations
- Develop novel measurement approaches for understanding how model capabilities emerge and evolve during RL training
- Lead strategic evaluation coverage across the company
- Shape the evaluation narrative for model releases
- Lead and mentor a small team of researchers and research engineers, setting research direction and fostering a culture of rigorous, creative research
- Design evaluation frameworks that balance scientific rigor with the practical demands of production training schedules
- Build and maintain relationships across Anthropic's research organization to ensure evaluation insights inform training and deployment decisions
- Contribute to the broader research community through publications, open-source contributions, or external engagement on evaluation best practices
Requirements
- Significant experience designing and running evaluations for large language models or similar complex ML systems
- Experience leading technical projects or teams, either formally or through sustained ownership of critical research directions
- Comfortable designing experiments and writing code, moving between research and implementation fluidly
- Strategic thinking about what to measure and why
- Ability to synthesize information across multiple teams and workstreams to form a coherent picture of model capabilities
- Clear communication of complex technical findings to both technical and non-technical audiences
- Results-oriented and thrive in fast-paced environments where priorities shift based on research findings
- Deep care for AI safety and a desire for work to directly influence AI development and deployment