Research Engineer, Post-Training Model Evaluations
$500k - $850k • San Francisco, CA | Seattle, WA
Posted 1mo ago
Job Location
San Francisco, CA | Seattle, WA
Tech Stack
Remote Work Policy
On-site
Categories
AI Research Engineer
About the job
Anthropic's production models undergo sophisticated post-training processes to enhance their capabilities, alignment, and safety. Decisions in production training depend on knowing how the model is really behaving. This role's mission is to have the best observability into production post-training runs at Anthropic, providing the clearest, most trusted view of model quality during training. You will shape what gets measured, keep the signal trustworthy and timely, and investigate surprising results to inform post-training and release decisions.
Responsibilities
- Steer the eval strategy for production Claude models, studying and optimizing the eval mix for accurate and complete model quality assessment.
- Run and monitor the eval fleet live to understand how each Claude model is developing during training.
- Advise teams across post-training on eval methodology and help eval authors meet production standards.
- Investigate regressions in production runs and inform training interventions.
- Build dashboards, alerts, and reports for tracking model quality.
Requirements
- Designed, run, and analyzed evaluations for ML models at scale.
- Deep care for measurement quality and critical evaluation of numbers.
- Ability to translate ambiguous results into clear recommendations and influence cross-team direction.
- Strong Python skills and comfort with production systems.
- Clarity and rigor in debugging complex, time-sensitive issues.
- Thrive in high-pressure, multi-priority environments during live training runs.
- Care about societal impacts of work and responsible shipping of frontier models.
- Hands-on experience post-training large language models (preferred).
- Research experience in ML evaluation or benchmarking (preferred).
- Background in statistics and experimental design (preferred).
- Experience developing robust evaluation metrics for ML systems (preferred).
Benefits
- Annual compensation range: $500,000 - $850,000 USD
- Visa sponsorship available
- Hybrid work arrangement (at least 25% in office)