Machine Learning Research Scientist, Evaluations
$166k - $207k • Remote • San Francisco, CA; Seattle, WA; New York, NY
Posted 5d ago
Job Location
San Francisco, CA; Seattle, WA; New York, NY
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
Scale works with leading AI labs to accelerate progress in GenAI research, focusing on LLM post-training and evaluation. This role is on the evaluation pod, building benchmarks and diagnosing model failure modes in text and multimodal modalities. You will develop rigorous evaluations to reveal where and why frontier models fail, collaborating with researchers and engineers to define best practices in evaluation-driven AI development. You will also partner with foundation model labs to translate failure analysis into technical and strategic input for next-generation generative AI models.
Responsibilities
- Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents, focusing on root cause analysis.
- Design and build benchmarks and evaluation methods to measure LLM capabilities in text and multimodal modalities.
- Apply post-training expertise (SFT, RLHF, reward modeling) to connect observed failures to data and training interventions.
- Publish research findings in top-tier AI conferences.
Requirements
- Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
- Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
- Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning.
- Experience with LLM evaluation or benchmark development.
- Excellent written and verbal communication skills.
- Published research in machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
- Previous experience in a customer-facing role.
Benefits
- Base salary
- Equity
- Comprehensive health, dental and vision coverage
- Retirement benefits
- Learning and development stipend
- Generous PTO
- Commuter stipend