Principal Applied Scientist for Copilot Evals
$143k - $275k • United States, Washington, Redmond • Full-time
Posted 1mo ago
Job Location
United States, Washington, Redmond
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
The Customer and Analytics Driven Evals Team (CADET) is building a customer-grounded quality system for Copilot. The mission is to rapidly identify critical customer scenarios, represent them in evaluation and learning assets, run continuous quality gates, and transform failures into product and model improvements. This role involves working directly with enterprise customers and Copilot teams to translate high-value workflows, expected outcomes, and recurring pain points into trusted evaluation and learning signals. The ideal candidate will set the scientific direction for customer-grounded quality, define what "good" means, assess eval portfolios, diagnose model and agent failures, and convert evidence into reusable evaluation assets and post-training priorities. This position requires a blend of scientific depth and product judgment to address ambiguous customer problems with rigorous, scalable methods in collaboration with applied researchers and ML engineers.
Responsibilities
- Set the scientific strategy for customer-grounded quality across priority Copilot intents, defining "good" and tradeoffs across quality and safety attributes.
- Translate user research, enterprise customer feedback, DSAT, and production incidents into evaluation and post-training priorities, leading the creation of reusable assets for high-value workflows and failure patterns.
- Develop methods to assess evaluation-set representativeness, coverage, freshness, discrimination, grader reliability, and alignment with production outcomes, identifying gaps and drift.
- Design behavior and task evaluations, rubrics, graders, and calibration methods to translate qualitative customer expectations into measurable quality.
- Establish methods to attribute quality losses across grounding, retrieval, tools, orchestration, model reasoning, response generation, and evaluation, linking offline and online signals.
- Uphold scientific rigor, reproducibility, documentation, and interpretation of results, while mentoring staff and influencing evaluation and post-training strategy.
Requirements
- Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience OR Master's Degree AND 4+ years related experience OR Doctorate AND 3+ years related experience OR equivalent experience.
- Advanced degree in computer science, machine learning, statistics, applied mathematics, or a related quantitative field, or equivalent practical experience.
- Significant experience applying machine learning, natural language processing, information retrieval, reinforcement learning, experimentation, or evaluation methods to complex production systems.
- Experience designing evaluations, metrics, experiments, datasets, graders, or reward functions for AI or agentic systems.
- Hands-on ability to inspect model outputs, identify behavioral patterns, and translate qualitative judgments into testable hypotheses and measurable evaluation criteria.
- Solid understanding of statistical inference, sampling, measurement validity, bias, uncertainty, and experimental design.
- Solid written and verbal communication skills, with demonstrated ability to drive results across organizational boundaries.
- Experience with large language models, copilots, agents, tool use, retrieval-augmented generation, or enterprise grounding.
- Experience running or partnering on RLHF, direct preference optimization, instruction tuning, fine-tuning, or human-preference data programs.
- Experience connecting offline metrics with online product behavior and customer outcomes.
- Experience working directly with enterprise customers or translating qualitative research and customer signals into scientific assets.
- Publication record, patents, or demonstrated industry impact in relevant applied research areas.
Benefits
- Base pay range: USD $142,800 - $274,800 per year (US)
- Base pay range: USD $188,000 - $304,200 per year (San Francisco Bay area and New York City metropolitan area)
- Certain roles may be eligible for benefits and other compensation.
