New: upload your resume and get matched to AI jobs. Free to start →

Senior Data Scientist for Copilot Evals

$120k - $235k • United States, Washington, Redmond • Full-time

Posted 24d ago

Job Location

United States, Washington, Redmond

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Engineer

About the job

The Customer and Analytics Driven Evals Team (CADET) is building a customer-grounded quality system for Copilot. The mission is to rapidly identify critical customer scenarios, represent them in evaluation and learning assets, run continuous quality gates, and translate failures into product and model improvements. This role involves building the measurement and decision system to guide quality investments by combining various signals like DSAT, usage, customer engagement, product feedback, and evaluation data to create a comprehensive view of customer quality. The goal is to define taxonomies, metrics, prioritization models, analyses, and reporting mechanisms that transform fragmented signals into clear decisions, investigate loss patterns, shape evaluation portfolios, and ensure quality gates reflect real customer experiences.

Responsibilities

  • Build and operate a data-driven prioritization framework for quality forums and partner teams, combining various signals to guide quality investments.
  • Define and maintain taxonomies for intent, customer scenarios, coverage, and loss patterns for consistent use across signal intake, triage, evaluation, and reporting.
  • Measure how well evaluation portfolios represent production traffic, customer segments, workflow complexity, locales, grounding paths, and material failure modes.
  • Identify underrepresented customers, intents, scenarios, and loss patterns, and translate these gaps into evaluation and data-collection priorities.
  • Design quality gates for top intents and customers, including success thresholds, segmentation, run cadence, escalation criteria, and reporting.
  • Build recurring scorecards connecting offline evaluation movement with online measures (DSAT, task completion, retries, abandonment, escalation) to detect quality changes and alert owners.
  • Analyze offline-online agreement, evaluation freshness, regression coverage, grader reliability, and quality movement over time.
  • Develop sampling, weighting, deduplication, clustering, and trend-detection approaches for noisy signals using resource- and performance-optimized data-analysis solutions.
  • Use causal and experimental methods to distinguish correlation, attribution, and treatment impact, translating findings into concrete product, model, data, and evaluation investment decisions.
  • Partner to translate customer evidence into valid task distributions, datasets, metrics, and reward signals, encoding metrics, taxonomies, data-quality checks, and reporting into automated pipelines.
  • Leverage team signals from customer engagements to understand workflows, business impact, and gaps, producing clear recommendations for investments and communicating them to senior leaders.
  • Establish solid practices for data provenance, privacy, responsible use, reproducibility, and metric governance.

Requirements

  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience OR Master's Degree AND 3+ years data-science experience OR Bachelor's Degree AND 5+ years data-science experience OR equivalent experience.
  • Solid experience using data to shape product strategy and decisions in a complex, high-scale product or platform environment.
  • Expertise in SQL and at least one analytical programming language such as Python or R.
  • Solid foundation in statistical analysis, experimentation, sampling, segmentation, measurement, and data visualization.
  • Experience integrating noisy quantitative and qualitative signals into actionable prioritization or measurement frameworks.
  • Ability to define durable metrics and taxonomies, explain their limitations, and prevent misleading interpretation.
  • Demonstrated ability to communicate complex analysis clearly to technical, product, and executive audiences.
  • Experience with AI product quality, LLM or agent evaluation, DSAT or customer feedback analysis, experimentation, or model telemetry.
  • Experience designing representative datasets and coverage models.