Principal Software Engineer for Copilot Evals
$143k - $275k • United States, Washington, Redmond • Full-time
Posted 1mo ago
Job Location
United States, Washington, Redmond
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
The Customer and Analytics Driven Evals Team (CADET) is developing a customer-grounded quality system for Copilot. Our mission is to quickly identify the most important customer scenarios, represent them accurately in evaluation and learning materials, continuously run quality gates, and transform every significant failure into reusable product and model improvements. We integrate customer signals and feedback with deep customer engagement to create representative evaluation sets. Operating in a fast-paced environment, we link customer-grounded quality issues with quality teams to enhance Copilot's quality and product innovation.
We are seeking a Principal Software Engineer to architect and build the production system for CADET's customer-grounded quality loop, collaborating with platform teams. This is a high-impact role that will define technical direction, drive rapid delivery, and align engineering, science, product, and platform teams as we transition from manual signals and one-time evaluations to an automated, agentic system. You will integrate diverse data sources, AI agents, evaluation infrastructure, RLE platforms, model and product telemetry, and human review into a reliable and inspectable end-to-end system. Your responsibilities will include setting architecture across organizational boundaries, establishing engineering standards, aligning partner teams, and leading the most technically challenging aspects of the system.
Responsibilities
- Own and drive the end-to-end CADET architecture, integration contracts, and technical execution for the customer-signal and evaluation system, spanning capture, enrichment, prioritization, triage, asset generation, execution, diagnosis, reporting, and retention.
- Create scalable services and pipelines that reproduce customer pain points across scenarios, business segments, and interventions through automated grounding-data generation, intent classification, deduplication, clustering, taxonomy assignment, coverage analysis, loss-pattern detection, and work-item routing.
- Automate recurring evaluation execution and quality gates for top intents and customers, including scheduling, versioning, reproducibility, failure handling, and result publication.
- Establish data contracts and extensible schemas for customer signals, workflows, evaluation cases, rubrics, graders, runs, results, attribution, and regression assets.
- Develop quality-attribution workflows that combine automated diagnosis with human review; route losses to the appropriate product, data, model, tool, orchestration, or evaluation owner; track remediation; add regression protection; and verify improvement in customer outcomes.
- Create observability and benchmarks for pipeline health, agent behavior, data quality, grader health, coverage, cost, latency, operational reliability, and regression detection across CADET workflows and tooling.
- Design safe human-in-the-loop gates for ambiguous data, sensitive customer evidence, taxonomy changes, evaluation publication, and consequential quality decisions.
- Mentor engineers across the team and raise engineering quality through clear interfaces, testing, security, operational discipline, and technical documentation.
Requirements
- Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python, OR equivalent experience.
- Extensive experience designing and delivering large-scale distributed systems, data platforms, developer platforms, or AI infrastructure in production.
- Practical exposure to large language model technologies, agent systems, or AI/ML platforms.
- Demonstrated principal-level technical leadership across teams and organizational boundaries.
- Solid understanding of security, privacy, data governance, and responsible engineering for sensitive data.
- Excellent communication and collaboration skills with the ability to influence senior engineering, science, and product leaders.
- Experience building AI/ML platforms, LLM or agent systems, evaluation infrastructure, experimentation systems, or model pipelines.
- Experience with retrieval, tool-using agents, prompt and rubric management, human-in-the-loop systems, or automated diagnosis.
- Experience building data and workflow platforms that combine batch, streaming, unstructured, and human-generated inputs.
- Experience with RLEs, synthetic environments, post-training pipelines, or large-scale evaluation execution.
- Experience integrating customer feedback, product telemetry, and engineering work-management systems.
- Experience operating high-availability services and driving platform adoption across multiple product teams, with knowledge of competing AI models.
