New: upload your resume and get matched to AI jobs. Free to start →

Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)

$143k - $275k • United States, Washington, Redmond • Full-time

Posted 1mo ago

Job Location

United States, Washington, Redmond

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Engineer

About the job

We are seeking a Principal AI Architect to define, build, and scale evaluation systems for AI products. This role bridges engineering, applied science, product architecture, and AI evaluation, requiring deep technical judgment and systems thinking. You will focus on evaluating agentic AI systems, which are often nondeterministic and context-dependent, making traditional testing insufficient. Your work will help teams understand agent behavior, identify failure points, and determine the best ways to address them, whether through prompts, tools, orchestration, retrieval, or product UX. The goal is to make AI product quality measurable, actionable, and integral to the entire product lifecycle, from early evaluation to post-release performance monitoring.

Responsibilities

  • Define the technical vision and architecture for AI, LLM, RAG, and agent evaluation systems.
  • Design evaluation frameworks for assessing product quality throughout the development lifecycle.
  • Build a deep understanding of nondeterministic agentic system behavior across various contexts.
  • Identify and analyze behavioral gaps, failure modes, and product quality risks.
  • Determine optimal solutions for identified issues across the system stack.
  • Partner with cross-functional teams to integrate evaluation methods into product codebases and workflows.
  • Integrate evaluation processes into build pipelines, release gates, and engineering workflows.
  • Develop systems for accessible and actionable evaluation results through dashboards and reports.
  • Build systems that connect various quality signals, including telemetry, offline evaluation, and human judgment.
  • Evaluate enterprise search, RAG, grounding, indexing, ranking, and relevance systems.
  • Translate ambiguous product goals into measurable evaluation strategies and technical plans.
  • Drive architecture decisions for evaluation components, services, and data pipelines.
  • Collaborate with product leaders to prioritize evaluation investments.
  • Mentor senior engineers and applied scientists on building reusable evaluation infrastructure.
  • Stay current with LLM evaluation methods, agentic systems, RAG evaluation, and responsible AI practices.
  • Evaluate products before code readiness, at launch, and post-ship.
  • Help teams understand AI system performance, failures, improvements, and responsible scaling.

Requirements

  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience OR Master's Degree AND 4+ years related experience OR Doctorate AND 3+ years related experience OR equivalent experience.
  • 5+ years experience creating publications (e.g., patents, libraries, peer-reviewed academic papers).
  • 2+ years experience presenting.