Staff ML Engineer, Frontier AI

About the job

Responsibilities

• Define the AI Quality and Evaluation Strategy, including automated graders, regression testing, human evaluation, failure taxonomies, and offline-to-online validation.
• Drive production model improvements by identifying failure modes and leading enhancements across prompting, retrieval, context, routing, data, and post-training.
• Advance post-training capabilities using techniques like supervised fine-tuning, preference optimization, reinforcement learning, distillation, and synthetic data.
• Architect agentic AI systems, setting technical direction for production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
• Build self-improving AI loops that convert production failures and user feedback into better datasets, evaluations, and model behavior.
• Co-develop the 12-month technical roadmap, balancing near-term product wins with longer-term research and platform investments.
• Raise the technical bar by leading technical reviews, mentoring engineers, establishing best practices, and creating reusable systems and frameworks.
• Stay at the cutting edge by distilling insights from recent research in LLMs, agents, post-training, NLP, speech, and multimodal AI, and driving experiments.

Requirements

• 7+ years in production ML, research engineering, or applied AI.
• Deep expertise with modern LLMs, transformers, and complex production AI systems.
• Experience leading or materially shaping consequential AI systems with real production impact.
• Ability to reason across models, data, orchestration, evaluation, serving, and product behavior.
• Significant experience designing evaluations for LLMs, agents, or other complex AI systems.
• Ability to turn ambiguous product-quality problems into robust metrics, datasets, and experiments.
• Experience with grader bias, leakage, contamination, misleading aggregate metrics, regression detection, and offline-online mismatch.
• Experience establishing whether model or system improvements translate into better user outcomes.
• Experience building or architecting production systems involving multiple models, tools, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
• Comfortable making architectural decisions that span multiple products or teams.
• Experience materially improving model behavior through post-training or model adaptation techniques.
• Understanding of how data quality, objective design, evaluation, and post-training choices affect downstream model behavior.
• Proficiency in Python and modern ML frameworks; PyTorch preferred.
• Comfort with modern MLOps, CI/CD, observability, and containerized deployments.
• Ability to remain deeply hands-on: writes code, inspects traces, analyzes failures, and prototypes solutions.
• Skilled at building large, high-quality datasets and feedback loops.
• Experience using production failures, user feedback, and active learning.