New: upload your resume and get matched to AI jobs. Free to start →

Staff Machine Learning Engineer, Agent Eval Platform

$176k - $308k • Mountain View, CALIFORNIA, United States • Full-time

Posted 7d ago

Job Location

Mountain View, CALIFORNIA, United States

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Agent Engineer

About the job

Moveworks' AI agents are designed to act, plan, and modify enterprise systems for millions of employees. This role focuses on improving these agents by building a system that enables them to self-improve. You will develop the loop that identifies failures in real-world usage, determines the responsible component within the agent's harness, validates proposed fixes against unseen cases, and converts each resolved failure into a permanent test. The work involves enhancing the system around the model, including instructions, tool descriptions, routing, knowledge, and memory, rather than training the model itself. This position blends ML and systems engineering, requiring you to build infrastructure for reproducible signal generation across numerous customer organizations, focusing on applied ML in the evolving domain of LLMs judging LLMs, especially within the context of agents performing irreversible actions in complex enterprise environments.

Responsibilities

  • Develop a shared base judge with per-item rubrics expressed as configuration for eval authors.
  • Implement deterministic validators for checkable world state and utilize an LLM judge for subjective aspects.
  • Design scoring that reports its own confidence to route uncertain judgments to humans.
  • Establish a calibration loop against human-labeled trajectories in partnership with an annotation team.
  • Guard against correlated blind spots between user simulators and judges.
  • Address offline-online divergence by explaining discrepancies and re-seeding simulation suites from production failures.
  • Build components for the agent harness to improve from its own use without changing the underlying model.
  • Surface areas where the agent falls short in real use, including failures that appear successful.
  • Generate synthetic data and environments with expected outcomes and checks from real failures or anticipated needs.
  • Determine the root cause of agent failures and identify the specific harness component needing modification (instructions, tools, routing, knowledge, or memory).
  • Optimize agent components against simulations rather than production traffic.
  • Verify proposed changes on cases they were not tuned on to catch regressions.
  • Ensure every fixed failure becomes a permanent test to strengthen the evaluation suite.
  • Implement mechanisms to scope learnings to individual organizations and require admin approval for org-specific changes.
  • Build and maintain the infrastructure for a reproducible evaluation loop that operates at scale across customer organizations.
  • Implement guardrails to ensure agent outcomes remain honest and aligned with end-state objectives.

Requirements

  • 5+ years in applied ML or ML-adjacent engineering with shipped, operated, and relied-upon systems.
  • Experience transforming subjective human judgment into a reliable measurement for team decision-making and validation against human labels.
  • Proven ability to improve multi-component systems (pipelines, retrieval, prompts, policies, routing) through measurement, not just by changing model weights.
  • Strong applied ML fundamentals and comfort evaluating and prompting LLMs as components.
  • Proficiency in Python and production engineering, including services, data pipelines, and designs for reproducibility (versioned data, deterministic replay, recorded inputs).
  • Ability to clearly articulate complex problems and convince others of the meaning of data.
  • High degree of ownership and a bias toward shipping at startup pace.
  • Comfort with ambiguity and the judgment to determine when a measurement is actionable.
  • Experience in LLM-as-judge or automated evaluation design and calibration against human judgment.
  • Experience with human annotation programs, including rubric authoring, label quality, and annotator throughput.
  • Experience with experimentation rigor, including held-out sets, upfront decision rules, variance/noise analysis, and regression detection.
  • Experience with failure analysis at scale, including classification, clustering, and error attribution in logs or traces.
  • Experience with replay, simulation, or sandboxed test infrastructure.
  • Experience with synthetic data generation for evaluation, including scenario generation, seeded environments, and quality controls.
  • Experience with prompt and harness optimization as an engineering discipline.