Staff Machine Learning Engineer, Agent Eval Platform
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
Job Location
Mountain View, CALIFORNIA, United States
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
Machine Learning Engineer
About the job
Moveworks' AI agents are designed to act, plan, and modify enterprise systems, impacting millions of employees. This role focuses on building the crucial 'judgement layer' for our agent evaluation platform. You will develop the rubrics, judges, and calibration methods necessary to accurately score agent actions across complex, multi-step interactions. The goal is to create a signal precise enough to train agents for improved performance, turning evaluation into a training mechanism. This position is centered on applied ML, tackling the open research problem of LLMs evaluating LLMs, specifically within the context of agents performing real-world actions in enterprise environments.
Responsibilities
- Design and calibrate a shared base judge with configurable rubrics for evaluation authors.
- Implement deterministic validators for verifiable state changes and utilize LLM judges for subjective aspects of agent actions.
- Develop scoring mechanisms that report confidence levels, routing uncertain judgments for human review.
- Establish and maintain a calibration loop with human-labeled trajectories in partnership with the annotation team.
- Fine-tune smaller judge models when off-the-shelf solutions are insufficient.
- Mitigate correlated blind spots between user simulators and judges.
- Analyze and resolve offline-online divergence issues, re-seeding simulation suites from production failures.
- Develop a process reward model from the trajectory judge to provide step-level signals for agent optimization.
- Utilize reward signals to optimize agent behavior, including prompts, tool selection, and planner logic.
- Build the infrastructure for future reinforcement learning efforts, including versioned scenarios and repeatable simulated worlds.
- Ensure step-level scores serve as training and diagnostic tools, while end-state outcomes define success criteria.
Requirements
- 8+ years in applied ML, data science, or ML-adjacent engineering with a proven track record of shipped products.
- Experience translating subjective human judgment into actionable measurements for humans and models.
- Strong applied ML fundamentals, with experience evaluating, prompting, and fine-tuning LLMs.
- Proficient in Python and capable of shipping production-grade code.
- Ability to clearly articulate complex problems and convince stakeholders of measurement validity.
- High degree of ownership and a bias towards rapid shipping in a startup environment.
- Comfort with ambiguity and the judgment to determine when a measurement is sufficiently reliable.
- Experience with LLM-as-judge or automated evaluation design and calibration against human judgment.
- Experience managing human annotation programs, including rubric authoring and label quality.
- Experience with search ranking, recsys, or online experimentation evaluation, including offline/online divergence and rater agreement.
- Experience with fine-tuning and evaluating small models (SFT, preference tuning, distillation).
- Experience with reward modeling, RLHF/RLAIF, or process reward models.
- Experience with agent trajectory analysis and step-level fault attribution.
- Experience with prompt engineering as a disciplined, versioned, and measured engineering practice.