Senior Machine Learning Engineer, Agent Eval Platform
Remote • Santa Clara, CALIFORNIA, United States • Full-time
Posted 1mo ago
Job Location
Santa Clara, CALIFORNIA, United States
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
Machine Learning Engineer
About the job
Moveworks' AI agents are designed to act, plan, and modify enterprise systems, impacting millions of employees. This role focuses on building the "judgement layer" of our agent evaluation platform. You will develop the rubrics, judges, and calibration methods necessary to accurately score agent actions across multi-step trajectories. The goal is to create a reliable signal that not only measures performance but also serves as a reward mechanism for training agents. This is an applied ML role tackling the open research problem of LLMs judging LLMs, specifically applied to agents interacting with real-world enterprise environments.
Responsibilities
- Design and calibrate a shared base judge with configurable rubrics for evaluating agent actions.
- Implement deterministic validators for checkable states and use LLM judges for subjective aspects of agent performance.
- Develop scoring mechanisms that report confidence levels, routing uncertain judgements to human review.
- Establish and maintain a calibration loop with human-labeled trajectories in partnership with an annotation team.
- Fine-tune smaller judge models when off-the-shelf solutions are insufficient.
- Mitigate correlated blind spots between user simulators and judges.
- Analyze and address offline-online divergence in agent performance.
- Develop a process reward model from the trajectory judge to provide dense, step-level signals for agent training.
- Utilize the reward signal to optimize agent behavior, including prompts, tool selection, and retrieval.
- Build the infrastructure for future reinforcement learning efforts, including versioned scenarios and a repeatable simulated world.
- Ensure step-level scores are used for training and diagnosis, while end-state outcomes remain the ultimate measure of success.
Requirements
- 5+ years in applied ML, data science, or ML-adjacent engineering with a track record of shipped products.
- Experience translating subjective human judgment into actionable measurements for humans and models.
- Strong applied ML fundamentals, with experience evaluating, prompting, and fine-tuning LLMs.
- Proficiency in Python and experience shipping production-grade code.
- Ability to clearly communicate complex problems and technical concepts.
- High degree of ownership and a bias towards shipping at startup pace.
- Comfort with ambiguity and the judgment to determine when a measurement is sufficient.
- Experience with LLM-as-judge or automated evaluation design and calibration against human judgment.
- Experience with human annotation programs, including rubric authoring and label quality management.
- Experience in search ranking, recsys, or online experimentation evaluation, including offline/online divergence and rater agreement.
- Experience fine-tuning and evaluating small models (SFT, preference tuning, distillation).
- Experience with reward modeling, RLHF/RLAIF, or process reward models.
- Experience with agent trajectory analysis and step-level fault attribution.
- Experience with prompt engineering as an engineering discipline (versioned, tested, measured).