Senior Machine Learning Engineer, Agent Eval Platform
$161k - $274k • Santa Clara, CALIFORNIA, United States • Full-time
Posted 7d ago
Job Location
Santa Clara, CALIFORNIA, United States
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
Moveworks' AI agents are designed to act, plan, and interact with enterprise systems, impacting millions of employees. This role focuses on improving these agents by building a system that identifies failures, determines root causes, verifies fixes, and creates permanent tests from resolved issues. The work involves enhancing the agent's surrounding components like instructions, tool descriptions, routing, knowledge, and memory, rather than training the core language model itself. It's a blend of machine learning and systems engineering, requiring the ability to develop infrastructure for reproducible signal generation across diverse customer environments. This position tackles the open research problem of LLMs evaluating LLMs, particularly in the context of agents performing irreversible actions in complex enterprise systems.
Responsibilities
- Build a shared judge with configurable rubrics for evaluating agent performance.
- Implement deterministic validators for checkable outcomes and LLM judges for subjective aspects.
- Develop scoring mechanisms that report confidence levels to route uncertain judgments for human review.
- Establish a calibration loop with human-labeled data to improve judge consistency.
- Design systems to guard against correlated blind spots between user simulators and judges.
- Address offline-online divergence by identifying reasons for discrepancies and re-seeding simulation suites with production failures.
- Develop components for agent self-improvement without altering the underlying model.
- Identify areas where agents fall short in real-world use, including subtle failures.
- Generate synthetic data and environments with expected outcomes from real failures or anticipated needs.
- Determine the cause of agent failures and which part of the harness needs modification (instructions, tools, routing, knowledge, memory).
- Optimize agent components against simulations rather than live production traffic.
- Implement checks for proposed changes on unseen cases to catch regressions.
- Ensure fixed failures are converted into permanent tests to strengthen the evaluation suite.
- Build infrastructure for reproducible, scalable agent improvement loops across multiple customer organizations.
- Develop guardrails to ensure agent outcomes align with desired end-state objectives.
Requirements
- 2+ years in applied ML, data science, or ML-adjacent engineering with shipped, utilized work.
- Experience translating subjective human judgment into actionable measurements for humans and models.
- Strong applied ML fundamentals, comfortable evaluating, prompting, and fine-tuning LLMs.
- Proficient in Python and capable of shipping production-grade code.
- Ability to clearly articulate complex problems and convince others of the meaning of data.
- High degree of ownership and a bias towards shipping at startup pace.
- Comfort with ambiguity and judging when a measurement is sufficient for action.
- Experience with LLM-as-judge or automated evaluation design and calibration against human judgment.
- Experience with human annotation programs, including rubric authoring and label quality.
- Experience with search ranking, recsys, or online experimentation evaluation, including offline/online divergence and rater agreement.
- Experience with fine-tuning and evaluating smaller models (SFT, preference tuning, distillation).
- Experience with reward modeling, RLHF/RLAIF, or process reward models.
- Experience with agent trajectory analysis and step-level fault attribution.
- Experience with prompt engineering as a disciplined, versioned, tested, and measured engineering practice.
Benefits
- Base pay of $161,300 - $274,200
- Equity (when applicable)
- Variable/incentive compensation
