Tech Lead Manager, AI Evaluations
$250k - $350k • Mountain View, CA
Posted 14h ago
Remote Work Policy
On-site
Categories
Applied AI Engineer
About the job
Glean is seeking a Tech Lead Manager of AI Evaluations to lead a team of applied AI scientists and engineers. This role will be responsible for building the evaluation strategy and platform that ensures Glean ships delightful and reliable AI experiences. The focus is on making evaluation a continuous part of the development and operation of models, harnesses, assistants, agents, and other AI products. The ideal candidate will guide the team in translating AI/ML research into reliable evaluation systems, build scalable infrastructure for evaluations, and collaborate with cross-functional teams to translate evaluation results into product improvements and better customer outcomes.
Responsibilities
- Own the AI evaluation engineering roadmap and technical direction from defining success to integrating results into product decisions.
- Build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture and replay, result analysis, and dashboards.
- Create and maintain representative evaluation sets covering common workflows, edge cases, adversarial inputs, and important user slices.
- Define a balanced quality measurement system for AI products, including correctness, groundedness, relevance, completeness, instruction following, citation quality, user preference, and task success.
- Evaluate agent performance including tool selection, multi-step trajectories, policy compliance, recovery, and final task state.
Requirements
- Lead a team of applied AI scientists and engineers.
- Translate emerging AI/ML research into reliable evaluation systems.
- Build scalable, efficient, and reusable evaluation infrastructure.
- Develop trusted quality signals for AI products.
- Work across R&D, product, and marketing teams.
- Ensure evaluations are reproducible, versioned, cost-aware, and easy to use.
- Protect held-out acceptance sets from tuning leakage.
- Measure AI product quality across various dimensions including correctness, groundedness, relevance, completeness, instruction following, citation quality, user preference, and task success.
- Evaluate agent-specific metrics like tool selection, multi-step trajectories, policy compliance, recovery, and final task state.