AI Engineer - Data and Evals
$200k - $280k • Remote • San Francisco Bay Area
Posted 11h ago
Job Location
San Francisco Bay Area
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Engineer
About the job
The bottleneck in building reliable, useful AI is no longer writing code, but defining and measuring what good looks like. This role focuses on data and evaluation science for Glean's AI products, building evalsets, agentic LLM-judges, and benchmarks that capture quality. You will develop the platform and infrastructure to run these evaluations reliably at scale, and create enterprise simulation environments to test assistant and agent performance. Starting from real production usage, you will translate learnings into evaluations that guide improvements across the entire stack, from agent harnesses to tool use, skill iteration, model routing, and post-training of task-specific models. You will also bring new research into production and collaborate with R&D, product, and GTM teams to translate evaluation findings into better products and customer outcomes.
Responsibilities
- Build evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, covering edge cases and key user slices.
- Build and scale the eval platform and agent observability, including experiment pipelines, trace capture and replay, and dashboards, ensuring reproducible and affordable runs.
- Build agentic LLM verifiers for correctness, groundedness, citation quality, and task success, and calibrate them against human labels using clear rubrics.
- Run evals at every stage, from CI and pre-release suites to sampled production traffic, and report results with confidence intervals.
- Build enterprise simulation environments with simulated users and realistic company data to test multi-step agent tasks.
- Diagnose failures by slice and trace, and turn eval results into concrete fixes, weighing trade-offs.
Requirements
- Experience building evalsets and benchmarks.
- Experience building and scaling evaluation platforms and agent observability.
- Experience building agentic LLM verifiers.
- Experience running evals at various stages of the development lifecycle.
- Experience building enterprise simulation environments.
- Ability to diagnose failures and translate eval results into actionable fixes.
- Familiarity with enterprise AI products and their evaluation.