Research Scientist, Agentic Data & Benchmarking
$150k - $450k • Remote • Sunnyvale, CA • Full-time
Posted 4mo ago
Job Location
Sunnyvale, CA
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Research Engineer
About the job
We are seeking a Research Scientist to focus on Agentic Data & Benchmarking. This role involves designing and executing evaluations for agentic capabilities, including multi-step reasoning, tool use, and long-horizon planning. You will build and maintain robust evaluation harnesses to ensure reliable, large-scale benchmarking against model checkpoints, identifying regressions and monitoring model health. Additionally, you will conduct experiments to understand how various factors like prompting, sampling, and environment design influence agentic performance on both internal and public benchmarks. A key part of this role is diagnosing anomalous evaluation results during training, pinpointing the root cause (model, data, harness, or infrastructure), and communicating findings clearly. You will also be responsible for sourcing, generating, and curating high-quality agentic training data, including trajectories and tool-use traces, and designing and scaling RL environments and reward signals. Managing technical relationships with external data vendors and domain experts, evaluating data quality, and developing QA frameworks to ensure data and benchmark integrity are also crucial aspects of this position. You will contribute to technical reports, research publications, and open-source benchmarks and tooling, partnering with research and product teams to translate capability goals into measurable data and evaluation artifacts.
Responsibilities
- Design and run evaluations of agentic capabilities, turning ambiguous notions of intelligence into defensible, reproducible metrics.
- Build and harden evaluation harnesses for reliable, large-scale benchmarks.
- Run experiments characterizing how prompting, sampling, scaffolding, and environment design affect agentic performance.
- Diagnose anomalous eval results mid-training-run and communicate the cause.
- Source, generate, and curate high-quality agentic training data.
- Design and scale RL environments and reward signals, and measure their impact.
- Manage technical relationships with external data vendors and domain experts.
- Develop QA frameworks to catch reward hacking, label noise, and contamination.
- Contribute to technical reports, research publications, and open-source benchmarks and tooling.
- Partner with research and product teams to translate capability goals into measurable data and evaluation artifacts.
Requirements
- BS, MS, or PhD (or equivalent experience) in Computer Science, Machine Learning, or a related field.
- 2+ years of experience with emphasis on evaluations and/or training-data curation for ML systems (LLM training/fine-tuning, RL, or distributed ML systems).
- Strong Python and PyTorch development experience.
- Demonstrated experience designing and deep-diving into evaluations, or curating and generating training datasets.
- Hands-on experience using LLM agents in personal or professional work.
- Habit of reading through raw data and trajectories to understand them and spot issues.
- Experience with reinforcement learning, reward design, or RL environment construction for LLMs.
- Background in statistics and experimental design.
- Experience with large-scale dataset sourcing, curation, and processing.
- Strong knowledge of the literature on agent evaluation, RL, LLM reasoning, and tool use.
- Experience building or operating data pipelines and evaluation infrastructure reliable at scale.
- Experience evaluating or generating data for software-engineering or computer-use agents.
- Contributions to published research, public benchmarks, and/or open-source ML software.