AI Research Engineer
$250k - $400k • Remote • San Francisco, CA
Posted 9h ago
Job Location
San Francisco, CA
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Research Engineer
About the job
Dscout is seeking an AI Research Engineer to join their new AI Training & Evals unit. This team focuses on building human data and evaluations to determine if AI performs effectively for real people in real-world contexts. While AI models may excel at benchmarks, they often surprise users in practice. This role bridges that gap by leveraging Dscout's decade of experience in capturing human behavior to create training data, evaluations, and benchmarks that AI teams can trust. The position offers a unique opportunity to work on unsolved problems in AI measurement, study real participants in their own context, and contribute to publishing novel methods and benchmarks.
Responsibilities
- Design rubrics and test their validity, reliability, and sensitivity.
- Measure the reliability of human raters and identify sources of disagreement.
- Calibrate AI judges against human judgment and study drift.
- Build quality signals to validate data points.
- Design Dscout's first long-horizon, real-user benchmark, including protocol, task design, and scoring.
- Develop methods to maintain the integrity of a living benchmark over time.
- Write research papers and manage the public release of benchmarks.
- Study how real-world human feedback impacts model performance.
- Explore differences between preference data from representative people and expert data.
- Develop methods for field deployment by engineers and the delivery team.
- Propose and lead new research programs, such as calibrating user simulations or evaluating AI agents.
- Co-author studies with external labs and academics.
- Lead the paper-reading group.
Requirements
- Strong publication record in evals, preference learning, or related fields is beneficial.
- Ability to identify important research questions.
- Skill in designing rigorous research methodologies.
- Capacity to execute projects rapidly and deliver useful answers.
- Experience with real-user data collection and analysis.
- Familiarity with AI evaluation methodologies.
- Ability to work independently and propose new research directions.
Benefits
- Work on a novel problem in AI measurement.
- Study real participants in their own context.
- Found and own a new benchmark.
- Opportunity to publish research with first authorship.
- Direct access to the Head of AI Research and GM.
- Work within a mature platform with live customers.
- Room to build and innovate.