Member of Technical Staff, Evals
$200k - $350k • San Francisco, CA • FullTime
Posted 18d ago
Job Location
San Francisco, CA
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Handshake is seeking a Member of Technical Staff, Evals to help define how frontier AI systems are measured, understood, and improved. This is a high-ownership role for researchers who build, partnering with AI researchers, domain experts, and customers to develop new benchmarks, reward and verifier systems, agent-evaluation methodologies, and data-quality techniques. Early members of the team will have significant influence over technical direction, operating culture, and the open-source software, benchmarks, and research products built. The company prioritizes demonstrated research capability, technical judgment, and a builder's mindset over specific titles, degrees, or career paths.
Responsibilities
- Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments.
- Develop reward models, programmatic verifiers, graders, and other feedback systems to make model behavior measurable and improvable.
- Research evaluation representativeness, difficulty, reliability, and resistance to shortcutting or reward hacking.
- Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
- Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and translate learnings into the next benchmark or system.
- Contribute to the field publicly through benchmarks, open-source tools, research, and technical writing.
Requirements
- PhD in ML/AI, computer science, data science, or related fields, or equivalent research experience in industry.
- Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
- Demonstrated ability to build and ship high-quality software, benchmarks, or datasets (e.g., strong GitHub profile/OSS contributions, or product portfolio).
- Strong Python skills and experience building scalable software and working with agents.
- Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes.
- Comfort operating in an ambiguous, fast-moving environment with substantial ownership.
Benefits
- Equity in a fast-growing company
- 401(k) match
- Competitive compensation
- Financial coaching
- Paid parental leave
- Fertility benefits
- Parental coaching
- Medical, dental, and vision insurance
- Mental health support
- $500 wellness stipend
- $2,000 learning stipend
- Ongoing development
- Commuting support
- Free lunch
- Gym in SF office
- Flexible PTO
- 15 holidays + 2 flex days
- Team outings
- Referral bonuses