Researcher, Evals
*HQ - San Francisco, CA • FullTime
Posted 9mo ago
Job Location
*HQ - San Francisco, CA
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Research Engineer
About the job
The New Horizons Evaluations team is reimagining how we measure progress in interactive machine intelligence. As Evaluations Lead, you will design evaluation frameworks that capture not just what models know — but how they reason, remember, and interact over time. You’ll work at the intersection of research, product, and infrastructure to develop metrics, systems, and studies that define what “intelligence” means in the next generation of AI. This role is ideal for someone who combines scientific rigor with technical execution, and who’s deeply curious about how people use — and want to use — intelligent systems. Your work will shape how Cartesia builds and evaluates frontier models, ensuring that progress isn’t measured solely by static benchmarks, but by deeper qualities like understanding, naturalness, and adaptability in real-world interaction.
Responsibilities
- Identify and define key model capabilities and behaviors for next-generation model evaluations.
- Develop and implement new evaluation pipelines with robust statistical analysis and clear reporting.
- Partner closely with model training and research teams to embed evaluation systems into model development loops.
- Prototype new user studies and behavioral experiments to ground evaluations in real-world use.
Requirements
- Experience designing or implementing evaluation frameworks for generative models (audio, text, or multimodal).
- Strong technical and analytical skills, with the ability to translate open-ended research ideas into production-ready systems.
- Creativity in defining novel quantitative metrics for subjective or behavioral qualities.
- Excitement for building evaluation systems that bridge research and real-world use.
- Curiosity and rigor, motivated by discovering how to measure meaningful progress in intelligent behavior.
Benefits
- Competitive base salary
- Attractive equity package
- Fully covered medical insurance along with dental and vision for you and your family
- 9 weeks paternity leave
- 12 weeks maternity leave
- 401(k)
- Monthly commuter allowance
- Flexible PTO
- Provided meals and snacks daily