Senior Research Scientist, Model Evaluation
Remote • Toronto • FullTime
Posted 9mo ago
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Cohere is seeking a Senior Research Scientist focused on Model Evaluation to join their team. This role is crucial for advancing AI capabilities by developing next-generation evaluation methods and infrastructure to accurately measure LLM progress. You will be responsible for creating new evaluation benchmarks, translating model feedback into reliable evaluations, and conducting research to push the state-of-the-art in LLM evaluation techniques. The ideal candidate is passionate about rigorously measuring AI capabilities and ensuring these measurements align with desired outcomes.
Responsibilities
- Create new evaluation benchmarks to test LLM capabilities.
- Develop trustworthy and repeatable evaluation methods by collaborating with cross-functional teams.
- Advance LLM evaluation methods through research, including training LLM judges and improving evaluation efficiency.
- Build scalable tools for analyzing model performance.
Requirements
- Experience in rapidly prototyping to demonstrate LLM capabilities and developing methods to measure them.
- Experience reviewing complex data and LLM outputs for quality assurance.
- Obsession with rigorously measuring AI capabilities and ensuring measurements align with desired outcomes.
- Strong software engineering skills.
Benefits
- Weekly lunch stipend of $75/£75 or local currency equivalent.
- Full health and dental benefits, including a mental health budget.
- Retirement savings plans (RRSP matching, 401K, Pension Scheme).
- 100% parental leave top-up for up to 6 months.
- Annual enrichment benefits for arts, culture, fitness, wellness, quality time, and workspace improvement.
- Education and learning stipend for conferences, courses, and coaching.
- 6 weeks (30 working days) of paid vacation.
- Budget for travel to other offices if remote.
- Annual company offsite.
- Co-working benefit for remote employees not near an office.
- $500 home office stipend.