Member of Technical Staff, North Modelling (Evals)
Remote • London • FullTime
Posted 19h ago
Job Location
London
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Cohere is seeking a Member of Technical Staff for the North Modelling (Evals) team. This role focuses on building evaluation systems, feedback loops, and applied modeling workflows to ensure that advancements in AI models translate into tangible product improvements for enterprise users of Cohere's North AI workspace platform. You will be the bridge between product needs and model development, defining what 'good' means for AI agents and enterprise workflows, and using this definition to guide model selection and updates. This is a hands-on applied Machine Learning Engineering role for individuals passionate about measurement, model behavior, and real-world product quality, rather than a pure research or conventional product engineering position.
Responsibilities
- Define the evaluation strategy for the North platform, covering agent workflows, tool use, and enterprise knowledge tasks.
- Develop high-quality evaluations based on product realities, including user feedback, production failures, and usage logs.
- Create systems to continuously translate user and product insights into evaluations, ensuring measurement keeps pace with product development.
- Act as the voice of North within the modeling teams, translating eval results and product context into actionable recommendations for model improvements.
- Extract clear insights from evaluation results and customer data to inform model failures and North-specific needs.
Requirements
- Experience improving LLM-powered, agent-powered, or AI-product systems through evaluations, feedback loops, data curation, prompting, model adaptation, or model selection.
- Deep understanding of evaluation as a craft, including representative tasks, precise rubrics, clean data, failure analysis, and regression tracking.
- Strong applied MLE judgment with clear reasoning about model behavior, eval validity, product outcomes, and production tradeoffs.
- Comfort working closely with users and product teams, translating qualitative signals into actionable measurements for modeling teams.
- Self-directed, practical, and motivated by open-ended problems requiring technical depth and product understanding.
- Commitment to making AI systems genuinely useful in real enterprise settings, beyond abstract benchmarks.
Benefits
- Weekly lunch stipend of $75/£75 or equivalent.
- Full health and dental benefits, including a mental health budget.
- RRSP matching, 401K, Pension Scheme.
- 100% Parental Leave top-up for up to 6 months.
- Annual enrichment benefits (Arts & culture, fitness/wellness, quality time, workspace improvement).
- Education & learning stipend for conferences, courses, and coaching.
- 6 weeks (30 working days) of paid vacation.
- Budget for traveling to other offices if remote.
- Annual company offsite.
- Co-working benefit for those not near an office.
- $500 home office stipend.