Applied AI, Evaluation Engineer
Remote • Paris • Full-time
Posted 6mo ago
Job Location
Paris
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
Applied AI Engineer
About the job
Mistral AI is seeking an Evaluation Engineer to join our customer-facing Applied AI team. This role is crucial for ensuring our AI solutions are production-ready by designing methodologies, building infrastructure, and defining evaluation standards across various industries and use cases. You will bridge the gap between research and customer needs, creating evaluation frameworks that measure LLM performance in real-world scenarios, moving beyond standard benchmarks to address domain-specific risks and requirements. This position offers a unique opportunity to impact the deployment of cutting-edge AI by directly contributing to its measurable success for enterprise clients.
Responsibilities
- Design and implement comprehensive evaluation frameworks to measure LLM capabilities across diverse customer use cases.
- Build scalable evaluation infrastructure and pipelines for rapid, reproducible assessment of model performance.
- Develop novel evaluation methodologies for emerging capabilities or verticalized use cases.
- Create custom evaluation suites tailored to enterprise customers' specific needs.
- Collaborate with research teams to translate evaluation insights into model improvements.
- Partner with product teams to continuously improve evaluation tooling based on customer feedback.
Requirements
- 3+ years of experience in ML evaluation or benchmarking for LLM or agentic systems.
- Proven experience in AI or machine learning product implementation with APIs and back-end systems.
- Deep understanding of machine learning and LLM concepts and algorithms.
- Strong technical coding skills in Python.
- Strong communication skills with the ability to explain complex technical concepts to both technical and non-technical audiences.
Benefits
- 25 days of holidays and 8-10 days of RTT days.
- Autonomy on working hours.
- Full health insurance coverage for you and your family.
- €600 annual mobility allowance covering 50% of public transportation costs and encouraging eco-friendly travel.
- Swile meal vouchers with 60% company contribution.
- Gymlib sponsorship for gym membership.
- 4 additional weeks of parental leave on top of French state offerings.