Evaluation Engineer, Applied AI
Paris • Full-time
Posted 1mo ago
Job Location
Paris
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
Mistral is a full-stack AI solutions provider, offering everything from frontier models to developer tools and applications. We partner with enterprises across high-stakes industries like finance, manufacturing, defense, and healthcare to co-create customized AI systems that they can operate on their own terms. We are a dynamic, collaborative team passionate about AI's transformative potential, with a diverse workforce that thrives in competitive environments and is committed to driving innovation. Our teams are distributed globally, and we foster a creative, low-ego, and team-spirited culture.
Responsibilities
- Lead discovery sessions with customers, industry experts, and internal teams to identify and prioritize valuable use cases.
- Design and develop new benchmarks for industry use cases and frontier AI capabilities, including defining scoring mechanisms.
- Build and execute evaluations where AI agents operate within real-world software environments (e.g., CAD, simulation tools), ensuring reliable and consistent results over time.
- Proactively identify and address flaws or bugs in existing evaluation frameworks.
- Source and curate data, including working with expert vendors and developing realistic proxy or synthetic data when customer data cannot be shared.
- Document and present findings to senior leadership to inform strategic decisions.
- Develop and share tools and guides to empower other teams in building their own evaluations.
Requirements
- 2+ years of industry experience (software, AI, consulting) or a PhD.
- Software engineering or research background with experience in writing maintainable code.
- Proven experience working directly with customers, industry experts, or cross-functional teams.
- Experience building solutions or projects involving Large Language Models (LLMs).
- Proficiency in handling and cleaning large datasets.
- Strong Python programming skills.
- Experience working within a large, shared codebase and navigating code reviews.
- Solid understanding of LLM training, inference, and evaluation principles.
- Excellent communication and presentation skills, capable of engaging both technical and non-technical audiences.
- High energy, proactive, and independent work style with the ability to manage uncertainty.
- Strong decision-making skills with the ability to articulate rationale.
- Collaborative, low-ego, and eager to learn new fields quickly.
Benefits
- Comprehensive benefits package supporting well-being, growth, and work-life balance.
- Healthcare coverage
- Parental leave
- Retirement plans
- Relocation support
- Wellness programs
- Meal and transportation allowances
- Location-specific perks
