New: upload your resume and get matched to AI jobs. Free to start →

Evaluation Engineer, Applied AI

Paris • Full-time

Posted 1mo ago

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Engineer

About the job

Mistral is a full-stack AI solutions provider, offering everything from frontier models to developer tools and applications. We partner with enterprises across high-stakes industries like finance, manufacturing, defense, and healthcare to co-create customized AI systems that they can operate on their own terms. We are a dynamic, collaborative team passionate about AI's transformative potential, with a diverse workforce that thrives in competitive environments and is committed to driving innovation. Our teams are distributed globally, and we foster a creative, low-ego, and team-spirited culture.

Responsibilities

  • Lead discovery sessions with customers, industry experts, and internal teams to identify and prioritize valuable use cases.
  • Design and develop new benchmarks for industry use cases and frontier AI capabilities, including defining scoring mechanisms.
  • Build and execute evaluations where AI agents operate within real-world software environments (e.g., CAD, simulation tools), ensuring reliable and consistent results over time.
  • Proactively identify and address flaws or bugs in existing evaluation frameworks.
  • Source and curate data, including working with expert vendors and developing realistic proxy or synthetic data when customer data cannot be shared.
  • Document and present findings to senior leadership to inform strategic decisions.
  • Develop and share tools and guides to empower other teams in building their own evaluations.

Requirements

  • 2+ years of industry experience (software, AI, consulting) or a PhD.
  • Software engineering or research background with experience in writing maintainable code.
  • Proven experience working directly with customers, industry experts, or cross-functional teams.
  • Experience building solutions or projects involving Large Language Models (LLMs).
  • Proficiency in handling and cleaning large datasets.
  • Strong Python programming skills.
  • Experience working within a large, shared codebase and navigating code reviews.
  • Solid understanding of LLM training, inference, and evaluation principles.
  • Excellent communication and presentation skills, capable of engaging both technical and non-technical audiences.
  • High energy, proactive, and independent work style with the ability to manage uncertainty.
  • Strong decision-making skills with the ability to articulate rationale.
  • Collaborative, low-ego, and eager to learn new fields quickly.

Benefits

  • Comprehensive benefits package supporting well-being, growth, and work-life balance.
  • Healthcare coverage
  • Parental leave
  • Retirement plans
  • Relocation support
  • Wellness programs
  • Meal and transportation allowances
  • Location-specific perks

About Mistral AI