Research Engineer, Inference Foundation
Munich • Full-time
Posted 1mo ago
About the job
Mistral is seeking a Research Engineer for its Inference Foundation team, which is responsible for the core of Mistral's inference stack, including the inference engine and its orchestration. This hybrid role involves optimizing the inference stack at scale for throughput and latency, ensuring elastic and cost-effective capacity scaling, and powering the training of frontier models. You will work on feature development, bug fixes, release processes, and performance optimization across the serving fleet, with a focus on maintaining production-grade quality and efficiency.
Responsibilities
- Develop and fix the core inference engine and orchestrator for maximum performance at scale.
- Own and manage the release process for the serving stack, ensuring validated and regression-free releases.
- Drive improvements upstream to open-source inference engines when appropriate.
- Optimize serving efficiency across the fleet by reducing pod startup time and addressing cold-cache regressions.
- Optimize and maintain the serving topology for efficient computation and communication.
- Build and optimize inference serving infrastructure to support RL and post-training for frontier models.
- Optimize inference performance across a wide spectrum of workloads.
Requirements
- Experience building and running ML/LLM services at scale with clear latency and availability targets.
- Hands-on experience with inference engines like vLLM, SGLang, TensorRT-LLM, or similar.
- Solid understanding of inference internals including prefill vs. decode, KV-cache, batching, scheduling, speculative decoding, and parallelism strategies.
- Familiarity with distributed and disaggregated serving architectures.
- Proficiency in debugging across the full stack: CUDA/NCCL, kernels, containers, networking, storage.
- Experience with Python for systems tooling and backend services, and PyTorch.
- Experience with Kubernetes for running infrastructure at scale.
- Strong fundamentals in GPU and networking: CUDA runtime, NCCL, InfiniBand/RDMA.
Benefits
- Healthcare coverage
- Parental leave
- Retirement plans
- Relocation support
- Wellness programs
- Meal and transportation allowances
