Staff Machine Learning Engineer, LLM Inference Optimization
Salary not listed • Amsterdam, Netherlands; Berlin, Germany; London, United Kingdom; Poland; Prague, Czech Republic; Zurich, Switzerland • Full-time
Posted 10d ago
About the job
Nebius is building a full-stack AI cloud platform for the global AI economy, supporting developers and enterprises from data and model training through to production deployment. This role focuses on production-grade inference for open-source, customer, and internally optimized models, owning the path from model artifact to a fast, reliable, and cost-efficient endpoint. The work involves model internals, inference engines, serving architecture, benchmarking, and production deployment.
Responsibilities
- Own a platform-level optimization pillar such as quantization, speculative decoding, KV-cache optimization, engine selection, or long-context serving.
- Create reusable evaluation, benchmarking, and rollout frameworks across multiple model families.
- Provide technical direction to MLEs, Applied Scientists, GPU kernel engineers, and infrastructure engineers on shared performance goals.
- Optimize LLM and VLM endpoints for latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.
- Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, or similar systems.
- Build and productionize model-compression workflows, including quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.
- Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.
- Build reproducible benchmark harnesses for TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.
- Partner with GPU kernel engineers and platform engineers to diagnose bottlenecks across model code, kernels, runtime, scheduler, gateway, and cluster layers.
- Write clear design docs, performance reports, rollout plans, and customer-facing technical explanations.
Requirements
- Strong Python and PyTorch engineering skills.
- Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference systems.
- Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, KServe, or equivalent internal systems.
- Strong understanding of transformer inference bottlenecks, including KV cache, attention, memory bandwidth, batching, parallelism, and long-context serving.
- Ability to reason quantitatively about latency, throughput, quality, utilization, and cost tradeoffs.
- Strong communication skills and ability to collaborate with research, kernel, infrastructure, product, and customer teams.
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams