Senior Machine Learning Engineer, LLM Inference Optimization
Remote • Remote - Europe
Posted 4d ago
Job Location
Remote - Europe
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
Nebius is building a full-stack AI cloud platform to support developers and enterprises from data and model training through to production deployment. As a Senior Machine Learning Engineer on the Applied AI team, you will own model and endpoint optimization from model artifacts through production deployment. Your work will focus on improving latency, throughput, memory efficiency, GPU utilization, and cost per token while preserving quality and reliability.
You will collaborate with kernel and platform engineers to investigate serving problems, compare configurations, and resolve performance and quality regressions. Your improvements will be validated through reproducible benchmarks and safe production rollouts under real-world workloads.
Responsibilities
- Own optimization projects for specific model families, customer endpoints, or serving backends.
- Compare inference engines and recommend serving configurations suited to individual workloads.
- Diagnose model-quality and performance regressions during production rollouts.
- Improve LLM and VLM endpoint latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.
- Deploy, configure, benchmark, and extend engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo.
- Develop production model-compression workflows covering quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.
- Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.
- Design and improve LLM request routers and scheduling policies that balance worker utilization, queueing, request characteristics, and KV-cache locality while meeting latency and reliability targets.
- Scale dense and mixture-of-experts inference across multiple GPU nodes, selecting and tuning parallelism strategies.
- Build reproducible benchmark harnesses measuring key performance indicators like TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.
- Collaborate with GPU kernel and platform engineers to trace bottlenecks across various system layers.
- Produce design documents, performance reports, rollout plans, and customer-facing technical explanations.
Requirements
- Strong engineering skills in Python and PyTorch.
- Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference.
- Practical knowledge of at least one modern inference stack (e.g., vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, KServe).
- Strong understanding of transformer inference bottlenecks (KV cache, attention, memory bandwidth, batching, parallelism, long-context serving).
- Ability to quantify trade-offs among latency, throughput, quality, utilization, and cost.
- Experience designing or optimizing distributed inference system architecture, with hands-on work in areas like request routing, distributed scheduling, PDD, multi-node inference, or expert parallelism.
- Strong communication and collaboration skills across research, kernel, infrastructure, product, and customer teams.