Senior Machine Learning Engineer, LLM Inference Optimization
$195k - $262k • Palo Alto, California, United States
Posted 4d ago
Job Location
Palo Alto, California, United States
Tech Stack
Remote Work Policy
On-site
Categories
Machine Learning Engineer
About the job
Nebius is building a full-stack AI cloud platform to support developers and enterprises from data and model training through to production deployment. As a Senior Machine Learning Engineer on the Applied AI team, you will own model and endpoint optimization from model artifacts through production deployment. Your work will span model internals, quantization, model compression, speculative decoding, KV-cache optimization, inference engines, and serving architecture to improve latency, throughput, memory efficiency, GPU utilization, and cost per token while preserving model quality and reliability. You will collaborate with kernel and platform engineers to investigate serving problems, compare configurations, and resolve performance and quality regressions, validating improvements through reproducible benchmarks and safe production rollouts.
Responsibilities
- Own optimization projects for specific model families, customer endpoints, or serving backends.
- Compare inference engines and recommend practical serving configurations suited to individual workloads.
- Diagnose model-quality and performance regressions during production rollouts.
- Improve LLM and VLM endpoint latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.
- Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo.
- Develop production model-compression workflows covering quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.
- Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.
- Design and improve LLM request routers and scheduling policies.
- Scale dense and mixture-of-experts inference across multiple GPU nodes, selecting and tuning parallelism strategies.
- Build reproducible benchmark harnesses measuring TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.
- Collaborate with GPU kernel engineers and platform engineers to trace bottlenecks across various layers.
- Produce design documents, performance reports, rollout plans, and customer-facing technical explanations.
Requirements
- Strong engineering skills in Python and PyTorch.
- Hands-on experience deploying or optimizing LLM, VLM.