Speculative decoding Jobs

Nebius is building a full-stack AI cloud platform to support developers and enterprises in the global AI economy, from data and model training through to production deployment. The Nebius Token Factory is seeking scientists to tackle frontier LLM and VLM inference bottlenecks by turning them into research questions, conducting rigorous experiments, and translating findings into production capabilities. You will own focused research and optimization projects, write strong code, and produce prototypes, publications, and technical artifacts for engineers to build upon. Your research will cover areas like quantization, distillation, speculative decoding, KV-cache optimization, and model/runtime co-optimization, alongside inference engines and distributed inference architectures. You will evaluate trade-offs across model quality, latency, throughput, memory footprint, and cost per token, and collaborate with engineering teams to bring promising results into production.

$195k - $262k

Inworld is a research lab and inference provider focused on real-time AI for consumer-facing applications. We build first-party speech models, serve LLMs, and run the inference behind modular APIs designed for high-volume, real-time workloads. Our models and infrastructure support consumer applications across companions, healthcare, fitness, education, media, and more, serving over 10 trillion LLM tokens per month. We are looking for strong individuals from varied backgrounds who learn fast, thrive in ambiguity, and can demonstrate their experience in building, breaking, and understanding complex systems.

$250k - $300k

Together AI

The Speculative Decoding team at Together AI focuses on developing draft models and algorithms to accelerate large language model generation without altering outputs. Our work spans model training, inference algorithms, and serving systems, aiming to reduce generation cost and latency. As a research intern, you will explore open questions in speculative decoding, including new drafter architectures, verification schemes, knowledge distillation, and adaptive drafters. You will train and fine-tune draft models at scale on GPU clusters and evaluate your ideas against realistic serving workloads, such as long-context and agentic traffic. Projects are designed to address key research questions in speculative decoding, leading to faster generation on frontier models, drafters that adapt to changing workloads, and robust evaluation under real traffic.

Baseten is seeking an Engineering Manager to lead a portion of their Inference Performance team. This team is responsible for accelerating and optimizing the performance of demanding AI workloads on GPUs. The role involves managing and growing a team of inference performance engineers focused on the inference engine and runtime, including areas like kernels, scheduling, batching, KV-cache management, and speculative decoding. It's a hands-on technical leadership position where you will define the team's direction, solve complex problems, and contribute technically to GPU performance while also focusing on hiring, developing, and supporting your team members. Your team's work will directly impact customer model inference speed and serving efficiency, and you will play a key role in shaping the team's structure as it scales.

$240k - $270k

Baseten powers mission-critical inference for leading AI companies, enabling them to bring cutting-edge models into production. We are seeking inference performance engineers to make demanding AI workloads run faster and more efficiently. You will work across the entire stack, from the inference engine and runtime through scheduling, serving, and routing, applying advanced techniques to optimize performance. This role is ideal for someone who thrives in a fast-paced startup environment and is eager to make significant contributions to LLM inference.

$180k - $360k

As a Senior Principal Machine Learning Engineer, you will be responsible for designing, developing, and optimizing machine learning models, with a focus on cutting-edge Large Language Models (LLMs), to run efficiently on SambaNova's specialized hardware architecture, including the RDU. This critical role bridges advanced LLM research and practical deployment, involving the development of model architectures, improving training and inference efficiency, and collaborating on hardware-software co-design with compiler, systems, and hardware teams. The engineer will also act as the ML expert, guiding the integration of LLM solutions into production systems and customer-facing products like SambaStack and SambaCloud, with work spanning the full lifecycle from training and inference to evaluation and data curation.

$220k - $300k