Software Engineer, GPU Inference
Sunnyvale, CA • FullTime
Posted 10mo ago
About the job
Cerebras is developing a new generation of disaggregated AI inference systems that combine GPU-accelerated prefill with ultra-fast decode on the Cerebras Wafer-Scale Engine. We are seeking a Software Engineer to productionize and optimize our GPU serving stack. This role involves working across custom inference APIs, the vLLM serving runtime, the AMD ROCm software stack, and rack-scale AMD GPU infrastructure to ensure this new serving path is reliable, numerically correct, observable, and exceptionally performant. You will write production code, establish operational practices for a new accelerator fleet, and drive improvements in time to first token, throughput, tail latency, and capacity efficiency. This is a hands-on role requiring deep debugging and optimization across application, runtime, distributed systems, and hardware layers.
Responsibilities
- Productionize the GPU inference stack, including API services, model-serving workers, vLLM, PyTorch, ROCm, GPU nodes, networking, and rack-scale infrastructure.
- Establish operational readiness for the AMD GPU fleet, covering deployment, upgrades, rollbacks, health-checking, capacity management, and failure recovery.
- Drive reliability in production by defining service-level indicators and objectives, and improving fault isolation, graceful degradation, automated recovery, and incident response.
- Improve inference performance by profiling and optimizing time to first token, throughput, tokens per second per GPU, tail latency, GPU utilization, memory efficiency, and rack-level capacity.
- Optimize model-serving behavior through tuning scheduling, continuous batching, prefix caching, KV-cache management, parallelism, request admission, quantization, graph execution, and distributed communication.
- Debug complex failures and performance regressions across application code, vLLM, PyTorch, ROCm/HIP, collective communication libraries, kernels, drivers, firmware, networking, and hardware.
- Ensure numerical correctness by building validation and regression infrastructure for model quality, numerical accuracy, precision changes, quantization, determinism, and compatibility.
- Build performance and correctness infrastructure, including benchmarks, workload replay tools, profiling automation, release qualification, dashboards, and regression gates.
Requirements
- 5+ years of software engineering experience with individual-contributor ownership of complex production systems.
- Experience building, operating, or optimizing production inference systems for LLMs, multimodal models, or similar GPU workloads.
- Strong programming ability in C++ and Python, including experience with multithreading, concurrency, memory management, and performance-sensitive software.
- Hands-on experience with a high-performance model-serving framework like vLLM, SGLang, TensorRT-LLM, or Triton Inference Server.
- Strong understanding of GPU execution and performance, including asynchronous execution, memory movement, synchronization, kernel launches, communication overhead, and profiling.
- Experience debugging distributed systems across multiple layers.
- Experience with Linux, containers, Kubernetes or comparable orchestration systems, observability, CI/CD, and operating latency-sensitive services in production.
- Ability to design rigorous benchmarks, interpret performance results, identify bottlenecks, and translate findings into improvements.
- Strong communication and technical leadership skills, with a demonstrated ability to drive cross-functional projects.
- Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or related discipline, or equivalent practical experience.