Staff Software Engineer, GPU Inference
Salary not listed • Toronto, CAN • Full-time
Posted 9d ago
About the job
Cerebras Systems is developing a new generation of disaggregated AI inference systems that combine GPU-accelerated prefill with ultra-fast decode on the Cerebras Wafer-Scale Engine. We are seeking a Software Engineer to productionize and optimize our GPU serving stack. This role involves working across custom inference APIs, the vLLM serving runtime, the AMD ROCm software stack, and rack-scale AMD GPU infrastructure to ensure reliability, numerical correctness, observability, and exceptional performance. You will write production code, establish operational practices for a new accelerator fleet, and drive improvements in key performance metrics like time to first token, throughput, tail latency, and capacity efficiency. This is a hands-on position requiring deep debugging and optimization across application, runtime, distributed systems, and hardware layers.
Responsibilities
- Productionize the GPU inference stack, including API services, model-serving workers, vLLM, PyTorch, ROCm, GPU nodes, networking, and rack-scale infrastructure.
- Establish operational readiness for the AMD GPU fleet, covering deployment, upgrades, health checks, capacity management, and failure recovery.
- Drive reliability in production by defining service-level indicators, improving fault isolation, and automating recovery processes.
- Optimize inference performance, focusing on time to first token, throughput, tail latency, GPU utilization, and memory efficiency.
- Tune model-serving behavior, including scheduling, batching, KV-cache management, parallelism, quantization, and distributed communication.
- Debug complex failures and performance regressions across application code, vLLM, PyTorch, ROCm/HIP, communication libraries, kernels, drivers, and hardware.
- Ensure numerical correctness through validation and regression infrastructure for model quality and accuracy.
- Build performance and correctness infrastructure, including benchmarks, workload replay tools, and dashboards.
Requirements
- 8+ years of software engineering experience with ownership of complex production systems.
- Experience building, operating, or optimizing production inference systems for LLMs or similar GPU workloads.
- Strong programming skills in C++ and Python, including multithreading, concurrency, and memory management.
- Hands-on experience with high-performance model-serving frameworks like vLLM, SGLang, TensorRT-LLM, or Triton.
- Strong understanding of GPU execution and performance, including asynchronous execution, memory movement, synchronization, and profiling.
- Experience debugging distributed systems across multiple layers.
- Experience with Linux, containers, Kubernetes, observability, and CI/CD.
- Ability to design rigorous benchmarks, interpret performance results, and identify bottlenecks.
- Strong communication and technical leadership skills.
- Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience.