Senior Software Engineer - AI Inference Performance
$184k - $357k • US, CA, Santa Clara
Posted 21d ago
About the job
NVIDIA is seeking a Senior Software Engineer to focus on AI Inference Performance, advancing innovative LLM and VLM inference on NVIDIA GPU-accelerated systems. This hands-on role involves pushing workloads to practical performance limits, delivering measurable gains in latency, throughput, efficiency, and scale. You will translate performance models and profiler data into working code, collaborating with various teams and contributing improvements to open-source inference engines. Your work will directly impact production deployments and help shape future NVIDIA platforms.
Responsibilities
- Lead end-to-end analysis of LLM/VLM inference processes, optimizing key performance metrics like time to first token and inter-token latency.
- Develop speed-of-light and roofline models to identify performance headroom and formulate optimization hypotheses.
- Profile workloads using NVIDIA tools and custom instrumentation to eliminate bottlenecks in host code, CUDA kernels, memory, and communication.
- Tune serving hyperparameters and techniques such as batching, KV-cache management, quantization, and model parallelism.
- Build and optimize performance-critical kernels for operations like attention, matrix multiplication, and routing using CUDA, CUTLASS, or Triton.
- Establish repeatable benchmarks and performance regression gates, balancing performance with accuracy.
- Contribute high-quality upgrades to open-source projects like TensorRT-LLM, vLLM, or SGLang.
Requirements
- Over 6 years of experience in LLM/VLM inference performance, covering models, serving, distributed runtimes, kernels, and hardware.
- Strong programming skills in Python, Rust, and/or C++, with hands-on CUDA or other GPU programming experience.
- Demonstrated expertise in performance analysis using tools like NVIDIA Nsight Systems and Nsight Compute.
- Deep understanding of GPU architecture, including Tensor Cores, memory hierarchy, and synchronization.
- Practical experience optimizing inference servers and model execution with techniques like batching, quantization, and parallelism.
- Understanding of distributed systems and networking for accelerated computing.
- BS or MS in Computer Science, Computer Engineering, or equivalent experience.
Benefits
- Competitive salaries
- Generous benefits package
- Equity