Staff Engineer, Inference Optimizations
$191k - $239k • Seattle • Full-time
Posted 4mo ago
This job has been open for more than 60 days. It may no longer be actively hiring, so check with the company before putting a lot into an application.
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
DigitalOcean is seeking a Staff Engineer to play a key technical role in their AI Inference Optimization team. The goal is to establish DigitalOcean as the Inference Cloud of choice for digitally native companies by ensuring industry-leading performance for inference services. This role involves making architectural decisions to maximize throughput and minimize latency for advanced large models. As an individual contributor leader, you will act as a force multiplier, solving complex bottlenecks in memory bandwidth and compute utilization, and guiding the technical roadmap for the high-performance inference fleet.
Responsibilities
- Lead the technical strategy for benchmarking and performance optimizations at the inference engine and GPU kernel layers.
- Engineer solutions for complex performance issues, including attention layer optimizations, memory and precision management, and advanced parallelization across multi-node GPU clusters.
- Implement cutting-edge optimization techniques for Gen AI models, such as improving batch size performance using AMD's AITER library, identifying kernel fusion opportunities for Transformer blocks, and tuning expert gateway router kernels for MoE models.
- Act as a subject matter expert on modern GPU architectures (NVIDIA/AMD) and their software stacks (CUDA, ROCm, TensorRT, OpenAI Triton), advising on hardware procurement and software integration.
- Develop and deploy state-of-the-art quantization techniques (FP8, INT8, FP4) to improve throughput without sacrificing accuracy.
- Elevate the technical bar for the team through high-quality code and design reviews.
- Collaborate with Product Management and TPMs to translate hardware capabilities into shippable product features.
- Contribute to and integrate open-source AI projects within the GPU infrastructure and model performance optimization communities.
Requirements
- 5+ years of experience in high-performance computing or AI infrastructure, with a proven track record of solving compute utilization and memory bandwidth bottlenecks.
- Deep familiarity with the Gen AI landscape (LLM, VLM, LMM) and the architectural requirements of major model families.
- Hands-on experience with attention-layer optimizations and parallelization strategies across distributed GPU environments.
- Comprehensive understanding of NVIDIA and AMD GPU architectures and their respective software ecosystems (CUDA, ROCm, etc.).
- Extensive experience integrating, building with, and contributing to open-source software projects.
- Excellent system design skills, particularly related to low-level GPU programming, optimization, memory access patterns, and parallel execution.
- Experience acting as a technical lead, driving design and delivery through cross-functional alignment.
- Deep understanding of GPU architectures (SMs, Warp scheduling, Tensor Cores).
- Expert-level experience with Triton or CUDA, including contributions to the Triton compiler or writing custom CUDA kernels for LLMs.
Benefits
- Flexible time off
- Local team meetups
- Mental health support
- Reimbursement for relevant conferences, training, and education