Principal Engineer, Inference Memory and Storage Systems
$250k - $312k • Seattle • Full-time
Posted 23d ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
DigitalOcean is seeking a Principal Engineer to define the technical vision and roadmap for the memory and storage layer of their inference platform. This role is crucial for optimizing the serving stack of frontier open models on GPU fleets at production scale. The focus will be on creating a unified, multi-tier memory and storage substrate that spans GPU HBM, host DRAM, local NVMe, and remote object storage, accessible through a coherent interface to all serving engines and routers. The ideal candidate will drive unit economics, contribute to open-source projects, and set direction across teams.
Responsibilities
- Define and evolve a unified memory layer across GPU memory, host memory, RDMA-accessible memory, NVMe tiers, and remote object storage for LLM inference.
- Architect deep integrations with serving engines (vLLM, SGLang, TensorRT-LLM) focusing on KV cache offload, reuse, eviction, and cross-node sharing.
- Design interfaces and protocols for disaggregated prefill/decode, peer-to-peer KV cache transfer, and cache-aware routing.
- Own the eviction and admission strategy across memory tiers and the metrics to validate policy effectiveness.
- Partner with GPU infrastructure, networking, and platform teams to leverage technologies like GPUDirect, RDMA, NVMe-oF, and NVLink.
- Drive unit economics by modeling the impact of cache performance on cost and GPU utilization.
- Set technical direction, lead design reviews, and mentor senior engineers.
- Represent DigitalOcean externally through upstream contributions, conference talks, and technical deep dives.
Requirements
- 15+ years of experience building large-scale distributed systems, high-performance storage, or ML systems infrastructure.
- Deep fluency in memory hierarchies (GPU HBM, host DRAM, NVMe, remote/object storage) and experience designing cross-tier systems.
- Experience with distributed caching or key-value systems optimized for low latency and high concurrency.
- Hands-on experience with networked I/O and technologies like RDMA, NVMe-oF, NVLink.
- Strong systems programming skills in C/C++, Go, Rust, or Python.
- Proficiency in profiling and optimization across CPU, GPU, memory, and network.
- Excellent written and verbal communication skills.
- Proven ability to lead cross-functional efforts with product, infrastructure, and customer-facing teams.
Benefits
- Annual bonus plan
- New hire equity grant
- Annual bonus plan
- New hire equity grant