Principal Software Engineer – Large-Scale LLM Memory and Storage Systems
$272k - $431k • Remote • US, CA, Santa Clara
Posted 1d ago
About the job
NVIDIA Dynamo is a high-throughput, low-latency inference framework for serving generative AI and reasoning models across multi-node distributed environments. Built in Rust for performance and Python for extensibility, Dynamo orchestrates GPU shards, routes requests, and manages shared KV cache across heterogeneous clusters so that many accelerators feel like a single system at datacenter scale. As large language models rapidly outgrow the memory and compute budget of any single GPU, this platform enables efficient, resilient deployment of cutting-edge LLM workloads. We are seeking a Principal Systems Engineer to define the vision and roadmap for memory management of large-scale LLM and storage systems.
Responsibilities
- Design and evolve a unified memory layer spanning GPU memory, host memory, RDMA-accessible memory, SSD tiers, and remote storage for large-scale LLM inference.
- Architect and implement deep integrations with LLM serving engines (e.g., vLLM, SGLang, TensorRT-LLM), focusing on KV-cache offload, reuse, and remote sharing across heterogeneous clusters.
- Co-design interfaces and protocols for disaggregated prefill, peer-to-peer KV-cache sharing, and multi-tier KV-cache storage.
- Partner with GPU architecture, networking, and platform teams to leverage technologies like GPUDirect, RDMA, and NVLink for low-latency KV-cache access.
- Mentor engineers, set technical direction for memory and storage subsystems, and represent the team in internal and external forums.
Requirements
- Master's or PhD or equivalent experience.
- 15+ years of experience building large-scale distributed systems, high-performance storage, or ML systems infrastructure in C/C++ and Python.
- Deep understanding of memory hierarchies (GPU HBM, host DRAM, SSD, remote/object storage) and experience designing multi-tier systems.
- Experience with distributed caching or key-value systems optimized for low latency and high concurrency.
- Hands-on experience with networked I/O and RDMA/NVMe-oF/NVLink-style technologies.
- Familiarity with disaggregated and aggregated deployments for AI clusters.
- Strong skills in profiling and optimizing systems across CPU, GPU, memory, and network.
- Excellent communication skills and experience leading cross-functional efforts.
Benefits
- Highly competitive salaries
- Comprehensive benefits package
- Equity