Principal Architect, AI Inference Networking - NIXL and Dynamo
Salary not listed • China, Shanghai • Full-time
Posted 1mo ago
About the job
Inference has become a networking challenge, especially with disaggregated serving architectures where different stages of computation (prefill, decode, KV cache streaming) occur on separate GPU sets. Performance hinges on efficient tensor movement between GPUs, nodes, and storage. These elastic deployments require dynamic APIs for managing worker joins/leaves, connection management, and runtime rerouting without pipeline stalls. NIXL (NVIDIA's Inference Xfer Library) addresses these needs, underpinning NVIDIA Dynamo and various inference engines. This role involves working directly with platform teams building demanding inference deployments in China, identifying and resolving scaling issues, and contributing to upstream code and roadmaps. You will collaborate daily with core NIXL and Dynamo engineers across different regions.
Responsibilities
- Drive NIXL adoption in China, from initial architecture discussions to production rollout and upstream contributions.
- Profile customer inference stacks to identify and resolve performance bottlenecks in throughput and time-to-first-token (TTFT), focusing on KV cache movement, memory registration, and transfer scheduling.
- Develop NIXL backends and plugins, enhance Dynamo integrations, and contribute optimizations to NIXL and GPUDirect data paths.
- Define dynamic data path APIs to support elastic scaling, worker churn, and runtime rerouting in production inference clusters.
- Influence NVIDIA's roadmap by representing regional requirements with strong technical authority.
- Foster the local NIXL community through presentations, benchmarks, and reference architectures.
Requirements
- M.Sc. or Ph.D. in CS/EE/CE, or equivalent industry experience.
- 12+ years of experience building or optimizing large-scale distributed systems, including AI inference/training infrastructure, communication libraries, high-performance networking, or HPC runtimes.
- Strong systems programming skills in C++ and Python, with a track record of shipping performance-critical code.
- Hands-on experience with RDMA networking (InfiniBand or RoCE) and GPU memory movement.
- Working knowledge of modern LLM serving, including disaggregated prefill/decode, KV cache management, tensor/pipeline parallelism, and at least one serving stack (e.g., Dynamo, TensorRT-LLM, vLLM, SGLang, Triton).
- Excellent communication skills with the technical credibility to engage effectively with customer engineering teams and core library maintainers.
- Fluent in Mandarin and professional English, with comfort working across China, Israel, and US time zones.