Senior Distributed Systems Engineer
$200k - $400k • Sunnyvale, CA
Posted 7mo ago
Job Location
Sunnyvale, CA
Tech Stack
Remote Work Policy
On-site
Categories
Machine Learning Engineer
About the job
The Institute of Foundation Models (IFM) is seeking a deeply technical Senior Distributed Systems Engineer to co-design and optimize the communication stack for large-scale distributed training, including hybrid parallelism and Mixture-of-Experts (MoE) workloads. This systems-level engineering position focuses on performance engineering, distributed debugging, and communication-runtime co-design, aiming to improve performance, fault tolerance, and scalability across model architecture, communication systems, runtime, and hardware topology for ultra-scale GPU supercomputing systems.
Responsibilities
- Design and optimize expert-parallel and hybrid-parallel communication patterns
- Drive high-performance hierarchical collectives for MoE workloads
- Co-design runtime orchestration with communication topology awareness
- Reduce tail latency and improve determinism across thousands of GPUs
- Architect fault-tolerant distributed execution under real-world cluster failures
- Optimize communication-compute overlap and topology-aware collective optimization
- Deeply debug NCCL, RDMA, and custom communication layers
- Implement hybrid expert parallel strategies in modern large-scale MoE systems
- Develop elastic and resilient distributed job orchestration concepts
- Analyze congestion and optimize routing across InfiniBand/RoCE fabrics
- Perform microbenchmarking and performance modeling for communication-heavy workloads
Requirements
- Experience optimizing distributed training at 1,000+ GPU scale (or equivalent depth)
- Hands-on expertise with RDMA, InfiniBand, RoCE, and GPUDirect RDMA
- Deep familiarity with NCCL and/or UCX internals
- Strong systems programming ability (C/C++, Rust, or Go)
- Strong familiarity with modern model training frameworks such as PyTorch
- Ability to troubleshoot and profile training performance issues related to communication bottlenecks
- Ability to translate research ideas into production-grade optimizations
- Experience debugging distributed hangs, desynchronization, and performance regressions
- Master’s, or Bachelor’s + 1 year of relevant experience