Senior GPU Supercomputer Scheduler Engineer
$152k - $288k • Remote • US, CA, Santa Clara
Posted 1d ago
Job Location
US, CA, Santa Clara
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
NVIDIA's Managed AI Research Superclusters (MARS) team is seeking a Senior GPU Supercomputer Scheduler Engineer to design and implement groundbreaking GPU compute clusters. This role involves developing solutions for demanding deep learning, high-performance computing, and computationally intensive workloads. You will contribute to the design and implementation of new scheduling features and services to optimize GPU compute clusters for fairness, occupancy, resilience, performance, and power usage. This is an excellent opportunity to deliver production-grade solutions, work with cutting-edge technology, and collaborate with technical leaders on significant challenges in machine learning, cloud computing, and system co-design.
Responsibilities
- Design and develop new scheduling features and services to improve GPU compute clusters in areas like resource usage, fairness, occupancy, resilience, performance, and power usage.
- Design and develop batch workload management and orchestration services.
- Provide support to staff and end users for batch scheduler issues.
- Build and improve the ecosystem around GPU-accelerated computing.
- Perform performance analysis and optimizations for deep learning workflows.
- Develop large-scale automation solutions.
- Conduct root cause analysis and suggest corrective actions for problems.
- Proactively identify and resolve issues before they occur.
Requirements
- Bachelor’s degree in Computer Science, Electrical Engineering or related field, or equivalent experience.
- 5+ years of work experience.
- Strong understanding of batch scheduling, with experience in schedulers like SLURM or K8s batch schedulers (Kueue, Volcano).
- Significant experience in systems programming languages such as C/C++ & Go, and scripting languages like Python and bash.
- Established experience with the Linux operating system, environment, and tools.
- Experience analyzing and tuning performance for various AI workloads.
- In-depth understanding of container technologies like Docker, Singularity, Podman.
- Flexibility and adaptability to work in a dynamic environment with different frameworks and requirements.
- Excellent communication, interpersonal, and customer collaboration skills.
Benefits
- Equity
- Benefits