HPC Infrastructure Engineer - GPU Clusters
Remote • United States • FullTime
Posted 1d ago
About the job
ElevenLabs is seeking an engineer to join their research infrastructure team to ensure their NVIDIA GPU clusters are fast, reliable, and performant. This is a builder-operator role with significant autonomy, where your impact will be directly measured by training throughput and researcher velocity. You will be responsible for the entire stack beneath the training code, from OS images and NVIDIA drivers to high-speed networking and job scheduling. The role involves both hands-on hardware work and writing automation to eliminate manual tasks, with a focus on maintaining cluster security and performance.
Responsibilities
- Operate and improve the GPU fleet, including provisioning, scheduling, monitoring, upgrades, and capacity planning.
- Build automation for fleet health, such as node health checks and automated remediation.
- Manage the stack supporting training code, including OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, and high-speed networking.
- Optimize job scheduling (e.g., Slurm) for fair and fast researcher access to compute.
- Build and maintain high-performance storage for datasets and checkpoints.
- Troubleshoot and resolve performance issues within the cluster.
- Evaluate and benchmark rented GPU capacity from providers.
- Perform hands-on hardware tasks like racking, cabling, and diagnostics.
- Ensure cluster security through access control, network isolation, and secrets management.
Requirements
- Experience running large-scale Linux server or GPU environments in production.
- Proficiency with the NVIDIA stack (drivers, CUDA, NCCL, DCGM) or strong systems experience with a fast learning ability for hardware stacks.
- Comfort with bare-metal environments, server hardware, and high-speed networking.
- Ability to write solid automation in Python and/or Bash, using IaC tools like Ansible or Terraform.
- Skill in analyzing metrics and logs (e.g., PromQL) to identify and resolve issues.
- Desire to own tasks end-to-end and be a reliable point of contact.
- Willingness to perform any task required, including datacenter work.
Benefits
- Innovative culture with opportunities to define the trajectory of AI.
- Dynamic team environment with opportunities for impact beyond immediate responsibilities.
- Annual discretionary stipend for professional development.
- Annual discretionary stipend for social travel to meet colleagues.
- Annual company offsite in various global locations.
- Monthly co-working stipend for remote employees not near a main hub.