Member of Technical Staff (AI Infrastructure Engineer)

About the job

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads.
• Manage and optimize Slurm-based HPC environments for distributed training of large language models.
• Develop robust APIs and orchestration systems for training pipelines and inference services.
• Implement resource scheduling and job management systems across heterogeneous compute environments.
• Benchmark system performance, diagnose bottlenecks, and implement improvements for training and inference infrastructure.
• Build monitoring, alerting, and observability solutions for ML workloads on Kubernetes and Slurm.
• Respond to system outages and collaborate to maintain high uptime for critical training runs and inference services.
• Optimize cluster utilization and implement autoscaling strategies for dynamic workload demands.

Requirements

• Strong expertise in Kubernetes administration, including custom resource definitions, operators, and cluster management.
• Hands-on experience with Slurm workload management, including job scheduling, resource allocation, and cluster optimization.
• Experience deploying and managing distributed training systems at scale.
• Deep understanding of container orchestration and distributed systems architecture.
• High-level familiarity with LLM architecture and training processes.
• Experience managing GPU clusters and optimizing compute resource utilization.
• Expert-level Kubernetes administration and YAML configuration management.
• Proficiency with Slurm job scheduling, resource management, and cluster configuration.
• Python and C++ programming with a focus on systems and infrastructure automation.
• Hands-on experience with ML frameworks like PyTorch in distributed training contexts.
• Strong understanding of networking, storage, and compute resource management for ML workloads.
• Experience developing APIs and managing distributed systems for batch and real-time workloads.
• Solid debugging and monitoring skills with expertise in observability tools for containerized environments.
• Demonstrated experience managing large-scale Kubernetes deployments in production.
• Proven track record with Slurm cluster administration and HPC workload management.
• Previous roles in SRE, DevOps, or Platform Engineering with a focus on ML infrastructure.
• Experience supporting both long-running training jobs and high-availability inference services.
• 3-5 years of relevant experience in ML systems deployment with a focus on cluster orchestration and resource management.