Tech Lead Manager, GPU Cluster Infrastructure
$225k - $325k • Remote • Remote (International) • Full-time
Posted 21d ago
About the job
FAR.AI is seeking a Tech Lead Manager for its GPU Cluster Infrastructure team. This role involves leading a sub-team focused on managing and scaling the organization's GPU compute platform, which supports cutting-edge AI safety research. You will be responsible for the technical direction and roadmap of the platform, ensuring its performance, fault tolerance, and security, especially as AI agents increasingly utilize the cluster for experiments. This position requires a blend of technical expertise and leadership, with at least half of your time dedicated to hands-on engineering work. You will also be responsible for hiring and growing the infrastructure team, setting priorities, and providing mentorship.
Responsibilities
- Set the platform's technical direction and own its roadmap, including system selection, scheduling, storage, and measurement.
- Own architecture and scheduling/storage design, and debug complex multi-layer failures.
- Hire and grow a small team of senior engineers, setting priorities, scoping projects, and providing feedback.
- Set security direction for a shared cluster used by AI agents, covering identity, access, isolation, and sandboxing.
- Define and participate in operational processes like on-call rotations, incident response, and postmortems.
- Act as an escalation point for research teams and providers, and implement platform fixes for recurring issues.
Requirements
- Experience leading engineers as a manager, tech lead, or project lead, including setting technical direction and providing feedback.
- 5+ years in systems or infrastructure engineering on production Linux, running GPU, HPC, or large-scale batch platforms.
- Experience owning systems from design through operation, with depth in scheduling, storage, networking, security, or GPU systems.
- Experience running production Kubernetes for GPU workloads with a batch layer (e.g., Slurm, Kueue), including quotas, priority, and preemption.
- Experience with infrastructure as code (e.g., Terraform, Ansible) and observability (e.g., Prometheus) for a production fleet.
- Proficiency in at least one infrastructure programming language (e.g., Python, Go, Rust, C++).
- Strong written communication skills for technical documentation and escalations.