Senior Software Engineer, GPU Cluster Infrastructure
$150k - $275k • Remote • Remote (International) • Full-time
Posted 21d ago
About the job
FAR.AI is seeking a Senior Software Engineer to join their Foundations team, which is responsible for the institute's compute platform, research tools, and workflow automation. This role will focus on accelerating AI safety research by managing and scaling the GPU cluster fleet, which includes bare-metal and managed Kubernetes environments. The engineer will work directly with researchers and other engineers to ensure large-scale experiments are performant and fault-tolerant, contributing to the core infrastructure that enables cutting-edge AI safety advancements.
Responsibilities
- Operate the Kubernetes GPU fleet, managing node lifecycle, upgrades, driver rollouts, and capacity planning.
- Manage batch scheduling and multi-tenancy, including queues, quotas, priorities, and fair share across research teams.
- Design and manage storage solutions for the fleet, including shared filesystems, object storage tiers, quotas, and backups.
- Ensure fault tolerance for multi-node training runs by managing node health, debugging NCCL and fabric issues, and implementing checkpoint/restart patterns.
- Harden the platform by managing identity and access, network policy, secrets, workload isolation, and sandboxing for AI agents.
- Bring new capacity online by acceptance-testing providers, integrating new clusters with infrastructure as code, and ensuring SLA adherence.
- Collaborate with research teams to address infrastructure problems and develop platform fixes for recurring issues.
- Participate in the on-call rotation and contribute to runbooks and postmortems.
Requirements
- 3+ years in systems or infrastructure engineering on production Linux, running GPU, HPC, or large-scale batch platforms.
- Experience owning at least one system from design through operation.
- Experience running production Kubernetes for GPU workloads with a batch layer (e.g., Slurm, Kueue, Volcano), including quotas, priority, preemption, and node health.
- Experience with infrastructure as code (e.g., Terraform, Ansible) and observability (e.g., Prometheus) for a production fleet.
- Proficiency in at least one infrastructure programming language (e.g., Python, Go, Rust, C++).
- Ability to write clear technical documentation for engineers, researchers, and providers.