Senior AI Infrastructure Engineer, Physical Infrastructure
$166k - $220k • Costa Mesa, California, United States
Posted 12d ago
Job Location
Costa Mesa, California, United States
Tech Stack
Remote Work Policy
On-site
Categories
AI Infrastructure Engineer
About the job
Anduril Industries is seeking a Senior AI Infrastructure Engineer to lead the vision, execution, and long-term stability of how the company trains with GPUs at scale. This role involves taking ownership of cluster robustness, ensuring high-performance GPU systems are highly available, fault-tolerant, and resilient for ML platform and research teams. The primary focus is on logical stability and automated resilience, building self-healing mechanisms, tuning networking, and optimizing scheduling to ensure massive-scale training infrastructure runs seamlessly and scales efficiently.
Responsibilities
- Rack, stack, cable, and bring up GPU compute systems, including physical topology, power, cooling, firmware/BIOS, and burn-in validation.
- Build and tune the interconnect fabric (NVLink, InfiniBand, RoCE, Spectrum-X) for low-latency training and inference clusters.
- Integrate high-performance parallel storage to sustain throughput for distributed training and large datasets.
- Automate cluster deployment and configuration end-to-end using infrastructure as code.
- Operate and extend Kubernetes/Run:AI environments for GPU scheduling, quota management, and multi-tenant workload isolation.
- Own fleet health through monitoring, alerting, and rapid triage of hardware and network faults.
- Onboard engineers and researchers onto the platform and assist them in debugging and optimizing workloads.
- Partner with product teams to understand emerging compute needs and translate them into platform capabilities.
Requirements
- 10+ years in hands-on infrastructure, HPC, or datacenter engineering supporting GPU compute at scale.
- Hands-on experience with H200/B200/B300 (or comparable) GPU systems: bring-up, cabling, firmware/driver management.
- Experience with high-performance interconnects (NVLink, InfiniBand, RoCE, Spectrum-X) in large GPU clusters.
- Experience with high-performance parallel storage (VAST, DDN, Weka, Lustre, or similar).
- Proficiency in Kubernetes; Run:ai or similar GPU scheduling/orchestration experience strongly preferred.
- Strong automation background, building repeatable deployment pipelines.
- Ability to lift/move 50+ lbs and perform physical datacenter work.
- Eligible to obtain and maintain an active U.S. Top Secret clearance.
Benefits
- Highly competitive equity grants
- Top-tier benefits for full-time employees