Infrastructure Engineer (GPU & Compute)
$180k - $220k • London, England, United Kingdom; New York, New York, United States; Remote; San Francisco, California, United States; Seattle, Washington, United States • Full-time
Posted 6mo ago
This job has been open for more than 60 days. It may no longer be actively hiring, so check with the company before putting a lot into an application.
About the job
Lightning AI is seeking an experienced Senior GPU & Compute Infrastructure Engineer to join their Infrastructure Engineering team. This role focuses on bringing up, validating, and operating large-scale bare-metal compute infrastructure, with a strong emphasis on GPU-enabled systems. You will work at the intersection of hardware, systems, and software, owning system diagnostics, validation, automation development, reliability improvements, and ensuring clusters are ready for demanding AI/ML and HPC workloads. The position involves GPU and system-level qualification, running validation environments and test clusters, and enhancing tooling and workflows for infrastructure bring-up at scale, including managing image pipelines and provisioning systems for consistency, performance, and reliability.
Responsibilities
- Own and evolve systems for image management, deployment, and validation across bare-metal infrastructure.
- Run and maintain test clusters for system validation, diagnostics, and bring-up.
- Validate firmware, drivers, and OS images across compute and GPU-enabled systems.
- Support hardware qualification efforts for next-generation platforms.
- Own GPU diagnostics and validation workflows across large-scale infrastructure.
- Diagnose and resolve complex issues across GPUs, drivers, OS, and hardware layers.
- Analyze system and GPU performance using tools like NVIDIA DCGM.
- Identify failure patterns and drive improvements in system stability and validation coverage.
- Build and maintain automation for provisioning, validation, and system bring-up.
- Develop Python-based tools and workflows to improve efficiency and reduce manual operational overhead.
- Improve the reliability, repeatability, and scalability of image pipelines and validation systems.
- Manage and operate Linux-based systems in production and validation environments.
- Manage virtualization technology.
- Support bare-metal provisioning workflows, including PXE and image-based systems.
- Interface with hardware management systems (e.g., IPMI, Redfish) for monitoring and debugging.
- Partner with Infrastructure, Hardware, and Data Center teams on system bring-up and validation.
- Collaborate with platform and ML teams to ensure systems meet workload requirements.
- Contribute to best practices for provisioning, diagnostics, and lifecycle management of infrastructure.
Requirements
- 5+ years of experience in infrastructure engineering, systems engineering, or related roles.
- Strong Linux systems experience in production environments.
- Hands-on experience with GPU-enabled systems and tools such as NVIDIA DCGM.
- Familiarity with bare-metal provisioning and system bring-up workflows.
- Proficiency in Python or similar scripting/programming languages for automation.
- Ability to debug complex issues across hardware, OS, GPUs, and system software.
- Experience with high-performance interconnects (e.g., InfiniBand, NVLink).
- Experience with PXE boot environments, LiveCD systems, or image-based provisioning workflows.
- Experience with hardware management interfaces such as iDRAC, IPMI, or Redfish.
- Data center operations experience, including working with physical hardware.
- Experience supporting AI/ML or HPC workloads at scale.
- Experience with GPU validation frameworks or large-scale hardware qualification processes.