(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure
$250k - $320k • Remote • San Francisco
Posted 10h ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
We are seeking a software engineer with a strong focus on reliability to manage the production operations of our Kubernetes-native control plane, which provisions and runs our GPU inference fleet. This platform team develops the API and controllers that enable the inference team to define their needs, such as clusters, model deployments, or capacity changes, and ensures these are reconciled. Your role will be to maintain this functionality in the real world by troubleshooting issues like InfiniBand problems, diagnosing serving faults, and enhancing our automated self-healing systems to permanently resolve recurring failures. The primary goal is to ensure the decoupling remains effective even during disruptions and to maintain fleet efficiency without introducing new instabilities. You will be responsible for inference serving SLOs and error budgets, leading incident response, and implementing lessons learned from postmortems into system changes. Additionally, you will address the reliability aspects of capacity efficiency, recognizing that misconfigured defragmentation or bin-packing can pose production risks. We value candidates with hands-on production reliability experience in large-scale distributed or GPU-serving systems, proficiency in troubleshooting InfiniBand/RDMA and Kubernetes controller-based platforms, and a software engineering approach to GPU infrastructure.
Responsibilities
- Manage the production provisioning state machine, including host lifecycle, discovery, inference bring-up, GPU driver/CUDA validation, health checks, and decommissioning.
- Triage and extend automated self-healing systems to address new failure modes, including InfiniBand issues, GPU errors, and model-specific faults.
- Own the end-to-end reliability of the provisioning and fleet health pipeline, including SLOs, error budgets, and incident response.
- Collaborate with the inference/ML platform team to incorporate real-world failure modes and constraints into platform abstractions.
- Partner with researchers, engineers, and product managers to deploy new technologies and enable advanced research.
- Introduce new techniques, tools, and architectures to improve the performance, latency, throughput, and efficiency of the model inference stack.
- Develop tools for visibility into bottlenecks and instability sources, and implement solutions to address them.
- Lead cross-functional reliability efforts, building consensus, managing stakeholders, and influencing planning to prioritize reliability.
Requirements
- Experience operating a production provisioning or host-lifecycle pipeline, troubleshooting state transitions at scale, and improving state machine design based on failure patterns.
- Extensive experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is highly desirable.
- Hands-on experience troubleshooting InfiniBand/RDMA networking issues in multi-node or disaggregated inference environments.
- Familiarity with the operational aspects of multi-tenant model serving (e.g., LoRA) and managing differentiated SLOs.
- Proven ability to eliminate operational toil through automation and custom tooling.
- Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
- Comfort with event-driven architectures, backpressure/circuit-breaking design, and integrating observability into systems.
- A software engineering mindset, with experience writing and shipping production code and treating GPU infrastructure as software.