(Senior/Junior AI Inference Reliability Engineer, Compute Infra
$160k - $280k • Remote • San Francisco
Posted 2d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
We are seeking a reliability-minded software engineer to manage the production operations of our Kubernetes-native control plane, which provisions and runs our GPU inference fleet. This platform team develops the manifest-driven API and controllers that enable the inference team to define their needs, such as cluster configurations, model deployments, and capacity adjustments, and ensures these are reconciled. Your role will be to maintain the reliability of this system in production, addressing issues like InfiniBand problems during node communication, diagnosing faults in hero-model and LoRA serving that generic health checks miss, and enhancing automated self-healing to permanently resolve recurring failure modes. The primary goal is to ensure the decoupling remains effective even during failures and to maintain fleet efficiency without introducing new instabilities. You will be responsible for inference serving SLOs and error budgets, leading incident response, and implementing systemic fixes based on postmortem analyses. Additionally, you will contribute to the reliability aspects of capacity efficiency, recognizing that misconfigured defragmentation or bin-packing can pose production risks. We value candidates with hands-on production reliability experience in large-scale distributed or GPU-serving systems, proficiency in troubleshooting InfiniBand/RDMA and Kubernetes controller/CRD-based platforms, and a software engineering approach to GPU infrastructure and its supporting tools.
Responsibilities
- Manage the production provisioning state machine, including host lifecycle, discovery, inference bring-up, GPU driver/CUDA validation, health checks, and decommissioning.
- Triage and extend automated self-healing for new failure modes, including InfiniBand issues, GPU Xid variants, and model-specific faults.
- Own end-to-end reliability of the provisioning and fleet health pipeline, including SLOs, error budgets, and incident response.
- Collaborate with the inference/ML platform team to incorporate real-world failure modes into platform abstractions.
- Partner with researchers, engineers, and product managers to deploy new technologies into production.
- Introduce new techniques and architecture to improve inference stack performance, latency, throughput, and efficiency.
- Develop tools for visibility into bottlenecks and instability sources, and implement solutions.
- Lead cross-functional reliability efforts, building consensus and shaping decisions around shared priorities.
Requirements
- Experience operating a production provisioning or host-lifecycle pipeline, troubleshooting state transitions at fleet scale, and improving state machine design based on failure patterns.
- Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus.
- Hands-on experience troubleshooting InfiniBand/RDMA networking issues in multi-node or disaggregated inference contexts.
- Familiarity with operational aspects of multi-tenant model serving (e.g., LoRA/adapter-based serving) and differentiated SLOs.
- Demonstrated track record of eliminating operational toil through automation and self-built tooling.
- Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
- Comfort with event-driven architectures, backpressure/circuit-breaking design, and integrated observability.
- A software-engineering mindset applied to GPU infrastructure, with experience writing and shipping production code.