[Junior / Senior / Staff] Software Engineer, Inference / Compute Infrastructure Engineering
$160k - $280k • Remote • San Francisco
Posted 24d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
We are seeking a software engineer to develop a Kubernetes-native control plane for provisioning and managing our GPU inference fleet. This role involves designing a manifest-driven API that allows the inference team to declare their needs, such as clusters, model deployments, or capacity adjustments. Our controllers will handle the underlying reconciliation, provider/runtime selection, and lifecycle management, abstracting away the complexities of the serving stack, scheduler, or hardware pool. You will also build systems to ensure fleet efficiency, including defragmentation, rebalancing logic for consolidating workloads, and scheduling/bin-packing improvements to maximize GPU utilization without compromising latency. The core objective is to decouple platform users from operational complexities while optimizing hardware capacity. This includes developing controllers, reconciliation loops, and self-service interfaces (API/CLI) for seamless operation, alongside event-driven systems for health monitoring, remediation, and utilization management, all designed to operate without human intervention. A product-oriented mindset is crucial, with a focus on building internal platforms and APIs that enhance the developer experience for consuming engineering teams. This role operates under a "you build it, you own it" philosophy, encompassing both software delivery and production support.
Responsibilities
- Design and implement the provisioning state machine, modeling the full lifecycle of a physical host from discovery to decommission.
- Develop a self-service API enabling the inference team to request, scale, and tear down inference clusters declaratively.
- Automate self-healing processes for degraded or failed nodes, including safe draining, repair/replacement triggering, and capacity reintroduction.
- Ensure the reliability of the provisioning pipeline through idempotency, retries, rollback, and drift detection.
- Collaborate with the inference/ML platform team to understand and abstract necessary cluster shapes and constraints.
- Apply software engineering best practices to infrastructure code, including strong typing, automated testing, code review, versioning, and CI/CD.
Requirements
- Strong software engineering background in Go, Python, Rust, or similar languages.
- Experience with durable workflow orchestration tools like Temporal or Cadence.
- Experience building software control planes or orchestration systems with state modeling and reconciliation.
- Experience with event-driven systems using message queues, event streams, or pub/sub.
- A product mindset, with experience building internal platforms or APIs for other engineering teams.
- Exposure to bare-metal provisioning, networking fundamentals, or GPU/accelerator infrastructure (nice to have).
- Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE) (nice to have).
- Prior experience at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization (nice to have).
- Systems programming in Rust or Go (nice to have).
Benefits
- Competitive compensation
- Startup equity
- Health insurance
- Other competitive benefits