Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering
Salary not listed • Remote • India • Full-time
Posted 2mo ago
This job has been open for more than 60 days. It may no longer be actively hiring, so check with the company before putting a lot into an application.
Job Location
India
Also open in Amsterdam
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
We are seeking a software engineer to develop a Kubernetes-native control plane for managing our GPU inference fleet. This role involves designing a manifest-driven API that allows the inference team to declare their needs for clusters, model deployments, or capacity changes. Our controllers will handle the underlying reconciliation, provider/runtime selection, and lifecycle management, abstracting away the complexities of the serving stack, scheduler, and hardware. You will also build systems to optimize fleet efficiency, including defragmentation, rebalancing logic for consolidating workloads, and scheduling/bin-packing improvements to maximize GPU utilization without impacting latency. The primary goal is to decouple users of the platform from operational complexities while maximizing hardware capacity. This includes building controllers, reconciliation loops, and self-service APIs/CLIs, along with event-driven systems for health, remediation, and utilization, all designed to operate without human intervention. A product mindset is crucial, focusing on the developer experience of internal platforms and APIs, with a 'build it, own it' philosophy that includes production operation and support.
Responsibilities
- Build the provisioning state machine to model the full lifecycle of a physical host, from discovery to decommission, as explicit, versioned states and transitions.
- Develop the self-service API and control plane for inference teams to request, scale, and tear down inference clusters declaratively.
- Automate self-healing by detecting and replacing degraded or failed nodes, and reintroducing healthy capacity.
- Ensure the reliability of the provisioning system through idempotency, retries, rollback, and drift detection.
- Collaborate with the inference/ML platform team to understand and abstract their cluster requirements into the platform.
- Engineer infrastructure code with strong typing, automated tests, code reviews, versioning, and CI/CD.
Requirements
- Strong software engineering background in Go, Python, Rust, or similar.
- Experience with durable workflow orchestration tools like Temporal or Cadence.
- Experience building software control planes or orchestration systems with state reconciliation.
- Experience with event-driven systems using message queues or event streams (e.g., Kafka, NATS, SQS).
- A product mindset, with experience building internal platforms or APIs for other engineering teams.
- Exposure to bare-metal provisioning (PXE/iPXE, Redfish/IPMI, BMC) and/or networking fundamentals.
- Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE).
- Prior work at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization.
- Systems programming in Rust or Go.