Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)
$250k - $485k • Remote • San Francisco • FullTime
Posted 1d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Perplexity is seeking a Member of Technical Staff to join their GPU Cluster Infrastructure team. This role will own the underlying GPU infrastructure that powers Perplexity's AI inference requests, which serve hundreds of millions of queries monthly. The goal is to abstract away the complexity of GPU provisioning, cluster configuration, and multi-cloud management into a unified, self-serve platform for both training and inference workloads. You will be responsible for the entire lifecycle of the GPU fleet, ensuring reliability, efficient capacity utilization, and seamless operation across different cloud providers.
Responsibilities
- Build and own a self-serve compute platform for launching training jobs and operating inference services.
- Manage GPU fleet provisioning, lifecycle, reliability, and capacity integration across cloud providers.
- Develop scheduling and placement logic to optimize GPU capacity utilization across providers.
- Support both long-running distributed training jobs and high-availability production inference services on the same fleet.
- Own Kubernetes for GPU orchestration, including writing operators and CRDs for multi-cluster management.
- Implement fault tolerance, autoscaling, and observability to ensure fleet utilization and workload resilience.
- Set technical direction for the platform architecture and roadmap in collaboration with other engineering teams.
Requirements
- Deep Kubernetes experience, including custom operators, CRDs, and multi-cluster federation.
- Experience managing GPU clusters at scale, including NVIDIA hardware, CUDA, and high-speed networking (InfiniBand or RoCE).
- Experience orchestrating compute across multiple cloud providers (e.g., CoreWeave, AWS, GCP).
- Strong distributed systems fundamentals, including scheduling, resource allocation, and fault tolerance.
- Proficiency in infrastructure and systems-level programming languages like Go, Rust, or C++.
- Experience supporting both long-running training jobs and high-availability inference services.
- Proven ability to own problems end-to-end and navigate ambiguity.