Staff Software Engineer, Inference
$188k - $275k • Sunnyvale, CA / Bellevue, WA
Posted 5d ago
Job Location
Sunnyvale, CA / Bellevue, WA
Tech Stack
Remote Work Policy
On-site
Categories
AI Infrastructure Engineer
About the job
CoreWeave is seeking a Staff Software Engineer to join the Inference Platform Team. This team builds and operates CoreWeave’s Kubernetes-native inference platform, designed for low-latency, high-throughput AI workloads at massive scale. As a technical leader, you will drive architecture, performance, and reliability across multiple services and teams. Your responsibilities will include optimizing inference performance (latency, throughput, and GPU utilization), improving system reliability, and working deeply in distributed systems and Kubernetes-based infrastructure, focusing on areas like scheduling, batching, and memory optimization. This role requires hands-on technical leadership and the ability to influence engineering direction across the organization.
Responsibilities
- Act as a technical leader driving architecture, performance, and reliability across multiple services and teams.
- Lead cross-team design initiatives.
- Optimize inference performance, including latency, throughput, and GPU utilization.
- Improve system reliability at scale.
- Work deeply in distributed systems and Kubernetes-based infrastructure, focusing on scheduling, batching, and memory optimization.
- Influence engineering direction across the organization.
Requirements
- 8–12+ years of experience building and operating large-scale distributed systems or cloud platforms.
- Proven experience leading cross-team technical initiatives impacting multiple services or organizations.
- Strong programming skills in Go, Python, or C++.
- Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design.
- Strong understanding of distributed systems, networking, and performance optimization.
- Experience designing and operating low-latency, high-throughput systems with strict P95/P99 latency requirements.
- Hands-on experience with inference systems, including batching or micro-batching strategies, caching, and memory optimization.
- Experience improving system performance using metrics-driven approaches (e.g., latency, throughput, utilization).
- Familiarity with mixed precision (BF16, FP8) and streaming inference workloads.