Sr. Engineering Manager, Inference

About the job

Responsibilities

• Lead a team of senior and staff engineers building and operating CoreWeave's Kubernetes-native inference platform.
• Own the end-to-end production service, including availability, latency, capacity, and cost.
• Define and defend SLIs/SLOs and error budgets, and build the operational practice behind them.
• Manage multi-tenant platforms, focusing on isolation, quota, noisy-neighbor mitigation, and fair sharing.
• Grow and promote engineers, including into senior and staff-level roles.
• Manage underperformance directly and fairly, and retain top performers.
• Build and own a hiring bar and interview process for senior engineering roles.
• Operate low-latency, high-throughput request-serving systems against strict P95/P99 targets.
• Manage autoscaling, load balancing, admission control, and traffic management at scale.
• Perform capacity planning and demand forecasting for constrained or expensive resources.
• Manage GPU-backed workloads and their scheduling and utilization trade-offs.
• Implement progressive delivery practices like canarying, feature gating, and safe rollback for always-on services.
• Operate customer-facing cloud services or developer platforms, including API and control-plane design.
• Operate multi-region fleets at a cloud provider or hyperscaler.
• Scale teams, tooling, and processes in a high-growth environment.

Requirements

• 8+ years of experience building and operating large-scale distributed systems or cloud services.
• 3+ years of engineering management experience, including managing senior and staff-level engineers.
• Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design.
• Strong understanding of distributed systems, networked services, and performance under load.
• Experience with multi-tenant platforms: isolation, quota, noisy-neighbor mitigation, and fair sharing.
• Excellent communication and stakeholder management skills.
• Demonstrated track record of growing and promoting engineers.
• Experience running performance calibration and making accurate leveling decisions.
• Experience building and owning a hiring bar and interview process for senior engineering roles.
• Working knowledge of autoscaling, load balancing, admission control, and traffic management at scale.
• Familiarity with GPU-backed workloads and their scheduling and utilization trade-offs.
• Familiarity with inference-serving stacks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe.
• Familiarity with observability stacks (Prometheus, Grafana, OpenTelemetry).
• Ability to build frictionless products for developers.
• Curiosity about AI and MLOps tooling.
• Expertise in building inference systems that scale for production workloads.

Benefits

• Discretionary bonus
• Equity awards
• Comprehensive benefits program