Engineering Manager, Inference Router
Sunnyvale, CA • Full-time
Posted 1mo ago
Job Location
Sunnyvale, CA
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
The Inference Router team at Cerebras is responsible for building the intelligent control plane that directs every inference request across Cerebras' global data centers and third-party integrations. This system balances heterogeneous compute, maximizes cache utilization, and aims to deliver the industry's lowest time-to-first-token (TTFT) at massive scale. As an Engineering Manager, you will lead this team, focusing on global request routing, prefill/decode disaggregation, prefix-aware routing, autoscaling, and high-availability infrastructure to ensure a fast and reliable platform under demanding, real-time load. This role blends technical leadership, people management, and execution ownership at the intersection of distributed systems, Kubernetes, routing algorithms, and machine learning.
Responsibilities
- Own the architecture and evolution of the inference routing layer across all Cerebras data centers and third-party inference integrations.
- Guide the design of global, multi-region request routing and load balancing.
- Guide the design of routing for heterogeneous compute, including prefill/decode disaggregated serving.
- Guide the design of prefix-aware/cache-aware routing to maximize KV-cache reuse and utilization.
- Guide the design of low-TTFT, high-throughput request scheduling under concurrent, bursty workloads.
- Drive routing algorithms that trade off cache locality, queueing, load, and latency in real time.
- Build high-fidelity simulation models of the fleet and its workloads to validate routing policies, failover behavior, and scaling decisions.
- Partner with inference runtime, cloud/platform, hardware, and ML teams to optimize end-to-end performance.
- Build, manage, and grow a team of distributed systems, infrastructure, and ML systems engineers.
- Provide technical direction, mentorship, and career development to the team.
- Foster a culture of ownership, velocity, and operational excellence.
- Recruit top talent in distributed systems, Kubernetes/infrastructure, and routing/ML systems.
- Drive execution of complex, cross-functional initiatives spanning inference runtime, cloud/infrastructure, and ML teams.
- Own delivery of capabilities such as prefix- and cache-aware routing policies.
- Own delivery of capabilities such as P/D-disaggregation-aware scheduling and placement.
- Own delivery of capabilities such as autoscaling and capacity management across heterogeneous fleets.
- Own delivery of capabilities such as high availability, failover, and graceful degradation.
- Own delivery of simulation and load-generation frameworks that exercise the router under diverse traffic patterns, failure modes, and fleet topologies.
- Ensure high-quality releases through strong testing, load/chaos validation, and operational rigor.
- Scale the routing layer to handle very large volumes of concurrent requests at the lowest possible latency.
- Drive improvements in TTFT, throughput, cache-hit rate, and compute efficiency across the fleet.
- Design for reliability and high availability: failover, redundancy, scale-out, and fail-closed behavior under partial outages.
- Identify and prioritize technical debt and system bottlenecks.
- Maintain Cerebras’ industry-leading inference speed advantage.
- Partner with inference runtime teams (execution and serving).
- Partner with cloud/platform and SRE teams (deployment, scaling, reliability).
- Partner with ML researchers and engineers (model enablement, routing signals).
- Partner with third-party integration partners.
- Act as a bridge between infrastructure, routing, and production systems.
Requirements
- 8+ years of experience in large-scale software engineering, distributed systems, or infrastructure.
- 2+ years of engineering management experience.
- Strong programming skills in C++ and/or Go for performance-critical, concurrent production systems.
- Deep experience operating production services on Kubernetes: deployments, autoscaling, rollout strategies, and multi-cluster/multi-region topologies.
- Proven track record building and operating highly available, scalable systems: load balancing, failover, scale-out, and reliability engineering.
- Experience designing or operating request routing, load balancing, scheduling, or traffic-management systems at scale.
- Experience with cloud infrastructure and best practices for building scalable microservices.
- Experience with large-scale LLM/inference serving and the systems around it (e.g. vLLM, TensorRT-LLM, SGLang, or comparable).
- Familiarity with prefill/decode (P/D) disaggregation, KV-cache management, and prefix/cache-aware routing.
- ML background: understanding of inference workloads, batching, and the performance characteristics of serving LLMs and multimodal models.
- Experience with autoscaling.
