Senior Software Engineer, Inference
$152k - $204k • Remote • Sunnyvale, CA / Bellevue, WA
Posted 5d ago
Job Location
Sunnyvale, CA / Bellevue, WA
Tech Stack
Remote Work Policy
Fully remote
Categories
LLM Engineer
About the job
CoreWeave is seeking a Senior Software Engineer focused on Inference to join their team. In this role, you will be an area owner, responsible for leading designs, elevating engineering standards, and delivering significant improvements in latency, throughput, and reliability across multiple services. You will collaborate with product, orchestration, and hardware teams to enhance our Kubernetes-native inference platform and meet stringent P99 SLAs at scale. This is an opportunity to work on a cutting-edge AI cloud platform during a period of hyper-growth, contributing to innovative solutions for complex problems.
Responsibilities
- Lead design reviews and drive architecture within the team, decomposing multi-service work into clear milestones.
- Define and own SLIs/SLOs, ensuring post-incident actions lead to improved reliability.
- Implement advanced optimizations for inference, such as micro-batch schedulers, speculative decoding, and KV-cache reuse, and quantify their impact.
- Strengthen incident response through capacity planning, autoscaling policy, graceful degradation, and traffic-shift strategies.
- Mentor junior engineers and review cross-team designs to elevate coding and testing standards.
- Own an area spanning multiple services and teams, including request routing, adaptive scheduling, cost-per-token analytics, and GPU resource isolation.
Requirements
- Approximately 5-8 years of industry experience building distributed systems or cloud services.
- Strong coding proficiency in Python or Go, with C++ as a plus.
- Deep familiarity with networked systems and performance optimization.
- Experience optimizing end-to-end ML system performance by developing and tuning CUDA kernels, reducing model latency, maximizing compute and memory bandwidth, and leveraging custom accelerators.
- Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry).
- Practical knowledge of inference internals, including batching, caching, mixed precision (BF16/FP8), and streaming token delivery.
- Proven track record of improving tail latency (P95/P99) and service reliability through metrics-driven work.
Benefits
- Discretionary bonus
- Equity awards
- Comprehensive benefits program
- Opportunity for growth within the organization