Senior ML Infrastructure Engineer
$200k - $275k • Remote • Austin, Texas, United States; San Jose, California, United States
Posted 17h ago
Job Location
Austin, Texas, United States; San Jose, California, United States
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
SambaNova is seeking a Senior Software Engineer to join the ML Infrastructure team. This role is central to our mission as an inference-first company, focusing on designing, building, and operating the production-grade inference infrastructure that powers SambaNova's serving stack on our Reconfigurable Dataflow Unit (RDU) architecture. You will transform state-of-the-art inference techniques into reliable, high-throughput, low-latency services exposed to customers through SambaStack and SambaCloud. This position involves end-to-end ownership of systems including request scheduling, advanced decoding algorithms, caching layers, API surfaces, and accuracy infrastructure, and requires close collaboration with ML, compiler, runtime, and product teams.
Responsibilities
- Design and productionize advanced inference techniques on RDU for performance and cost optimization, including speculative decoding, constrained decoding, function/tool calling, prompt caching, and long-context inference.
- Own SambaNova's integration with vLLM and adjacent serving frameworks, adapting them to RDU's architecture.
- Own the public inference API surface exposed through SambaStack and SambaCloud.
- Build and maintain accuracy verification and regression infrastructure for inference features.
- Partner with ML, compiler, runtime, and product teams to ship inference features from prototype to production.
- Contribute to technical design discussions, code reviews, and architectural decisions as a senior individual contributor.
Requirements
- B.S. in Computer Science, Electrical Engineering, or related field.
- 5+ years of industry experience building and operating large-scale distributed systems, preferably in ML serving.
- Strong software engineering fundamentals in algorithms, data structures, concurrency, and systems design.
- Experience designing and maintaining production services with strict latency, throughput, and availability requirements.
- Working knowledge of modern LLM inference techniques and familiarity with open-source serving stacks like vLLM, TensorRT-LLM, or SGLang.
- Proficiency in Python.
- Experience collaborating across teams to deliver complex, system-level engineering solutions.
Benefits
- Equity
- 95% premium coverage for employee medical insurance
- 77% premium coverage for dependents
- Health Savings Account (HSA) with employer contribution
- Dental insurance
- Vision insurance
- Short-term disability insurance
- Long-term disability insurance
- Basic life insurance
- Voluntary life insurance
- Accidental Death & Dismemberment (AD&D) insurance
- Flexible Spending Account (FSA) options (Health Care, Limited Purpose, Dependent Care)
- Well-being benefits for employees and dependents