Staff / Principal Machine Learning Engineer, Serving - USA
$270k - $500k • Remote • Mountain View, California, USA • FullTime
Posted 5mo ago
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Inworld is seeking a Staff/Principal Machine Learning Engineer to join their team in Mountain View, USA. This role focuses on optimizing real-time inference for state-of-the-art AI models, which power large consumer-facing applications. The ideal candidate will have a strong background in systems programming and a proven ability to take models from research to production, ensuring reliability and performance. The company values engineers who can navigate ambiguity, drive impact, and deeply understand the underlying logic of their work, fostering a collaborative and fast-paced environment.
Responsibilities
- Optimize inference for real-time AI models.
- Develop and implement techniques for model acceleration, such as quantization and distillation.
- Build and maintain high-performance systems using languages like C++, CUDA, or Rust.
- Design and scale distributed systems for handling thousands of concurrent connections.
- Take ownership of the full lifecycle of a model, from research to production serving.
- Collaborate with researchers and engineers to improve model performance and reliability.
- Contribute to open-source projects and share work that advances the field.
Requirements
- Deep understanding of modern serving frameworks and techniques (e.g., vLLM, TRT-LLM).
- Hands-on experience with quantization, distillation, caching strategies, continuous batching, paged attention, and speculative decoding.
- Proficiency in C++, CUDA, Rust, or highly optimized Python.
- Experience with profiling code and optimizing for NVIDIA GPUs.
- Experience with Kubernetes, Ray, custom load balancing, multi-GPU/multi-node inference.
- Experience reliably handling thousands of concurrent connections.
- Demonstrated ability to take a model from research to production serving.
- PhD in CS, Physics, Math, or equivalent practical experience building backend or ML systems.
- Ability to pick a direction and build the map as you go.
- Bias for impact and shipping stable, reliable code.
Benefits
- Bonus
- Equity
- Benefits
- Relocation assistance