ML Model Serving Engineer
$175k - $280k • Remote • San Francisco • FullTime
Posted 1y ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Sesame is building a new kind of computer focused on making voice agents a part of our daily lives, aiming for lifelike interactions. We are seeking an ML Model Serving Engineer to enhance our serving layer, which handles a variety of LLM, speech, and vision models. This role involves partnering with ML infrastructure and training engineers to create a fast, cost-effective, accurate, and reliable serving layer for a new consumer product. You will modify and extend LLM serving frameworks, identify opportunities to improve model speed without sacrificing quality, and implement techniques like in-flight batching and caching to accelerate inference and reduce initialization times.
Responsibilities
- Optimize the serving layer for LLM, speech, and vision models.
- Collaborate with ML infrastructure and training engineers to build a high-performance serving layer.
- Modify and extend LLM serving frameworks (e.g., VLLM, SGLang) for high-performance serving.
- Work with the training team to improve model speed and quality.
- Implement techniques like in-flight batching, caching, and custom kernels to speed up inference.
- Reduce model initialization times without compromising quality.
Requirements
- Expertise in a differentiable array computing framework, preferably PyTorch.
- Expertise in optimizing machine learning models for reliable, high-throughput, low-latency serving.
- Significant systems programming experience, including high-performance server systems.
- Significant performance engineering experience, including bottleneck analysis and profiling.
- Up-to-date knowledge of the latest model serving optimization techniques.
- Familiarity with high-performance LLM serving frameworks like VLLM and SGlang.
- Experience with public cloud platforms (GCP, AWS, Azure).
- Experience deploying and scaling inference workloads in the cloud using Kubernetes or Ray.
- Proven ability to ship projects and lead complex, multi-month initiatives independently.
- Enthusiasm for learning new technologies and working across multiple roles.
Benefits
- 401(k) with 3.5% employer match
- 100% employer-paid health, vision, and dental benefits for employee and dependents
- Unlimited PTO and sick time
- Flexible spending account with up to $1,650/year employer match
- Guardian Employee Assistance Program (EAP)
- Competitive stock options