Senior Software Engineer, Model Serving
$166k - $225k • Remote • San Francisco, California
Posted 5d ago
Job Location
San Francisco, California
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
Databricks' Model Serving product offers enterprises a unified, scalable, and governed platform for deploying and managing AI/ML models, including traditional ML and large language models. It provides real-time, low-latency inference, governance, monitoring, and lineage, making it a core pillar for customers operationalizing models at scale with strong SLAs and cost efficiency. As a Senior Engineer, you will be instrumental in shaping both the product experience and the foundational infrastructure of Model Serving. You will design and build high-throughput, low-latency inference systems for CPU and GPU workloads, influence architectural direction, and collaborate with cross-functional teams to deliver a world-class serving platform.
Responsibilities
- Design and implement core systems and APIs for Databricks Model Serving, ensuring scalability, reliability, and operational excellence.
- Drive architectural decisions and trade-offs to optimize performance, throughput, autoscaling, and operational efficiency for CPU and GPU serving workloads.
- Contribute to key components of the serving infrastructure, including model container builds, deployment workflows, and runtime systems like routing, caching, observability, and intelligent autoscaling.
- Collaborate with product, platform, and research teams to translate customer needs into reliable and performant systems.
- Lead technical initiatives to improve latency, availability, and cost-effectiveness across serving layers.
- Establish best practices for code quality, testing, and operational readiness, and mentor other engineers.
Requirements
- 5+ years of experience building and operating large-scale distributed systems.
- Experience in model serving, inference systems, or related infrastructure (e.g., routing, scheduling, autoscaling, observability).
- Strong foundation in algorithms, data structures, and system design for large-scale, low-latency serving systems.
- Proven ability to deliver technically complex, high-impact initiatives.
- Experience building architecture for large-scale, performance-sensitive CPU/GPU inference systems.
- Strong communication skills and ability to collaborate across teams.
- Customer-focused mindset.
- Passion for mentoring engineers and fostering technical excellence.
Benefits
- Eligibility for annual performance bonus
- Equity
- Comprehensive benefits and perks