Associate Director/Director, Inference Engineering
$204k - $348k • Cambridge, MA USA; San Francisco, CA USA • Full-time
Posted 5d ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
Lila Sciences is seeking an Associate Director/Director, Inference Engineering to lead the team responsible for the company's model serving platform. This role involves defining how models are served at scale, balancing key performance metrics like latency, throughput, quality, and cost. The position acts as a technical authority, bridging the gap between research and production-grade serving capabilities, and encompasses people leadership, service ownership, cross-functional collaboration, and deep technical expertise in modern LLM serving.
Responsibilities
- Own the inference service end-to-end, including API contracts, latency SLAs, and throughput guarantees.
- Drive decisions on serving architecture, including batching, KV cache, speculative decoding, quantization, and multi-node deployment.
- Establish SLOs for latency, throughput, and availability, and implement on-call and postmortem practices.
- Partner with ML, research, and product engineering teams to make new models production-ready.
- Build systems for model onboarding, deployment safety, evaluation gates, and observability.
- Build, mentor, and retain a high-performing engineering team, and own the serving roadmap.
Requirements
- Deep understanding of the LLM inference stack, including serving frameworks (e.g., vLLM, TensorRT-LLM, TGI), batching strategies, memory constraints, and latency/throughput tradeoffs.
- Experience owning a customer- or product-facing API service end-to-end, including SLAs, incident response, and service reliability.
- Proven track record of hiring, mentoring, and retaining senior technical talent and scaling engineering organizations.
- Strong technical judgment across latency, throughput, reliability, cost, and delivery speed.
- Demonstrated ability to set technical strategy and execute effectively.
- Ability to align stakeholders with competing priorities across ML, research, product engineering, and executive levels.
Benefits
- Competitive base compensation with bonus potential
- Generous early-stage equity
- Comprehensive medical, dental, and vision coverage
- Employer-paid life and disability insurance
- Flexible time off with generous company-wide holidays
- Paid parental leave
- Educational assistance program
- Commuter benefits
- Company-subsidized lunch program