Senior Machine Learning Engineer
$204k - $255k • San Francisco • FullTime
Posted 2mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Ambience is building an AI intelligence platform to improve healthcare by reducing administrative burdens for providers. Our technology assists with real-time coding and clinical workflow support across various healthcare settings. We are seeking a Senior Machine Learning Engineer to build and enhance the AI systems powering our clinical products. This hands-on role involves end-to-end ownership of complex projects, from diagnosing production issues and designing evaluations to developing, deploying, and iterating on models and agentic systems. You will collaborate with clinicians, product managers, and engineers to translate cutting-edge research into reliable, production-grade AI solutions.
Responsibilities
- Design and own evaluation pipelines for LLM and agentic systems, incorporating automated graders, regression testing, production feedback, and human evaluation.
- Diagnose and address high-impact failure modes in production models through interventions in prompting, retrieval, context, routing, data, or fine-tuning.
- Develop production systems for agentic AI, including tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
- Build data and improvement flywheels by leveraging production failures and user feedback for active learning and systematic iteration.
- Distill insights from recent research in LLMs, agents, NLP, speech, and multimodal AI to drive practical experiments.
- Manage AI systems end-to-end, encompassing models, data, evaluation, orchestration, serving, and observability, with a hands-on approach to code and debugging.
Requirements
- 5+ years of experience in production ML, research engineering, or applied AI.
- Proven experience building consequential production AI systems or significantly improving model behavior in production.
- Strong understanding of modern LLMs, transformers, and production AI systems.
- Experience designing evaluations for LLMs, agents, or complex AI systems, capable of translating ambiguous problems into measurable dimensions and experiments.
- Familiarity with challenges in AI evaluation such as grader bias, leakage, misleading metrics, regression detection, and offline-online mismatch.
- Experience building production systems involving multiple models, tools, retrieval, context, state, routing, or orchestration.
- Proficiency in Python and modern ML frameworks, with PyTorch preferred.
- Comfort with deployment, observability, CI/CD, and containerized systems.
- Skilled in building high-quality datasets and feedback loops, utilizing production failures, user feedback, and active learning.
- Ability to collaborate effectively with clinicians, product managers, and fellow engineers, simplifying complex AI concepts for diverse audiences.
- Comfortable owning ambiguous technical problems and driving them to measurable outcomes.