Lead Machine Learning Engineer, Evaluations
$170k - $190k • Mountain View • Full-time
Posted 1mo ago
Job Location
Mountain View
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
Machine Learning Engineer
About the job
ASAPP is seeking an enterprising, talented, and curious Machine Learning Engineer to join their AI Engineering team. This role will focus on building and evaluating the core intelligence behind agentic AI systems, playing a key part in designing and owning evaluation frameworks that ensure quality, safety, and performance across complex agentic systems. The Lead Machine Learning Engineer will own and grow the evaluation platform that measures quality, safety, and performance across ASAPP's agentic AI systems, ensuring confidence in model and agent changes before they reach customers.
Responsibilities
- Develop the technical roadmap and architecture for the evaluation platform, including offline benchmarking and online monitoring of agentic and LLM-based systems.
- Design evaluation methodologies for different pipeline stages, such as golden test sets, human-in-the-loop review, LLM-as-judge approaches, and automated metrics for task success, safety, and hallucinations.
- Build data infrastructure for evaluation, including annotation pipelines, dataset versioning, data quality checks, and tooling for researchers and product teams.
- Partner with Research, Product, and Platform teams to productize experiments into robust AI solutions.
- Represent the evaluation platform to stakeholders, setting expectations for model/agent releases and reporting on platform health.
- Stay current with advancements in ML, NLP, voice, and LLM systems, and contribute to technical discussions.
- Mentor and support other engineers through design reviews, feedback, and knowledge sharing.
Requirements
- Deep, hands-on experience building and operating evaluation systems for modern ML/LLM/agentic systems.
- Demonstrated experience leading the technical direction of a project or small team, including setting architecture and driving design reviews.
- Strong architectural skills with proven experience designing complex, data-intensive software systems.
- Production experience with Python, AWS, Kubernetes, and/or Docker.
- Experience designing data pipelines for ML evaluation, including labeling workflows, dataset versioning, quality control, and reproducible benchmarking.
- A Bachelor’s Degree in CS or other related fields.
- Demonstrated technical mentorship of junior and mid-level engineers.
- Desire to learn, teach, and collaborate closely with cross-functional peers.