Member of Technical Staff - Evaluations
San Francisco • Full-time
Posted 10d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
Liquid AI, a rapidly growing company spun out of MIT CSAIL, is building general-purpose AI systems designed for efficiency across various deployment targets, from data center accelerators to on-device hardware. We focus on delivering low latency, minimal memory usage, privacy, and reliability for enterprise clients in sectors like consumer electronics, automotive, life sciences, and financial services. This role is crucial for our Post-Training team, ensuring that all post-training decisions are backed by internal evaluations that accurately reflect real-world behavior and are both cost-effective and reproducible. You will be instrumental in developing and maintaining our internal evaluation suite and its underlying infrastructure, collaborating closely with researchers and other teams to provide the evidence needed for critical model development decisions.
Responsibilities
- Maintain the internal eval suite and infrastructure, ensuring reliability, optimization, and reproducibility at scale.
- Design held-out evaluations for key capabilities like agentic workflows, computer use, and cybersecurity, including documentation and contamination checks.
- Develop evaluations specifically for small edge models, addressing limitations of public evaluations designed for frontier models.
- Debug unexpected evaluation scores by reviewing individual generated outputs.
- Translate customer and open-source feedback into usable environments for data, training, and evaluation.
- Partner with other teams to design evaluations tailored to their specific model needs.
- Release open-source evaluations, publish research papers on them, and document failure modes of popular evaluations.
Requirements
- Demonstrated strength in eval design, capable of translating high-level capabilities into accurate, general-purpose, and game-resistant evaluations.
- Proven experience building recognizable and verifiable evaluations, with a strong understanding of coverage, reproducibility, contamination handling, and signal integrity.
- Proficiency in evaluation fundamentals such as contamination, saturation, judge bias, and variance.
- Hands-on experience managing and running evaluations within a team environment.
- Strong software engineering skills to independently manage eval infrastructure and debug inference/infra failures at scale.
- Ability to debug unexpected scores by examining individual generated answers.
Benefits
- Competitive base salary
- Equity
- Unlimited PTO
- Company-wide Refill Days
- Long weekends
- Comprehensive health coverage
- Paid leave for pregnancy, parenting, medical needs, and caring for family
