Director, Research - AI Evals
$258k - $348k • Remote • San Francisco, CA • New York, NY • United States
Posted 2mo ago
Job Location
San Francisco, CA • New York, NY • United States
Tech Stack
Remote Work Policy
Fully remote
Categories
Applied AI Engineer
About the job
Figma is seeking a Director, Research - AI Evals to lead the measurement of quality for Figma's AI-powered experiences. This role is crucial as Figma integrates more AI capabilities, requiring a rigorous approach to defining and assessing "good" AI features. You will establish frameworks, quality standards, and trusted signals that guide product teams in their AI feature development and release decisions. The ideal candidate possesses deep, hands-on experience in evaluating AI/LLM products, combining human and automated methods, with strong product intuition and communication skills to ensure evaluation is impactful. You will collaborate with Product, Design, Engineering, and Data Science teams, influencing AI shipping decisions that affect millions of users.
Responsibilities
- Own AI evaluation methods and operations for Figma's AI-powered experiences, defining quality dimensions, measurement strategies, and translating results into actionable signals.
- Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, integrating human evaluation with automated/model-based approaches.
- Partner with engineering to establish repeatable and reproducible evaluation pipelines and regression testing for AI features.
- Produce clear readouts and dashboards to support stakeholder go/no-go and prioritization decisions.
- Socialize a unified definition of quality and advocate for evaluation as a strategic partner in the product development process.
- Manage a team to execute AI evaluations in collaboration with contractors, internal staff, and/or LLMs.
Requirements
- 10+ years of experience in product, research, applied research, or a related field, with at least 2 years of management experience.
- Direct, hands-on experience owning the evaluation of AI/LLM-powered products.
- Expertise in designing and running AI evaluations, including human evaluation programs, rubric and golden dataset construction, and inter-rater reliability.
- Proficiency in applying automated/model-based approaches (e.g., LLM-as-judge) and understanding their limitations.
- Strong qualitative and quantitative methods skills, comfort with data and metrics, and ability to reason about model behavior.
- Demonstrated success in identifying and prioritizing risky assumptions for ambiguous quality questions and designing appropriate evaluations.
- Proven track record of gaining buy-in from executive and cross-disciplinary stakeholders, articulating user value and impact.