Member of Technical Staff (Data Scientist, Evals)
Remote • San Francisco • FullTime
Posted 2mo ago
About the job
Perplexity is seeking a Member of Technical Staff (Data Scientist, Evals) to join their team. In this role, you will build specialized evaluations to improve answer quality across Perplexity's LLM-first search engine and other user-facing scenarios. You will architect and maintain automated evaluation pipelines, design evaluation sets to measure the impact of tool calls on answer quality, and develop VLM-based solutions for visual rendering evaluation. You will also adapt and incorporate public benchmarks and academic evaluations into regular performance measurements, working closely with technical leadership to shape product changes.
Responsibilities
- Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products.
- Design evaluation sets and methods to measure the impact of tool calls on final answer quality.
- Develop VLM-based solutions to programmatically evaluate how final answers render visually.
- Continuously review public benchmarks and academic evaluations for applicability to the Perplexity product.
- Adapt and incorporate public benchmarks and academic evaluations into regular performance measurements.
- Operate within a small, high-impact team where evaluation metrics directly shape product changes.
- Collaborate closely with technical leadership to measure and improve Answer Quality.
Requirements
- PhD or MS in a technical field or equivalent experience.
- 4+ years of experience in data science or machine learning.
- Strong proficiency in Python and SQL.
- Experience building within a modern cloud data stack, specifically AWS and Databricks.
- Comfortable with agentic coding workflows and using AI-assisted development tools.