Member of Technical Staff (Answer Quality & Evals)
Remote • San Francisco • FullTime
Posted 5mo ago
About the job
Perplexity is seeking a Member of Technical Staff to join the Answer Quality team. This role focuses on building and improving the technical foundations that support answer quality across Perplexity's LLM-first search engine. You will develop evaluation systems that are fast, reliable, production-faithful, and actionable, including shared evaluation infrastructure and a platform for replaying and analyzing agent traces. You will collaborate with data scientists, engineers, and product teams to identify quality issues, measure their impact, and translate findings into product enhancements, directly shaping how Perplexity measures and improves answer quality.
Responsibilities
- Build shared evaluation infrastructure for reliable evals, result analysis, and product/model decisions.
- Develop a platform for replaying and analyzing agent traces to reproduce production behavior and diagnose failures.
- Build and operate scalable systems for processing, storing, and monitoring interaction, trace, and evaluation data.
- Partner with data scientists, engineers, and product teams to translate answer-quality problems into evaluations, analyses, and product improvements.
- Operate within a small, high-impact team to shape answer quality measurement and improvement.
Requirements
- 4+ years of software, data, or machine learning engineering experience shipping and operating production systems.
- Strong proficiency in Python and SQL.
- Solid fundamentals in system design, data modeling, and distributed systems.
- Experience building big-data systems, including distributed compute, large-scale storage, and high-volume pipelines.
- Demonstrated ownership of ambiguous technical projects from initial design through production operation.
- Ability to work effectively with data scientists, engineers, and product partners.