Lead Product Data Scientist, Agents
$172k - $258k • Boston, MA • Full-time
Posted 1d ago
About the job
Klaviyo is seeking a Lead Product Data Scientist, Agents to spearhead the measurement and data science strategy for their AI-powered agent products, Composer and Customer Agent. This high-visibility, hands-on technical leadership role involves defining how to measure the adoption, engagement, quality, cost, and business impact of these AI agents. You will establish the analytical foundations to understand customer value, tackle emerging measurement challenges, and develop rigorous approaches to evaluating agent performance. The role requires combining experimentation, causal inference, and advanced analytics to inform product strategy, pricing, and monetization, translating complex findings into actionable recommendations. You will also mentor other data scientists and define the technical direction for agent analytics at Klaviyo.
Responsibilities
- Own agent measurement end-to-end across Composer and Customer Agent, developing comprehensive measurement frameworks.
- Define successful agent performance by establishing meaningful measures of effectiveness, task completion, customer outcomes, and failure rates.
- Build scalable agent evaluation frameworks using conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification.
- Measure causal impact and incremental value through experimentation and advanced statistical methods.
- Connect agent performance to business strategy by evaluating pricing, packaging, credit economics, and model costs.
- Establish trusted metrics and analytical standards by defining consistent measurement methodologies and governed data models.
- Lead with an AI-first approach to analytics, leveraging coding agents and LLMs to accelerate workflows.
- Build scalable, self-service analytics capabilities for stakeholders to explore agent performance.
- Partner with Engineering and Analytics Engineering to develop reliable instrumentation and data pipelines.
- Lead the technical direction of agent analytics, reviewing approaches, mentoring data scientists, and establishing best practices.
Requirements
- Deep technical expertise in data science, AI product development, and business strategy.
- Experience defining measurement frameworks for AI products.
- Proficiency in experimentation and causal inference.
- Experience with advanced statistical methods (e.g., difference-in-differences, matched comparisons).
- Ability to translate complex analytical findings into actionable recommendations.
- Experience mentoring other data scientists.
- Familiarity with AI agents, LLMs, and emerging analytical tools.
- Experience developing data pipelines and dbt models.
- Strong understanding of data quality and statistical rigor.