Staff Applied Scientist - Agentic Interfaces
$276k - $345k • New York, New York, USA
Posted 18d ago
Job Location
New York, New York, USA
Tech Stack
Remote Work Policy
On-site
Categories
Applied AI Engineer
About the job
The Agentic Interfaces team at Datadog is seeking a Staff Applied Scientist to lead the evaluation strategy for AI agent integrations. This role focuses on defining and building the measurement systems that ensure AI agents interacting with Datadog's observability, security, and software delivery data are continuously improving. You will be responsible for establishing metrics that cover answer quality, tool selection accuracy, retrieval relevance, latency, and cost, and will partner with AI engineers to implement these systems. This is an opportunity to tackle open research questions in agent-data interaction, such as evaluating non-deterministic trajectories, scoring tool selection in large catalogs, and building scalable measurement systems that benefit both first-party and third-party agents.
Responsibilities
- Own the evaluation strategy for Datadog's AI agent integrations, defining key metrics for quality, cost, and performance.
- Build reusable evaluation datasets, golden traces, and regression harnesses to monitor and improve agent quality.
- Drive measurable improvements in retrieval relevance, tool selection accuracy, and context efficiency.
- Conduct applied research on challenges in agent-data interaction, including tool selection, multi-turn evaluation, and cost/quality tradeoffs.
- Collaborate with internal agent teams to ensure consistent measurement and knowledge sharing.
- Provide technical leadership through design reviews, working groups, and mentorship, and represent the team externally.
Requirements
- BS/MS/PhD in a scientific field, or equivalent experience.
- 10+ years of relevant engineering or applied science experience, including technical leadership.
- Proven track record of leading ML or GenAI initiatives from research to production in a product-driven environment.
- Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
- Strong product mindset and ability to drive initiatives across cross-functional teams.
- Ability to make sound technical decisions in ambiguous situations.
Benefits
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Continuous professional development, product training, and career pathing
- Inclusive company culture, giving programs, and Community Guilds
- Competitive global benefits and global Spring Health benefits
- Healthcare, dental, parental planning, and mental health benefits
- 401(k) plan and match
- Paid time off
- Fitness reimbursements
- Discounted employee stock purchase plan