Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)
$220k - $405k • Remote • San Francisco • FullTime
Posted 7h ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Agent Engineer
About the job
The Enterprise Agent Evals & Efficiency team at Perplexity focuses on enhancing the capabilities of AI agents to perform real-world tasks across various enterprise connectors, tools, and data sources. The primary objective is to enable agents to select the appropriate tools, utilize relevant context, and execute complex workflows reliably, while minimizing time and computational resources. This role involves end-to-end ownership, from developing realistic enterprise evaluation environments to implementing improvements in the agent harness, tools, and skills. You will create intricate scenarios involving connected systems, data, and permissions to pinpoint agent weaknesses and refine the tools, skills, context, and models they employ. Collaborating with AI Research, Connectors, and Platform teams, you will translate experimental findings into tangible improvements in task success rates, latency, and cost efficiency.
Responsibilities
- Build realistic enterprise evaluation environments with multi-step workflows and reproducible system states, encompassing connected applications, datasets, user roles, and permissions.
- Own evaluation infrastructure, task suites, and automated grading systems to measure agent performance on enterprise tasks, calibrated against human judgment and outcome-based metrics.
- Analyze agent behavior and production failures to identify root causes and implement improvements in the agent harness, tools, and skills to enhance task success and reliability.
- Develop reusable skills and refine tool descriptions, schemas, and error handling to improve agent tool selection, workflow execution, and error recovery.
- Design and evaluate model selection and routing strategies to optimize the balance between quality, latency, reliability, and cost for each task.
- Enhance agent efficiency through optimized context management, skill loading, caching, and execution strategies to reduce token usage and cost per task without compromising quality or safety.
- Implement regression gates and launch-readiness signals for various components and validate that evaluation gains translate to improved production outcomes.
Requirements
- 5+ years of software engineering or machine learning engineering experience with ownership of complex production systems.
- Strong system design skills for building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applications.
- Proficiency in backend languages like Python, Go, or Rust, and experience with multi-language production stacks.
- Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflows.
- Deep expertise in one or more of the following: evaluation infrastructure, agent harnesses, test environment simulation, or model selection/inference efficiency.
- Strong experimental and statistical judgment for defining success criteria, diagnosing failures, and identifying genuine improvements.
- Sound technical judgment in balancing task quality, latency, reliability, and cost, with a history of translating evaluation findings into shipped improvements.
- Proven track record of leading ambiguous, cross-team initiatives, setting technical direction, and mentoring others to raise engineering quality.