New: upload your resume and get matched to AI jobs. Free to start →

AI Agent Evaluation Engineer

Remote • Canada (Remote); United States (Remote) • Full-time

Posted 8d ago

Job Location

Canada (Remote); United States (Remote)

Tech Stack

Remote Work Policy

Fully remote

Employment Type

Full-time

Categories

AI Engineer

About the job

Cresta is seeking an AI Agent Evaluation Engineer to define and measure the quality of their AI Receptionist, which handles millions of customer conversations monthly. This role is crucial for ensuring the AI's performance across various customer interactions. You will be responsible for establishing quality standards, building systems to measure performance, and providing the company with a trusted assessment of the AI's effectiveness. The work involves analyzing conversations, designing rigorous evaluation frameworks, and implementing automated evaluation systems, with a focus on identifying and prioritizing areas for improvement.

Responsibilities

  • Define and maintain evaluation rubrics for AI agents across adherence, outcome, and conversation experience.
  • Develop annotation guidelines and failure taxonomies for precise communication about agent failures.
  • Classify and prioritize failure modes based on severity and frequency.
  • Review production conversations weekly to identify and cluster failures.
  • Trace failures to their root causes and route them to the appropriate engineering or product teams.
  • Evaluate agent performance across different languages and segments.
  • Build and operate the evaluation pipeline for scoring production conversations.
  • Calibrate automated judges against human labels to ensure accuracy.
  • Design sampling strategies for quality reporting.
  • Build and maintain golden sets and regression suites for validating changes.
  • Maintain a coverage map of tested behaviors and identify gaps.
  • Design evaluations for complex and high-stakes conversation types.
  • Define quality and latency thresholds for the AI agent.
  • Report quality trends to stakeholders clearly and concisely.
  • Collaborate with engineering and product teams to translate findings into recommendations.
  • Document evaluation methodologies for reproducibility.

Requirements

  • 3+ years of professional experience in software engineering, applied ML, or data-intensive research.
  • 1-2 years of hands-on experience with LLM-based systems in production.
  • Bachelor's or Master's degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.
  • Experience building and operating an end-to-end evaluation system for a production LLM product.
  • Deep understanding of evaluation methodologies (golden sets, judges, regression suites, calibration).
  • Fluent understanding of how LLM agents work (prompts, tool calls, context handling, multi-turn state).
  • Strong programming skills in Python.
  • Experience with LLM APIs.
  • Ability to build and ship evaluation tooling independently.