New: upload your resume and get matched to AI jobs. Free to start →

Senior AI Evals Engineer

Bangalore • Full-time

Posted 1mo ago

About the job

The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents. This role bridges AI development and reliable deployment, ensuring agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement. The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards. The role is 70% Automation and 30% Manual Testing.

Responsibilities

  • Design and build agent evaluation frameworks for correctness, latency, cost, safety, and task completion.
  • Build synthetic testing environments and simulation strategies to stress-test AI agents.
  • Curate domain-specific evaluation sets and build test harnesses.
  • Build and enhance a centralized evaluation platform for self-service agent testing.
  • Automate regression and drift detection using CI/CD-integrated eval pipelines.
  • Define structured scoring criteria (eval taxonomies and rubrics) for agentic workflows.
  • Build real-time monitoring for agent quality in production.
  • Run structured model comparisons to inform model selection decisions.
  • Analyze and interpret evaluation results, providing actionable quality and safety reports.
  • Define comprehensive end-to-end QA strategies for conversational and goal-oriented AI agents.
  • Develop and maintain detailed test artifacts for agent features, tool invocation, memory, and reasoning.
  • Identify, document, prioritize, and track defects and performance regressions.
  • Evaluate new technologies and patterns for Agent Evals and influence the platform roadmap.
  • Partner with policy and legal teams to ensure evaluations cover regulatory compliance and AI governance.
  • Provide technical leadership, including solutioning, design reviews, code reviews, and mentoring.
  • Collaborate cross-functionally to embed eval-driven quality gates into the agent development lifecycle.

Requirements

  • Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.
  • 8+ years of overall software Test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling.
  • Demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or LLMs.
  • 2+ years working with ML/AI systems, building, testing, or evaluating models in production environments.
  • Hands-on experience with LLM-based applications, prompt engineering, RAG pipelines, or agentic workflows.
  • 2+ years as a senior/lead, owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems.
  • 1+ years in designing or operating evaluation frameworks specifically for generative AI / LLM agents.
  • Deep hands-on experience with LLM evaluation frameworks and libraries (e.g., LangSmith, DeepEval, Promptfoo, RAGAS).
  • Strong understanding of agentic AI patterns (tool use, function calling, multi-step planning, RAG).
  • Production experience building automated, CI-integrated evaluation infrastructure.
  • Proficiency in Python with strong scripting and tooling skills (e.g., pytest, async patterns).
  • Familiarity with Google Cloud Platform (GCP) or other relevant AI/ML cloud services (e.g., Vertex AI).
  • Experience integrating testing into MLOps workflows.
  • Statistical literacy for designing experiments and interpreting results.