New: upload your resume and get matched to AI jobs. Free to start →

Senior QA Engineer, AI Agents & Agentic Workflows

Pune, MH, India • Full-time

Posted 15d ago

Job Location

Pune, MH, India

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Agent Engineer

About the job

QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi-

agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use,

orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks.

You will design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM-

driven systems.

Responsibilities:

  • Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows,

including agent routing, tool/function calls, and multi-turn conversations.

  • Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and

completeness, using both human review and automated methods.

  • Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated

regression suites that run as agents, prompts, models and tools change.

  • Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,

tool-call failures, context loss, looping or runaway behavior.

  • Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
  • Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and

quantify them with metrics.

  • Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool

and orchestration causes.

  • Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
  • Document test strategy, eval results and quality trends clearly for global stakeholders.
  • Bachelor's degree in software engineering, computer science or a related field.
  • 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or

agentic workflows.

  • Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and

acceptance thresholds for AI responses.

  • Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets,

scoring rubrics).

  • Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning,

memory/context, RAG.

  • Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.