Skip to content

Senior QA Engineer, AI Agents & Agentic Workflows

QAD, Inc.Pune, MH, IndiaHybrid

Estimated salary, not published by the employer: Est. ₹5.5–8 LPA

Needs a free Superjobs account. You'll finish on QAD, Inc.'s SmartRecruiters page.

Skill score

Summary

This person owns continuous quality validation of QAD's Champion Assist multi-agent AI product, testing agent reasoning, tool use, orchestration, handoffs, and response quality beyond standard UI and API checks. They suit a senior QA engineer with LLM or agentic testing experience who can design evaluation frameworks and regression suites for non-deterministic systems.

Function
Testing
Estimated salary
Est. ₹5.5–8 LPA
Estimated for a senior role, ~3 yrs experience, India. Not published by the employer.
Work mode and location
Hybrid, Pune, MH, India
Experience
3–5 yrs
Employment type
Full-time

Job description

Overview

QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi- agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use, orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks.

You will design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM- driven systems.

Roles and responsibilities

  • Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows, including agent routing, tool/function calls, and multi-turn conversations.
  • Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and completeness, using both human review and automated methods.
  • Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated regression suites that run as agents, prompts, models and tools change.
  • Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs, tool-call failures, context loss, looping or runaway behavior.
  • Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
  • Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and quantify them with metrics.
  • Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool and orchestration causes.
  • Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
  • Document test strategy, eval results and quality trends clearly for global stakeholders.

Skills required

Top skills

  • QA engineering
  • LLM application testing
  • AI agent testing
  • Agentic workflows

All skills (9)

  • Evaluation rubrics
  • Eval regression suites
  • Golden datasets
  • LLM-as-judge
  • Test automation
  • Selenium
  • Playwright
  • API testing
  • RAG

Qualifications

  • Bachelor's degree in software engineering, computer science or a related field.
  • 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or agentic workflows.
  • Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and acceptance thresholds for AI responses.
  • Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets, scoring rubrics).
  • Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning, memory/context, RAG.
  • Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.

Other details

  • Your health and well being are important to us at QAD. We provide programs that help you strike a healthy work-life balance.
  • Opportunity to join a growing business, launching into its next phase of expansion and transformation.
  • Collaborative culture of smart and hard-working people who support one another to get the job done.
  • An atmosphere of growth and opportunity, where idea-sharing is always prioritized over level or hierarchy.
  • Compensation packages based on experience and desired skill set

Company info

QAD | Redzone is redefining manufacturing and supply chains through its intelligent, adaptive platform that connects people, processes, and data into a single System of Action. With three core pillars — Redzone (frontline empowerment), Adaptive Applications (the intelligent backbone), and Champion AI (Agentic AI for manufacturing) — QAD | Redzone

More jobs at QAD, Inc.