Lead AI Ops Engineer
Salary not listed • Gurgaon, India • Full-time
Posted 25d ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
Mastercard's Operational Intelligence team is developing advanced data products and AI solutions to provide customers with novel ways to access and utilize operational insights. As AI agents transition from proof-of-concept to production, their real-time behavior, including context retrieval, skill selection, and exception handling, continuously evolves. These agents learn from production outcomes, necessitating active management of their reliability, safety, and trustworthiness beyond simple monitoring. This is the domain of AI Ops: ensuring agents' reasoning remains accurate, their context is up-to-date, and their behavior is governed as they learn.
Responsibilities
- Oversee end-to-end production monitoring for AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out-of-scope rates using observability dashboards.
- Provide L1/L2 production support, including initial monitoring, triage, and escalation to technical teams for newly deployed agentic services.
- Identify and diagnose issues by monitoring evaluation scores and drift alerts, inspecting failing traces, and classifying root causes like intent, tool, parameter, or hallucination errors.
- Evaluate the ongoing correctness of agent reasoning in production, ensuring context retrieval remains current, resolutions align with ground truth, and the learning loop is not drifting.
- Monitor and manage post-production reliability, addressing API and authentication failures, and issues from external AI endpoints to safeguard revenue-generating agents.
- Collaborate with AI Engineering to prioritize fixes, conduct experiments on prompts, context, and routing in development environments, and validate improvements through evaluations before deployment.
- Integrate governance into the agent's learning loop, ensuring approval workflows and audit logging accompany every production change.
Requirements
- Experience monitoring or operating production AI/ML or agentic systems, including observability, evaluation, and drift detection practices.
- Strong analytical skills for diagnosing root causes across intent classification, tool selection, parameter errors, and hallucination patterns.
- Understanding of AI/LLM observability concepts such as tracing, telemetry, guardrails, and evaluation engineering (benchmarks, automated evals, human review, regression testing).
- Familiarity with governance, compliance, and responsible AI practices (e.g., PII detection, access control, audit logging, policy enforcement) is a plus.
- Ability to work independently alongside existing Tech teams while owning the post-launch health of a dedicated agent.
- Excellent communication skills to translate production signals into clear, prioritized requests for AI Product and AI Engineering teams.
- Comfort working in a rapidly evolving environment where the process itself is continuously operated on and improved.