News: Meet Avesta Labs at the Bengaluru Tech Summit, 17–19 November, Stall H1A036 at BIEC. See event details

Production Monitoring for AI Agents: Catch Wrong Answers

Production Monitoring for AI Agents: Catch Wrong Answers

Your dashboards show the agent is up, fast and busy. They can't show that it just told a customer about a policy you don't have.

Gaurav Soni10 min read

Key takeaways

  • Production monitoring for AI agents has to read the answers. Uptime, error rate and speed all look healthy while an agent states a policy the company doesn't have, which is one kind of hallucination.
  • The public record of Air Canada's and Cursor's wrong answers starts with a customer: a screenshot sent about three months later, and a post on Reddit and Hacker News.
  • Check every conversation with one narrow question per policy area, each with the current policy text beside it, plus a catch-all for any policy the agent states that belongs to none of them.
  • When the answer is a promise about money, law or safety, the first case should reach a named person within a working day. They read the exact reply and decide.

Your AI agent's dashboard is green. It is up, it answers in two seconds, and most chats end without a handover to a person. Meanwhile it has just told a customer about a refund rule your company doesn't have.

Nothing on that dashboard will change. The customer said thanks and left. You will hear about it when they try to claim the refund.

A green dashboard is not evidence the answer was right. You can catch that wrong answer yourself, within a working day.

What is production monitoring for AI agents?

Production monitoring for AI agents means watching a live agent for problems and telling someone in time to act. Most teams watch the system: uptime, errors, speed and cost. An agent that talks to customers needs a second kind of watching, on the answers themselves, because a wrong answer causes no error. Engineers call that a silent failure.

Put simply:

System monitoring tells you the agent replied. Answer monitoring tells you whether the reply was true.

The US standards agency NIST calls the failure confabulation: "the production of confidently stated but erroneous or false content" (NIST AI 600-1, July 2024). Most people call it hallucination. The same document suggests post-deployment monitoring "particularly for potential confabulation".

How did Air Canada's and Cursor's wrong answers come to light?

In both public cases, the record starts with a customer.

  • Air Canada. On 11 November 2022 a customer went to the airline's website to book a flight at its bereavement rate, a reduced fare for travel after a death in the family. They asked its chatbot, which said they could travel first and submit the ticket for the reduced rate within 90 days of its issue. The real policy did not allow claims after travel. The customer applied within the week and was still asking for the refund in February 2023, when they emailed a screenshot of the chat. On 8 February an Air Canada representative replied that the chatbot had used "misleading words". A small-claims tribunal later ordered the airline to pay C$812.02, finding it "did not take reasonable care to ensure its chatbot was accurate" (Moffatt v. Air Canada, 2024 BCCRT 149).
  • Cursor. In April 2025 the code editor's AI support assistant emailed a user that being logged out was expected under a new login policy. The reply was posted on Reddit and Hacker News. A co-founder answered in the thread: "We have no such policy", calling it "an incorrect response from a front-line AI support bot" (The Register, 18 April 2025).

At Air Canada, about three months passed between the chat and the airline's admission. At Cursor, the correction was made in public, under the complaint.

In that case the tribunal held it made no difference whether information "comes from a static page or a chatbot". Air Canada was responsible for both.

Why do the usual dashboards miss wrong answers?

Because each one measures something real that a wrong answer doesn't disturb.

What you monitor What it tells you What a wrong answer does to it
Uptime The agent is running Nothing. It replied
Errors and speed Requests succeed, and quickly Nothing. A wrong reply is fast and error-free
Cost per conversation How much the model was used Nothing
Handled without a person (containment rate) The chat ended without a handover Can improve it. A customer who trusts the answer doesn't ask for a person
Customer rating How the customer felt at the end Can improve it, when the wrong answer is the welcome one

The record engineers keep of each step the agent took, called a trace, has the same blind spot. AI agent tracing shows what the agent did and in what order. It does not say whether the sentence it sent was true.

Layers diagram of five checks a wrong answer meets, top to bottom. The wrong answer, "You can return it for a refund within 60 days", gets through the uptime check because the agent replied, through the error and speed check because the reply was fast and error-free, and through the handled-without-a-person count because the customer believed it and left. The fourth layer, an answer check on every conversation that asks whether each policy the agent stated matches your policy text, catches it. In the fifth, a person reads the exact reply, confirms and fixes.

  1. Uptime check. Passes.
  2. Error and speed check. Passes.
  3. Handled without a person. Counts as a success.
  4. Answer check on every conversation. It asks whether each policy the agent stated matches your policy text. This is where the wrong answer is caught.
  5. A person reads the evidence. They open the exact reply, confirm it is wrong and start the fix.

How do you monitor AI agents in production for wrong answers?

Add layer 4: a second AI model reads every finished conversation and answers a few questions about it. This is an LLM-as-a-judge, where LLM (large language model) is the kind of AI model behind most agents.

1. Pick the answers that would hurt

Don't try to check everything the agent says. Start with the statements a customer can act on: policies, prices, deadlines, eligibility and promises ("we'll refund that"). In the cases above, one agent misstated a real policy and the other described a policy that didn't exist.

2. Give each check its policy text

A judge with only the conversation can tell you the agent stated a refund rule. It can't tell you the rule is wrong. To judge that, it needs the current policy text beside the conversation.

Write one check per policy area (refunds, cancellations, delivery), each with its own short text. Have each one ask whether the agent stated a rule that differs from that text or doesn't appear in it. Keep the text current, or the check will flag correct answers against an old rule. An answer in an area you wrote no check for is not judged by these checks. That is the likely Cursor case: nobody writes a check for a login policy that doesn't exist.

3. Add a catch-all for policies you never wrote

Add one more check that lists the areas you cover and asks: did the agent state a rule, limit or policy outside them? It can't say the rule is wrong. It can put the reply in front of a person, who can.

Then watch the customer: "that's not what your website says", "are you sure?", or a request to have it in writing. A customer disputing an answer is worth a look, whether or not the answer was wrong.

4. Get the first case to a person within a working day

A weekly chart suits slow problems. A wrong policy answer is not slow: one reply is the whole incident. So when the answer is a promise about money, law or safety, the first case should reach a named person within a working day, not at the next weekly review.

Record card showing one flagged answer as a reviewer sees it, with made-up values. Conversation: a chat on 3 March about refunds. Check: does its refund answer differ from our policy? What the agent said: "Refund within 60 days of delivery". Checked against: the current refund policy. Finding: differs from the policy. How serious: high, a promise about money. Reviewer decision: open, a person confirms.

  1. One narrow question. A check that asks one thing is easier to trust than a general quality score.
  2. The exact reply. The finding links to the turn where the agent said it, so nobody rereads the whole chat.
  3. A person decides. The check orders the review. It doesn't close the case.

The judge will be wrong sometimes, in both directions. Before you rely on it, and each week after, have a person mark a sample of conversations and compare. Anthropic says of its own methods for reducing hallucinations that "they don't eliminate them entirely" (Anthropic docs, as of November 2026). In our view the same holds for a model that checks another model.

How we catch wrong answers in Avesta AgentOS

Avesta AgentOS is the platform we build and run AI agents on. Metricsense is its AI agent analytics module, and it is where the answer check lives.

The check is a plain-English question. In Metricsense you write a question such as "Did the agent state a refund rule that differs from the policy below, or isn't in it?". You paste the policy text into the question, because Metricsense holds no copy of your documents. It scores the conversations you send it in batches, about once an hour. Each finding names the turns in the transcript it rests on, and a click takes you to them.

The count is one number. A single-number chart (one figure on a dashboard) counts the conversations where the answer was yes. An email alert on that count is checked every 15 minutes, but it only sees conversations that started in the last hour. A conversation scored after its hour has passed never triggers it.

So treat the alert as a bonus and a daily look as the rule. Have someone open that number at the end of each working day; evening and weekend conversations are read on the next one. A table of the same check opens the flagged conversations. The reviewer reads the cited turns, checks they say what the finding claims, and marks the finding useful or not useful, with a reason.

The fix has an owner. You group the flagged conversations into an action, turn it into a short requirements document and push it to Jira. You can then add the confirmed case to the test cases that Avesta AgentOS runs against a saved version of the agent before release.

The same scoring also reads human phone calls: for Life Insurance Direct, Metricsense has read and scored 15,000+ sales calls, and a person makes every decision.

When is answer monitoring not worth it?

  • Low volume. With a few dozen conversations a week, read them all yourself.
  • The agent only quotes. If every reply is a passage copied from an approved document, with nothing reworded, check the documents instead.
  • Nobody will read the flags. A finding with no reviewer is another unread dashboard. Name the person first.
  • You need it stopped, not found. Monitoring finds a wrong answer after it was sent. For answers that must never go out, add a check before the reply is sent, and see our guide to AI agent guardrails.

The bottom line

Start with one check, on the policy that would cost you most if the agent got it wrong, and name the person who reads its flags. For everything else to watch once an agent is live, see our guide to AI agent monitoring.

If your agent invented a policy this morning, who in your company would know by tomorrow night?


Want to see what your agent is telling customers?

Safe AgentOps is how we run evaluation, guardrails and monitoring for AI agents in production.

Get started: hello@avestalabs.ai

Frequently asked questions

How do you monitor AI agents in production?
Monitor two things. First, the system: is the agent up, how fast does it reply, what does each conversation cost, and do its calls to other systems succeed. Second, the answers: read every conversation with a few narrow checks, such as whether the agent stated a refund rule that differs from your policy text, and send the flagged ones to a person.
How do you evaluate AI agents in production?
Evaluation before launch runs the agent on prepared questions with known answers. In production there are no known answers, so you judge each real conversation against a narrow question and your own policy text, then have a person confirm the flagged ones. Confirmed cases go back into the prepared questions, so the same mistake is tested for before the next release.
What is the difference between AI agent monitoring and AI agent observability?
Observability is the record of what the agent did: each step, each call to another system, the time and the cost. Monitoring is watching for a problem and telling someone. Most observability tools are built for engineers finding out why something failed. Catching a wrong answer needs a check on what the agent told the customer.
How do you test AI agents for accuracy?
Before release, compare its answers with known right answers. After release, count the flagged answers a reviewer confirmed as wrong, per policy area. That confirmed-wrong count is a floor, because the check misses some. To estimate what it misses, have a person read a random sample of unflagged conversations each week.
How do you monitor an LLM in production?
A large language model (LLM) on its own is monitored for speed, cost, errors and the quality of single replies. An agent adds what a bare model doesn't have: several steps, calls to other systems, your documents and a whole conversation with one customer. So agent monitoring also checks what was promised across the conversation, and whether it matches your policy.

Share this article

Written by

Gaurav Soni

Software Engineer