What is AI evaluation? A guide for teams putting AI into production

AI evaluation is the practice of measuring whether an AI system produces the right output on your own data. It covers accuracy, grounding, safety, cost and what happens at the edges.

It applies to language models, retrieval systems and agents. Anywhere the same input can produce a different output twice.

Traditional software testing checks whether code does what it was written to do, and the answer is the same every time you run it. AI evaluation checks whether a system that can answer differently on Tuesday is still answering well enough to ship.

What is AI evaluation, and why does it matter now?

AI evaluation is how a team answers one question: is this thing right, and how would we know if it stopped being right.

The question is new because the failure is new. Software that breaks usually breaks loudly. It throws, it times out, it returns a 500. A language model that's wrong returns a fluent, well formatted, confident answer in the same shape as a correct one. Nothing in your monitoring flags it. The user may not flag it either.

It matters more now because of who's shipping. Most teams putting AI into production this year are product teams who added a feature, and the gap between a working demo and a system anyone will sign off is almost always evidence rather than engineering. That gap is what evaluation closes.

The decisions it has to settle

An eval suite isn't a quality ritual. It exists to let somebody say yes or no to a specific question, with a reason they can defend afterwards. The same questions come up in every engagement.

• Do we ship this? Is it right often enough, on the cases that matter, to put in front of a customer
• Can we take the new model? A provider deprecates yours, or releases something cheaper. Can you move without finding out the hard way
• Is it safe to widen? It works for one use case. Does it hold when you point it at the next one
• Do we need a guardrail here? Which failures are rare enough to live with, and which need a check in front of them

If your evaluation can't answer those, it's producing numbers rather than decisions. That distinction decides how the suite gets built, which is the rest of this guide.

How is an eval different from a benchmark?

A benchmark is generic and external. MMLU, HumanEval, SWE-bench, GAIA. Someone else chose the problems and your users aren't in them.

An eval is specific to your product, your prompts, your documents and your edge cases.

The practical difference is who can write it. Anyone can run a benchmark. Nobody outside your team can write your evals, because nobody else has your traces.

A benchmark tells you which model is strong in general. An eval tells you whether your feature works for the people using it. Both are useful, and only one of them is yours.

A worked example. OnlyFacts' climate agent answers public questions from a live database and cites the query every time. No benchmark on earth measures "did it answer from the database rather than from memory", because that failure only exists in that product. The assertion had to be written by the people who knew the data.

How is evaluating an agent different?

Most writing about evaluation assumes one input and one output. An agent is neither.

A model answers. An agent decides: which tool to call, with what arguments, in what order, when to stop, and what to do when a step fails. By the time it produces an answer it has made a dozen choices you didn't see.

Settle how much autonomy it's supposed to have

One version finds the answer and acts on it. Another finds the answer and stops to ask before doing anything that can't be undone. Neither is wrong. They're different products, and the failure is the one you didn't design for: an agent built to act that keeps asking permission, or an agent built to ask that sends without confirming.

Teams skip this and then can't tell whether a behaviour is a defect or a gap in the specification. The argument that follows can't be settled, because the thing being argued about was never written down.

The unit of evaluation is the path

For a model you judge the output. For an agent you judge the trajectory. An agent can arrive at a correct final answer through a sequence you'd never sign off on: it called an expensive tool nine times, it looked up a record it had no business reading, it got there by luck after two failures it didn't report. An eval that only reads the last message passes all three.

A right answer reached down an unreliable path is a coincidence that held. Change the input slightly and it stops holding. So alongside end to end success, four things get measured that have no equivalent in model evaluation: tool choice, recovery when a step fails, termination, and the cost of the path, because the same answer can cost four calls or forty.

One tool call, four places it can fail

"The agent got it wrong" isn't a finding. A single tool call runs through four stages, each depending on the one before it, and a mistake in one carries into the next. Check only the final output and you know something broke. You don't know where, and the fix is different in every case.

Selection. Did it pick the right tool? Ask a support agent for a leave balance and it searches the policy library instead of the system that holds the balance. Nothing errors. The answer's just wrong.

Arguments. A call can be valid and still wrong. A number sent as text is caught by a schema. A date in a format the tool reads differently, or a filter the user asked for that the agent quietly dropped, is well formed, so the schema lets it through. Those are the ones that get past testing, and this stage is also where unsanitised user input becomes an injection risk.

Execution. The call runs and comes back empty, because the identifier doesn't exist in that system. No exception, no error, nothing in your monitoring. Nothing errored, so nothing was logged, and these are the failures nobody finds until a customer does.

Reading the result. The tool returned the right thing and the answer still came out wrong. It quotes a figure the same document supersedes two paragraphs later, or drops the caveat the result carried. Every sentence it did write checks out, so someone reviewing the answer has nothing to catch.

Stage	Where the fix usually is
Selection	The tool's name and description, not the system prompt
Arguments	A schema the generated call is validated against
Execution	Logging and surfacing the failure instead of swallowing
Reading the result	The prompt

The prompt is one row of four. Knowing which stage failed is what tells you which row is worth the week.

Also worth a line

Tool descriptions are prompts. When an agent keeps reaching for the wrong tool, it's usually two tools whose names sound alike, or a description that assumes context the agent doesn't have. Names, descriptions and parameter schemas are instructions to the model, so they get the same iteration a prompt gets.

Read tools and write tools aren't the same risk. A wrong lookup returns bad data, and you fix the query. A wrong send has already been read by someone. So anything the agent can't undo gets checked before it runs, which is where guardrails belong before an eval does.

Score the stages, not only the end. One overall number can't tell you what regressed. Score each stage separately and the number points at the stage, and it catches one stage getting worse while the end to end figure holds because something downstream is making up for it.

What this means for how you log

If you only log the final response, none of the above can be built later. Capture the whole trajectory from the beginning, including the calls that failed, the arguments they were called with, and what came back. It's a week one decision you live with for the life of the system, and if you adopt one tool before you have any evals, adopt the one that captures traces, because the reading is the work. Agent analytics in production covers what that capture looks like once the system is live.

What should an eval actually measure?

There's no universal list. There's a list for your system, and you find it by looking at what your system got wrong. Five dimensions recur in almost every production system we've built. The sixth decides whether the system is any good.

• Groundedness. Did the answer come from the source, or did the model recall it? In regulated work this is the one that matters. A return figure produced from memory rather than the factsheet is a licence problem, not a small error.
• Accuracy against a known answer. For questions with one right answer, does the system give it? The easiest to measure, and the easiest to over-index on, because many questions worth asking don't have one right answer.
• Task completion. For an agent, did the job get done? A support agent that holds a polite conversation and logs the wrong ticket type has failed, and every text-level metric will say it did well.
• Safety and refusal. Does it decline what it should decline, however the question is phrased?
• Cost and latency. A system that's right and takes nine seconds has failed a phone caller. These belong in the eval set, not in a separate performance ticket.

The sixth is whatever is specific to your task

The one that decides whether your system is worth having is usually not on anybody's list, including ours. Real examples, each from a system we built:

• Did the agent route to the right specialist team on the first pass, out of eight
• Did the answer come from an approved document, rather than from any document that happened to match
• Can a reader work out which individual contributed a given piece of advice
• Did the summary need correcting before the person acting on it could use it

None of those appear in an evaluation framework's defaults. Every one of them was the thing the client actually cared about.

It's also the dimension that's hardest to buy. Nobody can sell you "did it route to the right team", because that metric didn't exist until your system did. And the defaults can miss the second example entirely: faithfulness scores the answer against whatever was retrieved, so an answer faithful to a superseded document scores 1.0 and the dashboard stays green.

If your eval set contains only the five dimensions above, you've bought someone else's checklist. The measure that matters most is usually the one you had to name yourself.

How do you build your first eval set?

The order matters, and most teams do it backwards by starting with a tool.

1. Get traces

If the system is live, sample production traces. Cover the range of what people ask, and oversample what went badly: a thumbs down, the same request rephrased twice, a user who gave up and asked a human.

If it isn't live yet you generate them, and this is where teams go wrong. Asking a model for fifty user questions returns fifty versions of one question. Pick two or three axes along which real requests differ, the task, who's asking, how clearly they put it, build the combinations, and write one query per combination. The combination is what forces variety.

Write twenty by hand before scaling up, because writing them is how you learn the problem. Then, before running them, write down what you expect each query to do and where you think it will fail. The gap between that and what happens is where your assumptions were. Synthetic inputs are fine. Synthetic judgement isn't, which is step five.

2. Read them and name the failures

Read each trace and write a short note on what went wrong, in your own words. Don't start from a list of categories borrowed from somewhere else. Note the first failure, not every symptom. Same as a failing test run: one thing goes wrong upstream and the rest of the trace is consequence.

Then group the notes. The retrieval missed. The model guessed. The question was ambiguous and we picked the wrong reading. What comes out is a taxonomy, a named vocabulary for how this system fails, shared by everyone working on it. Everything you measure later sits on top of it, so don't freeze it early. This step is manual and it's the one that can't be skipped.

3. Decide which of them deserve an evaluator

Split the list in two. Some failures happen because the instruction was ambiguous or incomplete: nobody told the agent the summary had to carry the policy number. Those are specification problems, and a prompt edit fixes them. Others happen when the instruction was clear and the system still didn't follow it. Those are the ones worth an evaluator.

Fix the ambiguity first, then measure what survives it. An eval written for a failure a prompt edit removes is work done twice, and worse, build evaluators against vague instructions and you're measuring how well the model guesses what you meant.

The same split applies to the labels. In a routing run we did in October, 293 scripted helpdesk conversations across four models, six were marked wrong for all four, and in all six the models chose the same department: the second of two matters the employee had raised. When four different models agree against the label, the label is the thing to re-read. Until the helpdesk owner sets the rule, no model can be scored on it.

4. Turn each named failure into an assertion

Now the tooling earns its place. Each failure mode becomes something that produces a number on every run: a code assertion, a model-as-judge prompt, or a human review rubric.

5. Have the right people mark it

The step teams get wrong most often, and the cheapest to get right. The person who knows whether an answer is correct is usually not the engineer. For a fund assistant it's someone who knows the fund. For a support agent it's the specialist who picks up the ticket.

This is the rule acceptance criteria have always run on. Whoever built the thing isn't the one who decides whether it's right, because they'll write down what it does. Evals that describe the system pass by construction. The ones that describe what it should do are the ones that fail, and failing is what makes them worth running. If the annotation tool is too awkward for the expert to use, that's a tool to fix rather than a reason to give the job to someone else.

How are evals actually run?

A production system usually needs all four of the methods below. They differ in what they can judge and what they cost.

Code assertions

A deterministic check written in code. Is the citation present. Does the figure match the database. Is the JSON valid. Did the agent call the tool with the right arguments. Fast, free, and the same answer every run. If a check can be code, it should be code.

Many of these need no correct answer to compare against. That a generated query carries the filter the user asked for, or that every tool name in a call exists in the registry, is a check against a rule rather than a reference. A known correct answer is the expensive part of an eval set, because a person has to produce it. Where a rule can stand in for one, the check costs almost nothing and can run on live traffic rather than on a fixed set.

A model judging against a rubric

A second model scores the output against a written standard, commonly called LLM as judge. It's how teams evaluate what code can't reach: was the answer faithful to the source, was the tone right, did it refuse when it should have.

A judge needs its own evaluation before you trust it. Score a set of outputs with the judge, have a human score the same set, and measure how often they agree. If the agreement is poor the judge isn't measuring what you think, and tightening the rubric is the fix. Teams routinely skip this and then trust a number that was never checked.

Human review

A person marks the output, usually against the same rubric the judge uses. Expensive and irreplaceable. Human marks are the ground truth that validates everything else, and they're where new failure modes get discovered. The volume needed is lower than teams expect, because the job is to calibrate rather than to cover.

Ask for a yes or no, not a rating. Did this failure occur is reproducible. Rate the helpfulness from one to five isn't, unless the rubric is unusually tight: the number drifts between people and between days, and it hides which thing actually went wrong.

Online evaluation on live traffic

Everything above runs offline, against a fixed set. Online evaluation measures the system as real users hit it: sampling production traces for review, tracking whether the user rephrased or escalated, and A/B testing a change against the current behaviour. Offline tells you whether you broke something you already knew about. Online tells you what you didn't know to check.

Which to use where

Method	Judges	Cost	Runs
Code assertions	Anything mechanical	Nothing	Every change
Model as judge	Quality, faithfulness, tone	Low	Every change
Human review	Everything, and the truth	High	Weekly sample
Online	What you didn't anticipate	Medium	Continuous

A reasonable starting shape: code for everything that can be code, a validated judge for the rest, a human reading a sample each week, and online measurement once there's traffic to read.

Where does evaluation sit in the pipeline?

In the same place tests do, and for the same reason. We've written the test first since 2012, and the eval suite is the same discipline pointed at a different artefact, with one honest difference: a test is written before the code and can fail the first version, while an eval suite is written from traces after a first version exists, so it gates change rather than creation.

Evals run in CI, not in a notebook. If the suite only runs when someone remembers, it's documentation.

A failing eval blocks the release, exactly as a failing test does. Otherwise it's a dashboard.

The suite is version controlled with the prompts. A prompt change and the eval that covers it belong in the same commit.

There's a consequence that takes longer to feel. An eval suite is the only artefact in an AI system that doesn't go stale when the model changes. Prompts get rewritten, models get deprecated, retrieval gets rebuilt, and the set of questions your users actually ask, with the judgement about which answers were right, survives all of it.

That's why we treat the suite as the deliverable rather than a by-product. The model is rented. The evaluation is owned. It's also why an AI-native product engineering engagement starts by agreeing the eval baseline in discovery, before the first build cycle, so there's something to ship against from the first release.

What does this look like in production?

What evaluation meant on three systems we built.

An investor assistant under a financial services licence

India Avenue's assistant answers questions on three funds from roughly 200 documents. Accuracy was poor at first and we spent longer on the prompt than we should have. The prompt wasn't the problem. Much of the data sat inside images of charts and tables, so retrieval couldn't read it and the agent answered from whatever words sat nearby. No tool surfaced that. We found it reading traces one at a time.

Their own team then sent fifty real investor questions and marked every answer right or wrong, with the reason. Those fifty are now the suite. Every change runs against them first, which is what lets the firm adopt a newer model and still show the answers didn't change.

A voice agent on a support line

Talent Carriage's agent answers the HR support line and writes the ticket while the caller is still talking. The measure isn't how many calls it handles. It's whether the specialist who picks up the ticket has to fix it. On the client's own dashboard on 10 September, 40 of the first 41 tickets needed no correction, an early and deliberately small picture. The specialist flags a bad summary in one field while resolving the ticket, and that field is the accuracy number. The people doing the work are the eval.

One more decision on that system is worth copying. The model is a setting, not an architecture. We tested cheaper and faster models and rejected them, because they couldn't reliably work out the right next question. Model choice is a configuration value, so that decision can be revisited the week a better option lands, and the eval suite is what says whether the swap was safe.

The third is Life Insurance Direct's call quality system, which reads every sales call against 26 checks where the team had reached 2 to 5 per cent by hand, and reports nothing on its own. Everything it finds is ranked for a person to open. A system that acts on its own findings needs to be right; one that ranks them for a human needs to be useful, which is a lower and more honest bar.

What do most teams get wrong?

Buying the tool first. The tool runs assertions. It can't tell you what to assert, and that's the work.

Writing the evals themselves. Engineers write evals that test what the system does rather than what it should do. Those pass by construction.

Measuring once. An eval run at launch is a launch report. An eval run on every change is a gate.

Borrowing someone else's failure categories. Hallucination, instruction following, verbosity. These are the labels in the papers and the ones the tools ship with. They sound reasonable, and not one of them will tell you the agent offered a meeting slot the calendar had already blocked.

Treating a score as the output. A single number hides which failure mode got worse. Keep them named and keep them separate.

Skipping it because the demo worked. You chose the question and you'd already seen it work. Nobody using it in production will do either.

How often should evals run?

On every change that could affect behaviour. In practice that means four triggers.

Every prompt change, including the ones that look cosmetic. Reordering instructions changes outputs.

Every model change, including minor version bumps from the provider. This is the trigger teams forget, because it happens without a commit on their side.

Every retrieval change. New documents, a different chunking strategy, a reranker swap.

On a schedule, against live traffic. The offline suite tests what you knew about when you wrote it. A weekly sample of real production traces is how you find the failure modes that arrived since. The continuous improvement loop is built around that sample.

The first three are cheap and automatable. The fourth needs a person, and it's the one that keeps the suite from going stale.

Where to start

If you have an AI system in production and no way to tell whether it's still right, start with the traces. Read a hundred of them with someone who knows the domain, name what went wrong in each, and count. The suite comes out of that, and nothing you buy will do it for you.

If you're building the system now, agree the eval baseline before the first cycle. That's where our product engineering engagements start, and it's the part most teams would rather leave until there's something to look at. By then the week one logging decision has already been made.