Prompt Regression Testing: Evals Before Every Release

Say you add one line to a support agent's instructions: "keep answers short". It works. Answers get shorter. A week later a customer complains that the agent told them the wrong refund deadline.

The short version had dropped the deadline. Nobody saw it, because nobody re-ran last month's questions after the edit. The change fixed one thing and quietly broke another.

The fix is to re-run the saved tests on every change, before it goes live, and to know when a failed test should stop the release. That habit has a name, and it works for AI agents too, as long as their tests can't reach real systems.

What is prompt regression testing?

Prompt regression testing means re-running a fixed set of saved tests every time you change an AI system's prompt, model, documents or tools. You then compare each result with the last release. A regression is something that used to work and now doesn't. If one appears, the change doesn't go live until someone fixes it or accepts it.

Put simply:

Every change re-runs yesterday's tests. A test that used to pass and now fails stops the release.

An eval is one of those saved tests: an input, the answer you expect, and a way to mark the reply. OpenAI's evaluation guide recommends "continuous evaluation (CE) to run evals on every change". Prompt regression testing is that idea, aimed at one question: did this change break anything?

Which changes need a prompt regression test?

More than prompt edits. Anthropic's engineers describe automated evals "running on each agent change and model upgrade" (Anthropic). For an AI agent, a change is anything that alters what it reads or does.

The last row is easy to miss. Anthropic's model versioning page says each model ID stays fixed. It also says infrastructure updates can "produce minor differences in observable behavior" even when the ID hasn't changed. So run the regression set on a schedule too, not only when you ship.

What goes in the regression test set?

Each test has three parts: an input, the expected answer, and how to mark the reply. For an agent, add the tools it should call. Teams often call this set a golden dataset: tests whose expected answers an expert has checked.

Three rules keep the set useful:

• Real inputs first. OpenAI's guide says to "mine your logs for good eval cases". Real questions catch real mistakes.
• Exact checks where you can. A fee, a date or a required link can be checked by a written rule. Anthropic recommends "deterministic graders where possible". Use a judge model only for answers that need reading, and calibrate the judge first.
• Every failure becomes a test. When a mistake turns up in production, add it. Next time it is caught before release.

Who should write the questions and mark them is its own topic. Our guide on how to test a RAG chatbot covers it.

When should a regression test block a release?

When a test that passed last release fails now, and fails again when re-run. The re-run matters. OpenAI's guide notes that "models sometimes produce different output from the same input". So one failed run can be bad luck rather than a real regression.

Anthropic's article draws a useful line between two kinds of test set. Regression evals ask whether the agent still handles what it used to, and "should have a nearly 100% pass rate". Capability evals test new goals and start low. Our rule follows from that: only the regression set blocks a release.

Compare test by test, not only the average. A pass rate can stay flat while one important test flips from pass to fail and another flips the other way. The swap is what you need to see.

A pass rate tells you how much changed. A test-by-test comparison tells you what broke.

How do you run prompt regression tests in CI/CD?

Put the run inside your release pipeline (CI/CD, the automated steps every change goes through before it goes live). Then nobody has to remember to run it.

1. You make a change to a prompt, model, document or tool.
2. The pipeline runs every regression test, with stand-in replies for the agent's tools (next section).
3. It compares each test with the last release. If every test still passes, the change is ready to release.
4. It re-runs any test that failed. If it passes now, the change can ship, with that test flagged as varying.
5. A test that fails again blocks the release.
6. A person reads the failures. They fix the change, or accept the new answer on purpose and update the expected one.

Step 6 is a person, not a rule. Sometimes the new answer is better. Only someone who knows the work can say so, and the updated expected answer becomes the next baseline (the results every later change is compared with).

How we make agent regression tests repeatable in Avesta AgentOS

Evaluation is built into Avesta AgentOS. It runs the tests and marks them. The comparison with the last release is a small step you add around it: keep each run's results, and compare pass or fail per test with the last release's run. Here are the patterns that make the runs safe and repeatable, and that you can copy.

Every test gets a mark and a tool check

Each test is an input with its expected answer, and optionally the tools the agent should call. A run marks each test pass or fail. A written check marks exact answers; a judge model marks the ones that need reading and writes its reason. The run also reports whether the agent used the expected tools, names any it missed, and gives the pass rate. The judge's reasons are what make a failed test quick to read.

Tool calls get stand-in replies during a test

An agent's test can call real tools. Re-run that test on every change and it might send a real message to a customer or update a real order each time.

So Avesta AgentOS can answer each tool call in a test from a file of saved stand-in replies. The replies are written for each test conversation and matched on the call's details. If no saved reply matches, the call is refused and the test shows it. A short list of tools that only read data or work inside the test run itself still run for real.

This is switched on per run: the test run says it is a test and points to the replies file. Without that, tools run as normal. So any agent whose tools act on real systems should be tested this way.

1. The agent under test proposes tool calls, just as it would live.
2. Every call meets the stand-in replies first. A matched call gets its saved reply. An unmatched call is refused.
3. Real systems are kept out of reach. Only tools that read data or work inside the test run itself run for real.

This gives two things. A test is built so its tools can't reach real customers or records. And the stand-in tools give the same replies every run, so a changed answer points at your change, not at today's data.

The client's questions stay in the set

India Avenue's investor support assistant passed 50 real questions supplied and marked by India Avenue. Every change still runs against them (as of 21 Oct 2026).

When is prompt regression testing more than you need?

• A prototype nobody relies on yet. Write tests once real users arrive and their questions can become the set.
• Output a person checks every time. If someone reviews every reply before it goes out, a light test set is enough.
• Subjective tests marked by an unchecked judge. Don't let them block a release until you have calibrated the judge against an expert. Report them instead.
• Tests that flip often. A test that fails on every second run trains people to ignore failures. Fix it or move it out of the regression set.

The bottom line

The pipeline can find the test that broke. It can't tell you whether the new answer is worse or just different. That call belongs to a person who knows the work, and their decision becomes the next baseline.

For where this fits among the other checks on an AI system, read our guide to what AI evaluation is.

Which of your agent's last five changes went live without its old tests being re-run?

---

Want regression tests running on every change to your AI agents? See how Safe AgentOps runs evals, guardrails and monitoring.

Get started: hello@avestalabs.ai