How to Test an AI Agent Before Go Live for Business Process

An AI agent is a program that reads a message, checks your systems and acts, for example by placing an order. If you sell through dealers and distributors, orders can arrive on WhatsApp at all hours. Say a dealer messages your sales desk late at night asking for a large quantity of one fabric. The agent replies at once and places the order. The stock check was never run, and the warehouse holds less than the dealer asked for. The agent did not fail to reply. It replied with confidence about something it had not checked.

How do you test an AI agent before go live on dealer orders?

To test an AI agent before go live, run it through a fixed set of realistic scenarios, check each action it would take in your systems, and agree in writing what a pass looks like. Then let your own team try it before customers do, and name the person who signs off.

Each stage catches a different kind of failure:

What should a manufacturer's test scenarios cover?

A scenario is one realistic message or request, with the result you expect. Build your list from the last three months of real work, not from the cases that sound clever. Cover four kinds.

1. The ordinary cases

A normal dealer order for a product in stock, from a known customer. Most of the work sits here, so most scenarios should too. These are the cases where a small error still costs a customer's trust.

2. The shortfalls

Quantity above stock, stock spread across two warehouses, or an item that arrives later. The agent should say what it can do now and what comes later. It should never invent a delivery date.

3. The odd requests

Two items in one message, a quantity written in the wrong unit, a customer who changes their mind halfway, or a message with no item name at all. Expect the agent to ask one clear question, not three in one reply.

4. The wrong person

A message from a number that is not on any account, or from someone who is not allowed to order. The agent must stop and pass the request to a named person.

The textile order agent's evaluation data held 147 scenarios as of October 2026. The count matters less than whether each one has a written expected result.

Why must each order test start from a clean state?

A confirmed order reserves stock, so the next test sees less stock than it should. An order left behind in the test copy also changes the test customer's balance, so the next credit check reads the wrong figure. Good test scripts avoid both problems in the same way.

• One case, one item. Give each case its own item, so no case changes the numbers another case depends on.
• Reset the test customer. Put the credit limit, payment terms and balance back to their starting values before each run.
• Clear what you created. Delete or close every test order before the next case starts.

Write these steps into the script next to each case. Testers who skip the cleanup will not notice until a later test fails for no clear reason.

Which tests must run in a copy of your order system?

Any test that places, changes or reserves an order must run in a copy of your business system. Your business system is the software that holds stock, customers and orders, often called an ERP. Before the first test, check that the agent is connected to the copy. A setting that points at the live system turns every test into a live test.

Two rules keep this safe. First, use test customer accounts that exist only in the copy. Second, mark every real customer account in the script as off limits for writes, so no one uses one by accident. A short warning next to the account name costs nothing and prevents an expensive mistake.

How do you know the order agent is ready?

Set the pass rule before you test, not after. Three checks cover most cases:

1. Right answer. The price, quantity and date match what your team would say.
2. Right action. The agent checked the system that owns the fact before it spoke. A reply that quotes stock without a stock check is a failure, even if the number happens to be right.
3. Right hand-over. When a request falls outside the rules, the agent stops and passes it to a named person.

A confident answer with no check behind it is a failed test.

In two turns of one test run, the agent replied without checking any system at all. It confirmed a request that nothing had verified. Engineers call each check a tool call. We decided to write a rule that nothing is confirmed until the tool that owns the fact has run in the same turn. We also decided to count tool calls in every conversation log, so a reply with no calls can be found with a search.

Count the failures on must-pass cases. One wrong answer where a wrong answer costs money means you fix the cause and test again. Do not average it away across the other passing results.

Who in the factory should test it, and when is it not ready?

Your own people should try the agent. The people who take orders today know the habits that no document lists: the dealer who writes in shorthand, the item known by two names, the customer who always asks for a discount in the same breath. Ask them to send the messages they would really send, not the tidy ones you wrote.

Treat the agent as not ready if any of these is true:

• a must-pass case fails and nobody has fixed the cause;
• an action that changes a record has no test in the copy;
• nobody can say who takes over when the agent stops.

Waiting costs days. A wrong order costs more, and it arrives in front of a customer.

This full set of steps matters once the agent acts on its own. If the agent only drafts replies that a person reads and sends, you can start with a shorter list and widen it as trust builds. For more on which actions need a person's approval, read AI agent human approval: which actions require it.

What we learned testing an order agent in Avesta AgentOS

We built an order agent for a textile manufacturer in India on Avesta AgentOS. It takes orders from dealers on WhatsApp and email, checks live stock, and places orders in the client's business system. It has been live since September 2026, and a person still handles every exception. From start to production took 5 to 6 weeks (as of October 2026). Three patterns from that work apply to any manufacturer.

Test data is part of the product. Our test documents set out each case as one message, with the setup it needs, the result to expect and the cleanup afterwards. Keeping the cleanup beside each case stops a test customer's balance carrying over from one run to the next unnoticed.

A wrong environment looks like a wrong agent. The first step in our test documents is to check which copy the agent points at. If it points at the wrong copy, none of the figures in the script match, which can make a sound agent look broken. Check the connection before anything else.

A reply with no check is a defect, even when it sounds right. The zero-tool replies described above are why we decided to count tool calls. A tester reading the transcript sees a fluent answer. The log shows that nothing was checked.

The bottom line

A test is only as good as its written pass rule and its clean test data. Build the scenario list from real work, test every change in a copy, check that the agent asks the owning system before it answers, and let your own team try it first. Then name the person who says go. Our Safe AgentOps page describes how we test, guard and monitor AI agents, and our manufacturing AI agent page sets out our offer for manufacturers. For background on what evaluation means, see what is AI evaluation?

Anthropic's guide to building evaluations advises designing tests that mirror your real tasks and include edge cases (Anthropic: define success criteria and build evaluations). OpenAI's agent evaluation guide describes a trace as the record of one run, including its model calls and tool calls, which is how you spot a reply made with no tool calls (OpenAI: evaluate agent workflows).

Which scenario would hurt your business most if the agent got it wrong?

---

Before your agent takes its first real order, test it against the cases that would cost you the most.

Get started: hello@avestalabs.ai