The AI Demo Worked. Production Didn't. Here's Why

AI demos work because they run in ideal conditions: clean inputs, a builder who knows what to ask, one user, and someone watching every answer. Production removes all of that at once. Real users bring messy data and unexpected questions, tools fail mid-task, costs and delays add up, and nobody is watching each answer. A demo proves the AI can do the job. Production needs the engineering that makes it do the job reliably.

We've learned this running Avesta AgentOS for clients in production, once the hard way. This article explains why demos mislead, the six ways AI breaks with real users, what production needs, and a checklist to tell how far your AI prototype is from production.

Why do AI demos work so well?

A demo isn't dishonest. It's just built under conditions that production never offers:

• Curated inputs. The builder uses examples they know the system handles.
• One expert user. The person driving the demo knows how to phrase things and avoids the edges.
• The happy path. Every tool responds, every document is clean, nothing times out.
• Nobody counts cost. Ten requests cost nothing. Ten thousand a day is a budget line.
• A human checks every answer. If the AI gets something wrong, the builder notices and moves on.

AI makes this gap wider than in ordinary software, because AI demos are now very easy to build. An afternoon with a model API produces something that looks finished. That's useful for proving an idea. It's misleading as a measure of how close you are to launch.

A demo proves the AI can do the job. Production needs the engineering that makes it do the job reliably.

Why does AI break in production?

1. Real inputs are messy

Demos see clean PDFs and well-formed questions. Production sees scanned documents, photos of forms, tables split across pages, typos, mixed languages and requests the team never imagined.

We learned this on a logistics project: extracting vehicle identification numbers (VINs) from bills of lading. Our demo used scanned documents in good condition, and it worked perfectly. We built the full solution and handed it over for the customer's final testing. Our evals didn't include the poor-quality scans the customer actually receives from its vendors. One of their team tested it with exactly that kind of document. The solution returned the wrong VIN, and reported it with high confidence.

That was a bad moment for us. We had to rethink the extraction approach and rework the solution architecture, which pushed delivery back by two to three weeks. We've written up how we rebuilt the VIN extraction in a separate article. The bigger change was in how we work: since then, real edge cases go into the eval set from the first day, demo included.

2. The AI is confidently wrong

Models don't fail loudly. They produce fluent, plausible answers that are sometimes wrong. In a demo, the builder catches it. In production, the customer acts on it.

That's what made our bill of lading failure dangerous. A wrong answer flagged as uncertain gets checked. A wrong answer delivered with high confidence goes straight into the customer's system.

The best-known example is Air Canada. In 2024, a Canadian tribunal held the airline liable after its website chatbot gave a customer wrong information about bereavement fares, and rejected the argument that the chatbot was responsible for its own answers. Whatever technology sat behind that chatbot, the lesson holds: your customers will treat what your AI says as what your company says.

3. Tools and steps fail, and errors compound

Agents don't just answer. They call APIs, query databases and hand work between steps. In production, APIs time out, rate limits hit, and a step half-completes.

Multi-step work also multiplies risk. As an illustration: if each step in a ten-step task is right 95% of the time, the whole task is right only about 60% of the time (0.95 to the power of 10). Demos rarely run long enough for that to show.

This is why orchestration in AgentOS is built around detecting a failed step, retrying, recovering from where it stopped, and escalating to a person when it can't.

95% right per step. 60% right per task.

4. Cost and latency add up

A demo answers one question in a few seconds and costs a fraction of a cent. Production runs thousands of multi-step tasks a day, each calling the model several times. Costs that nobody modelled appear on the first invoice, and the answer that took eight seconds in a demo feels broken to a customer waiting in a chat window. Much of that cost is avoidable with better prompt and model choices.

5. Users don't behave like the builder

Real users ask things you didn't plan for, phrase the same request ten ways, paste in whole email threads, and give up halfway. They also bring their own habits and worries: if they don't trust the answer, they double-check everything or stop using it. None of this shows up when the person driving the demo is the person who built it. Designing for those habits and worries is a product skill, not just an engineering one.

6. Changes break things silently

A prompt tweak fixes one case and breaks three others. A model provider updates a model and behaviour shifts. In ordinary software, tests catch regressions. In AI products, without evals, nobody notices until users complain.

What does the data say?

The pattern shows up across the industry. LangChain's State of Agent Engineering report surveyed 1,340 practitioners in late 2025. LangChain sells observability and eval tools, so read it as an industry survey rather than independent research. It found:

• 57% of respondents now have agents in production, up from 51% the year before.
• Quality is the biggest barrier to production, cited by a third of respondents. That covers accuracy, consistency, tone and following policy. Latency came second, at 20%.
• Organisations with more than 10,000 employees named hallucinations and inconsistent outputs as their biggest quality problem.
• Almost 89% have some form of observability for their agents, but only about 52% run offline evaluations on test sets.

That last gap matters most. Most teams can see what their agents did. Far fewer can tell whether it was right.

RAND's 2024 study of AI project failures found a similar pattern earlier: teams that move quickly from prototype to prototype, without investing in infrastructure, often can't see failures once a model is deployed.

What does production actually need?

The gap between demo and production is a set of engineering components. A demo has almost none of them.

Seeing is not the same as knowing

Tracing tells you what happened in one run. Evals tell you whether a change made things better or worse. Agent behaviour analytics tells you how agents perform across thousands of real conversations: how often they get it right, where users had to retry, and which topics they're failing on.

Production needs all three. The survey numbers above suggest most teams stop at the first.

Tracing shows what an agent did. Evals and agent behaviour analytics show whether it was right.

Should you rebuild the prototype or harden it?

Usually, harden it. The prototype holds valuable learning: which prompts work, which documents matter, what users actually ask. Throw that away and you'll relearn it.

Rebuild only the parts that can't carry production load: a single giant prompt that can't be tested in pieces, no separation between steps, or data handling that can't meet your security requirements. Keep the rest and add the production layer around it. In our bill of lading case, the extraction approach itself couldn't handle poor scans, so we redesigned that part and kept everything around it.

Production-readiness checklist

Your AI is ready for real users when all of these are true:

• Evals: a set of real cases with expected outcomes, run on every change.
• Messy inputs: real-world documents and questions are in the eval set, not just clean samples.
• Failure handling: every agent step can fail, retry and recover, and escalates to a person when it can't.
• Grounding: answers come from your own data, with sources users can check.
• Access control: agents can only see and do what each user is allowed to.
• Approvals: high-risk actions need human approval.
• Tracing: every run is recorded, with inputs, decisions, tool calls and cost.
• Agent behaviour analytics: answer quality, retries and topics are visible across real use.
• Cost and speed: cost per task and response time are measured, with limits.
• Rollback: prompts and models are versioned, and you can roll back a change quickly.
• Ownership: someone owns what happens when an agent gets it wrong.
• Limited release: a small group of real users has used it before full launch.

If fewer than half are true, you have a demo, not a product. That's normal, and fixable.

How long does it take to move an AI prototype to production?

It depends on how many of the checklist items are missing and how risky the workflow is. Most of the time usually goes into the parts a demo skips, starting with evals.

That's why we changed the order. At Avesta, every demo now starts with writing evals, using real and messy cases from the customer, before the agent is built. AgentOS has eval generation built in, so this doesn't slow the demo down. The demo is production-grade from the start, and moving one workflow from demo to production typically takes us two to four weeks, including testing and integration.

If you're starting from a demo built without evals, budget for:

• Building the eval set from real cases, including the messy ones, which takes longer than teams expect. Techniques like LLM as a judge help you score those cases at scale.
• Integration with the systems agents need to read from and act in.
• Security and access control review.
• A limited release to real users, with time to fix what they find.
• Ongoing model and infrastructure costs, measured per task.

The cost of skipping this is higher. A launch that breaks in front of customers is expensive to recover from, and it makes the next AI project harder to fund.

Where to start

Take your demo and run it against twenty real cases you didn't pick. The failures will show you which of the six problems you have.

If you want help closing the gap, that's what our discovery sprint is for. Avesta's AI-native product engineering team has already built the production layer in AgentOS, so we can assess your demo against it and give you a plan to launch. Book a discovery call to start.

For the bigger picture of how this fits into building AI products, read What Is AI-Native Product Engineering?

Sources

• LangChain, State of Agent Engineering, 2026 (survey of 1,340 respondents, 18 November to 2 December 2025)
• ABA Business Law Today, BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot, February 2024 (on Moffatt v. Air Canada, 2024 BCCRT 149)
• James Ryseff, Brandon De Bruhl and Sydne J. Newberry, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, RAND Corporation, August 13, 2024