How to Build Dependable AI Systems Around Imperfect Models

The pilot was 90% accurate. Everyone was pleased. Then someone asked the question nobody could answer: what happens with the other 10%?

Australia's AI Reference Architecture (AIRA) is built around that question. It assumes AI models will sometimes be wrong, and asks you to build systems that stay dependable anyway. This fifth post in our AIRA series covers how.

Why does AIRA focus on systems, not models?

AIRA focuses on systems because no AI model is right every time, and better training only reduces errors rather than removing them. It acknowledges techniques such as adversarial training and ensembles, but it says dependability should not rest on the model alone. The system around the model should catch, contain and recover from its mistakes.

AIRA lists the patterns it expects:

• Confidence and abstention handling. The system recognises when it shouldn't proceed.
• Fallback pathways. There is another route when the AI path is unavailable, uncertain or out of bounds.
• Input and output validation. Checks run before the model runs and after it answers.
• Redundancy and hybrid approaches. AI is combined with deterministic rules, or with a second check.
• Arbitration. A clear rule decides what happens when two sources disagree.

This sits in AIRA's "validity and reliability" domain. Validity means the system does the job it is meant to do. Reliability means it keeps doing that job under stated conditions over time.

It also changes the question leaders should ask. Instead of "how accurate is the model?", ask "for every important output, how will we know when it is wrong?"

Don't ask how accurate the model is. Ask how you'll know when it's wrong.

How do you evaluate an AI system before deployment?

You evaluate an AI system by testing it against real examples from your own work, marked by people who know what a right answer looks like, before anyone relies on it. AIRA treats this as part of the architecture. Working out how a system will be evaluated, including whether meaningful evaluation is possible at all, comes before choosing the technology.

A sound pre-deployment evaluation has four parts:

1. Real cases, including the messy ones. Use actual requests and documents from your operation: blurry scans, ambiguous questions, unusual layouts. A demo built on clean examples tells you little about production.
2. The right people decide what is correct. That usually means the domain experts who do the work today, not the engineers who built the system. People who built it tend to test what it does, not what it should do.
3. Named failure modes. Read the results, name the ways the system goes wrong, and turn each into a check that produces a number.
4. A release gate. If the evaluation fails, the system doesn't ship, just as a failing test blocks ordinary software.

How many examples? Fewer than most teams expect. Fifty real questions marked by experts usually teach you more than a thousand made-up ones. Our guide to AI evaluation covers the full method.

A lesson from our own work. On a document extraction project, our demo used clear scans, and our first evaluation set left out the poor-quality scans some senders actually used. Customer testing found the gap. The model returned a wrong vehicle ID with full confidence, and the redesign pushed delivery back by two to three weeks. We've written up why that demo broke in production. Now every engagement starts by building evaluations from the customer's real, messy cases.

Fifty real cases marked by experts beat a thousand made-up ones.

What is a human baseline, and why does it matter?

A human baseline is a measure of how well the current process performs today, whether people do it alone or with existing tools. The DTA's AI technical standard expects AI systems to be compared against existing human and hybrid baselines. The useful question is not "is the AI perfect?" but "is it better than what we do now, and where is it worse?"

Without a baseline, two things go wrong:

• Good systems get rejected because they make some mistakes, even though the current process makes more.
• Weak systems get approved because a high accuracy number sounds impressive, even though it is lower than the people it replaces.

A baseline doesn't have to be accuracy alone. Coverage, speed and consistency count too. In Life Insurance Direct's call quality system, the team had been able to review only 2 to 5% of sales calls by hand. The AI reads every call against 26 checks and ranks what it finds for a person to review. The baseline that mattered was coverage, not whether the AI could out-judge an experienced reviewer.

Three tips for setting a baseline:

• Measure the current process on the same cases you use to evaluate the AI.
• Record where people disagree with each other. If experts disagree, the AI can't be scored against a single right answer there.
• Compare the hybrid too. AI plus a human reviewer often beats either one alone.

How strong the evidence needs to be depends on the use case's risk tier. Our post on applying AIRA by risk shows what changes at each tier.

What should an AI do when it isn't sure?

When an AI system can't confirm its answer, it should stop rather than guess. That can mean abstaining, taking a fallback route, or handing the case to a person. AIRA calls this abstention: not proceeding when predefined conditions are met, such as low confidence, missing inputs or a request outside the system's authorised scope.

The hard part is knowing when the AI is unsure. Don't ask the model. On scanned shipping documents, the language models we tested reported 100% confidence even on vehicle IDs they had misread. A model's confidence often tells you nothing about whether it is right.

The model said it was 100% sure. It was wrong. Use a check that sits outside the model.

Use a check outside the model instead. From most to least reliable:

1. Look it up. If the value already exists in your records, compare against it.
2. Use rules built into the data. Totals must add up, counts must match, and many IDs carry a check digit. On that shipping project, a 17-character rule plus a check-digit test meant only 5 to 7 documents in 100 needed a person. Our write-up on check digits versus confidence scores has the detail.
3. Ask two models and compare. If they disagree, a person decides. This costs more and sends more cases for review.
4. Use a separate judging model. It returns a probability. Test it on your own cases before trusting it. Our guide to LLM as a judge covers how.

Then make the routing rule explicit:

• The check passes: the work goes through.
• The check fails or can't run: the system abstains, takes a fallback route, or routes to a person with the evidence alongside.

AIRA's definition notes that, for generative AI, abstention can be as simple as declining to answer. Designing an AI to say "I don't know" is a feature, not a failure. We've written about how to make AI agents say "I don't know".

How do input and output validation reduce errors?

Input validation stops bad or unsafe requests before the AI acts on them. Output validation checks the AI's answer before anyone relies on it. Together they catch many errors without needing the model to be smarter, and AIRA lists both under validity, reliability and safety.

Four practical rules:

• If a check can be code, make it code. It is fast, free and gives the same answer every time.
• Validate tool calls, not just text. For an agent, a well-formed call can still be wrong: a dropped filter, a date in the wrong format, the wrong record.
• Count what should be there. A check can tell you a value is wrong, but not that it is missing. If a document lists six vehicles, there should be six IDs.
• Treat inputs as untrusted. Documents, emails and web pages can carry hidden instructions. Validation is part of your defence against prompt injection.

How do you keep AI reliable after go-live?

You keep AI reliable by re-running your evaluations whenever something changes, and by watching live behaviour for problems your evaluations didn't anticipate. The DTA's AI technical standard expects evaluation before deployment and throughout the system's life, and AIRA's observability domain is how you spot drift.

Re-run the evaluation suite on four triggers:

1. Every prompt or instruction change, including ones that look cosmetic.
2. Every model change, including provider updates you didn't start.
3. Every change to knowledge or retrieval: new documents, new sources, new search settings.
4. On a schedule, against live traffic. A weekly sample of real cases reviewed by a person finds failures your fixed test set doesn't cover.

Alongside that, monitor the signals that show drift: error and escalation rates, how often reviewers correct outputs, how often users rephrase or give up, and changes in the types of cases arriving. Our guide to reconstructing an AI decision covers what an AI observability dashboard should show.

Check the checker. Even good automatic checks can let errors through. A check digit catches most misreads, but not all. Every week, have a person review a small sample of the cases that passed automatically. You find leaks early, and you have evidence that the automation is safe.

Every failure found in production becomes a new evaluation case, so the same mistake is caught before the next release.

What happens when you swap the AI model?

When you swap the model, your evaluation suite tells you whether the change is safe. Run the same cases against the new model, compare the results with the current one, and switch only if it holds up on the cases that matter. Without that suite, every model change is a leap of faith.

Model changes are not optional. Providers retire models, release cheaper ones and update existing ones behind the same name. AIRA's sustainability domain asks agencies to keep the ability to replace models over time, rather than being locked to one.

This is why the evaluation suite is the most durable asset in an AI system. Prompts get rewritten, models get replaced and retrieval gets rebuilt. The set of real cases, and the judgement about which answers were right, survives all of it.

The model is rented. The evaluation is owned.

Two practical points:

• Make the model a setting, not an architecture. If changing models means rebuilding the system, you will avoid doing it even when you should.
• Cheaper isn't always fine. On Talent Carriage's voice support agent, we tested cheaper and faster models and rejected them. They couldn't reliably work out the right next question to ask. The evaluation suite made that decision with evidence, not opinion.

How we evaluate first in Avesta AgentOS

At Avesta Labs, evaluation comes first, not last. Every engagement starts by writing evaluations from the customer's real, messy cases, and Avesta AgentOS has evaluation generation built in. Evaluations run with every change, and a failing evaluation blocks the release, just as a failing test would. With that in place, one workflow typically goes from demo to production in two to four weeks, including testing and integration.

Three examples from systems we have built:

• A check outside the model. For a vehicle import and logistics company, every vehicle ID read from scanned shipping documents must pass a 17-character rule and a check-digit test. Only 5 to 7 documents in 100 now go to a person. The rest go straight through.
• The client's experts own the evaluation. For India Avenue's investor assistant, an Australian funds manager working under a financial services licence, their own team sent 50 real investor questions and marked every answer right or wrong, with the reason. Those 50 questions are now the suite. Every change runs against them first, which is what lets the firm adopt a newer model and show the answers didn't change.
• The people doing the work are the evaluation. On Talent Carriage's HR support line, the specialist who picks up each AI-written ticket flags any summary that needed fixing. In an early and deliberately small sample, 40 of the first 41 tickets were not flagged.

What to ask your AI vendor

• Show us the evaluation suite for this use case. Who wrote the cases, and who decided what was correct?
• How does this compare with our current process on the same cases?
• What does the system do when it can't confirm an answer? Show us an abstention or a handover.
• Which checks sit outside the model, and which rely on the model's own confidence?
• What blocks a release if quality drops?
• How do you test a model change before it reaches our users, and who owns the evaluation suite if we leave?

If you're planning how to evaluate an AI use case, or need to show how your AI copes when the model is wrong, book a discovery call and we'll work through it with you.

Sources

• Digital Transformation Agency, AI Reference Architecture, Version 1.0, 2026 (CC BY 4.0), sections "Building evaluation into the architecture" and "Validity and reliability", and the glossary entries for abstention and fallback pathway, available at digital.gov.au
• Digital Transformation Agency, Technical standard for government's use of artificial intelligence
• Avesta Labs, Bill of Lading VIN Extraction with LLMs: Check Digits Beat Confidence Scores
• Avesta Labs, What is AI evaluation?