How to Evaluate AI Vendors Against AIRA (Free Scorecard)

Three vendors. Three polished demos. All three say "responsible AI", "enterprise-grade" and "human in the loop". And you have to recommend one by the end of the month.

Australia's AI Reference Architecture (AIRA) was written partly to fix this problem. It gives buyers a shared structure for comparing AI offerings like for like. This eighth post in our AIRA series turns that structure into 42 questions, with what to score, what evidence to ask for and what to make vendors show you live.

Why use AIRA to evaluate AI vendors?

AIRA gives every vendor the same map to answer against, so you compare like with like instead of comparing pitches. It names the capabilities a trustworthy AI system needs across the functional stack, communication, trustworthiness and human oversight. A vendor either shows evidence for each one or it doesn't.

AIRA lists vendor evaluation as one of its main uses. It says procurement functions can use it to compare products and services and align them with government standards, and that it supports consistent, like-for-like evaluation of vendor offerings.

Three practical benefits:

• Same questions for everyone. Vendors can't steer you towards the parts of their product that demo well.
• Gaps become visible. A missing capability is easier to spot when you check against a full list.
• A shared language. Architects, security, legal, procurement and business owners can score the same sheet, each on the parts they know.

Two cautions. AIRA doesn't certify vendors, so "AIRA-compliant" or "AIRA-certified" claims mean nothing on their own. And it doesn't replace your procurement rules, security assessment or impact assessment. It sits alongside them.

Same map for every vendor. Compare evidence, not pitches.

How should you score and weight AI vendors?

Score each question from 0 to 3 based on the evidence a vendor shows, then weight each section by the risk of your use case. A vendor that claims a capability but can't demonstrate it scores barely above one that doesn't have it. That rule alone removes most of the noise from AI evaluations.

Scoring each question:

Weighting each section by risk tier. Our post on applying AIRA by risk explains how to set the tier.

These weights are a starting point, not a rule. Adjust them to your situation and agree them before you see any vendor, so the scoring can't be bent towards a favourite.

Two more rules:

• Set must-haves. Some questions are pass or fail for your use case, such as data staying in Australia or an IRAP assessment. A zero on a must-have rules the vendor out, whatever the total.
• Score as a team. Have security, legal, architecture and the business owner each score the sections they know, then compare.

A capability a vendor can't show you scores almost the same as one they don't have.

Functional stack: what should you ask an AI vendor?

These questions test whether the product can actually do the job, from the screens people use down to where it runs. They cover AIRA's five functional layers: experience, application and orchestration, intelligence and analytics, data and metadata, and platform and infrastructure.

Communication and integration: what should you ask?

These questions test how safely the product connects to your people, your systems and other AI agents. In AIRA, communication covers identity and trust, APIs, legacy integration, agent communication protocols and reusable assets.

Trustworthiness and assurance: what should you ask?

These questions test whether the product is built to be trusted, and whether it can prove it. They follow AIRA's trustworthiness and assurance domains: observability, traceability, explainability, validity and reliability, safety, fairness, sustainability, security and privacy.

Earlier posts in this series explain what good answers look like. Questions 16 to 19 are covered in building dependable AI around imperfect models, and questions 20 to 23 in reconstructing an AI decision. For question 25, see our guide to prompt injection in AI agents.

Human oversight: what should you ask, and what should vendors demonstrate?

Human oversight is where vendor claims are most often vague and most easily tested. Every vendor says "human in the loop". Ask them to show you exactly where the human sits, what they see, and what happens when nobody responds. AIRA's oversight domain covers both decision support and intervention.

Make them demonstrate these five things live:

1. An action held for approval, then approved with an edit
2. A request that expires with no answer, and what the agent does next
3. A supervisor pausing an agent mid-task
4. A complete decision record exported for one case
5. A rollback to the previous version

If a vendor can't do these in a demo, assume they can't do them in production. Our guide to keeping humans in control of AI agents explains what each control is for.

If a vendor can't show you a rollback in the demo, assume they can't do it in production.

Commercial, sovereignty and exit: what should you ask?

These questions decide whether you stay in control after you sign. They cover where your data lives, how dependent you become, and how easily you can leave. AIRA treats dependence on providers as part of sustainability, so these belong in the scorecard, not just the contract.

Our guide to sovereign AI and avoiding lock-in covers questions 35 to 41 in depth, including the difference between IRAP and the Hosting Certification Framework. If you're still deciding whether to buy at all, start with choosing build, buy or blend.

Contract with the Commonwealth's AI model clauses. In March 2025 the Commonwealth published AI model clauses, developed by the DTA with the Australian Government Solicitor. They help buyers contract for bespoke AI systems and for services where the seller uses AI, and they can be used with Digital Marketplace Panel 2 contracts or tailored for other agreements (AGS). Use this scorecard to choose the vendor, then the model clauses to hold them to it.

State agencies have their own guidance too. Victoria, for example, publishes guidance on navigating AI in procurement, including asking suppliers for evidence of their AI governance.

What are the red flags in AI vendor pitches?

The biggest red flag is a claim that can't be shown. If a vendor answers questions about oversight, traceability or data location with adjectives instead of evidence, assume the capability is weak or missing. These are the warning signs worth noting on the scorecard.

• "AIRA-compliant" or "AIRA-certified". AIRA has no certification. At best, a vendor can show how it aligns.
• "Our model is 99% accurate." Accurate on what, measured how, against which baseline? Accuracy on someone else's data says little about yours.
• "The AI tells you how confident it is." A model's own confidence is not a reliable signal of correctness. Ask what checks sit outside the model.
• "Human in the loop" with no detail. Ask exactly which actions wait for a person, and what happens when nobody responds.
• Demos only on the vendor's examples. If they won't run your real cases, including messy ones, the demo tells you little.
• One model, no choice. If you're locked to one model provider, their pricing, retirements and roadmap become yours.
• Vague data answers. "Your data is secure" isn't an answer to "where are our logs stored?"
• No production references. Pilots are easy. Ask to speak to a customer running it in production.
• No exit plan. If leaving hasn't been thought through, it will be expensive.

"AIRA-certified" is a red flag. AIRA doesn't certify anyone.

How should you run the evaluation?

Run the evaluation in three rounds: a written response against the scorecard, a live demonstration on your own cases, then a short, paid proof of value with your shortlisted vendor. Each round should cut the field and raise the bar for evidence.

Round 1: written response. Send all vendors the same questions, with your must-haves marked. Score no answer higher than 2 at this stage, because nothing has been shown live yet.

Round 2: live demonstration on your cases. Give each shortlisted vendor 20 to 50 real examples from your work, including the messy ones, a few days in advance. In the session, ask them to:

• Run your cases and show the results, including the failures
• Demonstrate the five oversight actions above
• Export a complete decision record
• Show where data is stored, live in the console

Round 3: proof of value. Run a two- to six-week trial with one vendor on one real workflow, with an agreed evaluation suite and success measures set in advance. A proof of concept asks "can this work?" A proof of value asks "does it work well enough, on our cases, against our current process, at an acceptable cost?"

Keep the evaluation suite you build. It stays useful after you choose, because it becomes the release gate for the system you buy.

Get the free AIRA vendor scorecard

All 42 questions above, in a spreadsheet you can use straight away:

• Each question with the evidence to ask for
• A 0 to 3 evidence score, with weights already set for low, medium and high-risk use cases
• Must-have flags that rule a vendor out automatically
• Side-by-side columns for up to four vendors

Ask for a copy of the AIRA vendor scorecard and we'll send it to you.

A note on how we answer it. We build and run Avesta AgentOS, so we answer these same questions ourselves. The last part of this series maps AgentOS against AIRA, including where we are still partial. Hold us to the same standard as anyone else.

Sources

• Digital Transformation Agency, AI Reference Architecture, Version 1.0, 2026 (CC BY 4.0), sections "Using AIRA in your agency" and "Appendix A: Key roles in AIRA", available at digital.gov.au
• Australian Government Solicitor, Artificial Intelligence (AI) model clauses, legal update no. 327, 4 April 2025
• Buying for Victoria, Navigating AI in procurement