AI Call Scoring: How It Works and How to Check It
An AI model can read and score every sales call. Before your quality assurance team relies on those scores, test them against calls your reviewers have already judged.
Key takeaways
- AI call scoring uses an AI model to read the transcript of every call and score it against a written call scoring rubric, instead of a person listening to a small sample.
- Every score should quote the lines of the call that earned it. A reviewer can check a quoted score in seconds, while an unexplained score has to be taken on trust.
- A check that doesn't apply to a call should be marked not applicable, not passed or failed. Otherwise the report fills with results that mean nothing.
- Before you trust AI call scores, run them on calls your QA team has already marked. Count false flags and missed calls separately, because fixing one can raise the other.
- The AI flags and a person decides. A failed compliance check goes to a reviewer: the scores order the review queue, they don't settle the outcome.
Life Insurance Direct's quality assurance (QA) team used to check 2 to 5% of its sales calls, picked and heard by hand. Between 16 April and 10 August 2026, an AI model read and scored all 9,218 of them, against 26 checks per call.
That is AI call scoring. Getting a score for every call turned out to be the easy part. The hard part is knowing that each score is right. Once reviewers open a few calls the AI flagged and find nothing wrong, they stop trusting the flags.
So the work goes into two places. One is writing checks the AI must back up with lines from the call. The other is testing the scores against calls your reviewers have already judged.
What is AI call scoring, and how does it work?
AI call scoring uses an AI model to read the transcript of every call and score it against a written list of checks, called a call scoring rubric. Each score should quote the moment in the call that earned it. Before you trust the scores, compare them with calls your QA team has already reviewed and marked.
The rule we give the model: no quote from the call, no score.
In practice it runs in four steps:
- The call becomes text. The recording is turned into a transcript, and personal details are removed before the checks run. Our post on redacting personal information before the LLM covers that step.
- Each check reads the transcript. A check is one plain English question, such as "Did the adviser explain what the cover excludes?"
- Each check returns a result with evidence. The result is the answer, how serious it is, and the lines of the call that support it, or a reason the check doesn't apply.
- The calls are ranked for review. The highest risk calls go to the top of the QA team's queue, and a person should make every decision.
What should a call scoring rubric include?
A rubric is the list of checks, and most scoring problems start there. It's tempting to write checks as broad areas, such as "compliance" or "empathy". A model can't score a broad area reliably, but it can answer a specific question.
Anthropic's guidance on model grading says the same: have detailed, clear rubrics and ask for a specific output. In its words, "purely qualitative evaluations are hard to assess quickly and at scale" (Anthropic).
The five parts of each check
| Part of each check | What to write | Example |
|---|---|---|
| The question | One thing, in plain English | "Did the adviser state the cooling-off period (the days a customer has to cancel)?" |
| Pass and fail | What each looks like on a call | Pass: the period is stated. Fail: it is skipped or wrong |
| When it applies | The kind of call it is for | Only calls where a policy is sold |
| How serious a fail is | For example: critical, warning or coaching | Critical: a required statement was missed |
| Evidence | Which lines of the call the AI must quote | The adviser's words, with the moment in the call |
The two parts teams skip
Two of these rows get skipped most often. When it applies stops a sales check from "failing" a call that was only a change of address.
Evidence is the other. To reduce made-up answers on long documents, Anthropic suggests asking the model to quote the text word for word first. It also suggests having it "cite quotes and sources for each of its claims" (Anthropic).
A quoted score takes seconds to check. An unexplained score has to be taken on trust.
Exhibit 1 shows one check's result on one call, filled in.

- The check is one plain English question.
- Applies to this call says why the check ran at all.
- Evidence quotes the lines of the call the result rests on.
- The result and how serious it is follow from the evidence, not the other way round.
- The reviewer's decision stays open until a person confirms or rejects the flag.
How do you know the AI call scores are right?
Test them against calls your reviewers have already judged. Take a set of calls your QA team has marked, run the scoring on them, and compare call by call. Every comparison lands in one of four boxes, as Exhibit 2 shows.

- Agreed flag: the AI and the reviewer both flagged the call.
- False flag: the AI flagged a call the reviewer cleared. Too many of these and reviewers stop trusting the queue.
- Missed call: the AI cleared a call the reviewer flagged. This is the risk you bought the tool to reduce.
- Agreed clear: both cleared it.
A single "accuracy" number hides the trade between boxes 2 and 3. Tightening a check cuts false flags but can raise missed calls. Loosening it does the reverse.
So report the two counts separately, per check, and decide which matters more for that check. For a required disclosure, a missed call is worse. For a coaching check, false flags do more damage.
Run the same test before every change: a reworded check, a new check or a new model. For a required disclosure, if the change moves calls from box 1 into box 3, it doesn't ship. Our post on LLM as a judge covers how to calibrate a grading model against human reviewers in more depth.
How we score every sales call with Metricsense in Avesta AgentOS
Metricsense is our call and agent analytics product, and one of the modules of Avesta AgentOS. For Life Insurance Direct, an Australian life insurance distributor, it reads every sales call. It scores both the adviser and the customer, and gives the QA team a review queue in risk order.
Three patterns from building Metricsense carry over to any call scoring system.
1. Every score must cite the call
In Metricsense, every check is told not to jump straight to "fail". The model is asked to cite two to five specific lines of the call and say what each one shows, skipping greetings and acknowledgements.
It also gives the result and how serious it is, and the result must match those lines. With Claude and Gemini models, the lines are written before the result. The reviewer sees the other side of this: each score links to the exact moment in the call, with the audio.
2. Give a check only the part it needs
One check compared the application questions read on the call with the insurer's form. The obvious build gave the model the whole form and the whole transcript.
But models often read the middle of long documents worst. Researchers found that performance "significantly degrades when models must access relevant information in the middle of long contexts" (Liu et al.). On an insurance application, the middle is where the medical questions sit.
So the application check now sees only the section it needs, with personal details removed. It is still a pilot, because we won't ship a check that guesses.
3. Not applicable is a result, not a pass
In Metricsense, a check that doesn't fit a call is saved as not applicable, with its own column on the dashboard. A call where nothing was triggered still got read.
Before any check changes, it is tested against calls the client's QA team has already marked. Each release is evaluated at scale with Claude Haiku 5.5.
The case study is open about the trade. Early on, some good calls landed in the high risk column. Tightening the rule fixed that, but risked missing other calls, and the marked calls let us measure that instead of guessing.
When is AI call scoring the wrong tool?
It isn't always worth building.
- Low call volume. If your team can already hear most calls, listen to them. Scoring pays off when a sample is all a person can reach.
- No marked calls to test against. Without a set of calls your reviewers have judged, you can't tell a good rule from a bad one. Mark calls first.
- The answer needs a record you don't have. A check that compares the call with a customer record needs a clean record. If the record holds a fraction of what the call covers, keep that check as a pilot.
- Exact words are the requirement. If a disclosure must be read word for word, a keyword rule is cheaper and easier to defend.
- You want the AI to decide. It should flag, never act. A failed compliance check goes to a reviewer. In Metricsense, a person confirms every flag, and nothing is reported to a regulator on its own.
The bottom line
AI call scoring is only as good as its checks. Write each one as a specific question the model must answer from the call itself, and let a person make the decision.
For grading models in general, see LLM as a judge. For monitoring live agents, see agent analytics in production.
Before you buy a call scoring tool, ask the vendor to score calls your QA team has already marked, and count both kinds of disagreement.
Want every sales call read and scored, with your QA team still making every decision? See how we built it for Life Insurance Direct.
Get started: hello@avestalabs.ai
Frequently asked questions
- Can AI score 100% of calls?
- Yes, if the calls are recorded and turned into transcripts. The model reads each transcript and runs every check that applies. The limit is not volume but trust: every call gets a score, so a wrong rule is wrong on every call too. That is why the scores need testing against calls your reviewers have already marked.
- Does AI call scoring replace quality assurance reviewers?
- No. It changes what they spend their time on. Instead of choosing which calls to hear, reviewers start with the calls the AI ranks as highest risk and confirm or reject each flag. The calls they mark also become the set that new rules are tested against, so the scoring keeps improving.
- How is AI call scoring different from keyword spotting in speech analytics?
- Keyword spotting finds words, such as whether a disclosure phrase was said. An AI model can judge meaning, such as whether an explanation was complete or a customer was led towards a product. Use keyword rules where the exact words are the requirement, and the AI model where the meaning is.
- What makes AI call scoring software accurate?
- Ask the vendor questions the demo won't answer. Can you see the lines of the call behind every score? Can a check be marked not applicable instead of passed? What happens when a check is reworded: is it re-run on calls your reviewers have already judged first? And who confirms a flag before anything happens? Clear answers matter more than the model's name.
Share this article




