Voice Agent Testing: Text, Audio and Real Calls
Real callers pause, interrupt and mix languages. Test for that before they hang up.
Key takeaways
- Voice agent testing has four layers: text conversations, audio runs, real people trying to break it, and measuring live calls.
- Test the conversation in text first. It is cheaper and repeatable. Save audio runs for pauses, interruptions and speech recognition.
- Testing tools warn that simulated callers do not fully represent real ones. Real callers open with a detail, pause to think and change what the query is about halfway through.
- The best live measure is one the business already owns: how often the person who picks up the ticket has to correct it.
- On the Talent Carriage HR support line (phone and WhatsApp), 40 of 41 tickets needed no correction (client dashboard, 10 Sep 2026).
An employee rings the HR support line. They don't say "I have a portal problem". They say the tabs in their inbox aren't showing, the way they show for everyone else. Then, in answer to the AI voice agent's next question, they say "Wait." and go to check something.
A scripted test call, where a tester reads the caller's lines, would never do that. Scripted callers open with the subject, answer each question in order and never leave the phone. Real callers break all three habits. A voice agent that only passed scripted calls can fail with its first real callers.
Testing a voice agent means closing that gap before launch. A live HR support line showed us which two steps only real people can test.
What is voice agent testing, and what should it cover?
Voice agent testing checks that an AI voice agent hears, understands and acts correctly on real phone calls. Test the conversation in text first. Then rerun key scenarios as audio for pauses, interruptions and mixed languages. Then let real people try to break it. Finally, measure live calls by how often a person must correct the result.
The order we test in:
Text tests check what the agent asks. Audio tests check how it handles speech. Real callers check everything else.
A voice agent has more moving parts than a chat agent. It has to hear: speech recognition turns the caller's voice into text. It has to think: the AI model decides what to ask next. It has to speak, and it has to take turns, knowing when the caller has finished.
Each part fails in its own way, so each needs its own test.
How do you test a voice AI agent before launch?
Test in layers, from cheapest to most realistic. Each layer catches failures the cheaper ones miss.

- Text conversations. A text scenario is the caller's lines plus what a correct result looks like: the questions, the ticket fields and the team. Vapi checks the agent's next reply by "exact matching, a pattern, or an AI judge". LiveKit's own docs call text mode "a cost-efficient and deterministic way to test agent behavior". Deterministic here means repeatable: the same test should give the same result.
- Audio runs. Rerun the important scenarios as speech: an AI tester's voice, a recorded clip or a person on the phone. Vapi's testing guide says voice runs exercise "transcription, pronunciation, pacing, and turn-taking", which text skips.
- Real people trying to break it. Before launch, staff who know the callers play awkward callers on purpose.
- Live calls, measured. After launch, every call is read by someone, and their corrections become the score.
| Layer | What it catches | What it misses | When to run it |
|---|---|---|---|
| Text conversations | Wrong questions, wrong ticket fields, wrong team | Timing, hearing, accents | Every change |
| Audio runs | Talking over people, dead air, misheard words | Requests nobody thought to script | Before each release |
| People trying to break it | The odd openings and topic changes real callers use | Volume, rare cases | Before launch |
| Live calls, measured | Everything left | Problems you could have caught earlier | Every day after launch |
Simulated calls are only as varied as the person who wrote them.
Vapi's guide makes the same point about its own AI testers: "synthetic callers do not fully represent real callers and their environments". That is why layers 3 and 4 exist.
Why do pauses, interruptions and mixed languages need audio tests?
Callers notice these three failures at once, and a text test cannot see any of them.
1. Pauses
A caller who stops mid-sentence to think is not finished. An agent that answers too early talks over them. One that waits too long leaves dead air. A 2025 study found that voice systems "sometimes do not understand when to speak up, can interrupt too aggressively and rarely backchannel". Backchannel means small sounds like "mm-hm" that show you're listening.
Test it with calls that pause mid-sentence, and with callers who leave the phone to check something.
2. Interruptions
Callers interrupt, often to answer before the question is finished. The agent should stop and listen. It should not stop for a cough or a laugh. Benchmarks such as Full-Duplex-Bench now test "pause handling, backchanneling, turn-taking, and interruption management" as separate skills.
Test it by interrupting the agent's questions, and by adding background noise that isn't speech.
3. Mixed languages
On the Talent Carriage line, callers use Hindi, English, or both inside one sentence. Some speech services have a special mode for this. Sarvam's speech-to-text has a "codemix" mode that writes English words in English and Indian-language words in their own script. Its docs also note that when several languages are detected, it reports only the "most predominant" one.
Test it with mixed-language calls, and check the transcript word by word, not only the language label.
What metrics matter for voice agent evaluation?
You could track many metrics. Three groups matter for most support lines:
- Hearing. Word error rate: the swapped, missing and extra words divided by the words in a correct transcript. Check it on your own callers' accents and languages.
- Timing. The pause before the agent answers, and how often it talks over the caller.
- Outcome. Did the call end with the right result? For a support line, that means a ticket that is complete, on the right team and needs no rework.
The first two are easy to measure and easy to over-trust. A perfect transcript and fast replies mean nothing if the ticket is wrong.
Judge a voice agent by the outcome the business already checks, not only by what the speech engine reports.
For more on grading answers and building a first test set, see what AI evaluation is and how to grade agent answers with an LLM as a judge.
What we learned testing the Talent Carriage voice agent
For Talent Carriage, which runs HR for more than 50 companies in India, we built the voice co-worker on its HR support line. It takes calls in Hindi, English or Hinglish (the two mixed) and logs each query as a ticket for one of eight teams. It went live at the end of August 2026.
The layers above are general practice. This project's record covers layers 3 and 4. Before launch, Talent Carriage's own staff, who know the callers, tried to break the agent alongside our own testing. Two things from it are worth copying.
1. A fixed question order could not cope with real callers
The first version asked questions in a fixed order: category first, then that category's questions.
Real callers open with a detail, not a subject. A question that sounds like leave turns out to be about payroll. Once the agent had picked a category it couldn't go back. It asked for things it had already been told, or filed the query under the wrong team.
We rebuilt it to decide what to ask next from the whole conversation, so it can change its mind halfway through. For your own tests, that means at least one scenario where the subject changes mid-call.
The model choice came from the same problem. We tried cheaper and faster models and rejected them, because they could not reliably pick the right next question.
2. The person who resolves the ticket records the correction
Every call ticket carries the recording, the transcript and the agent's summary. If the person resolving it finds the summary or the team wrong, they flag it in one field.

- A caller rings and the agent logs the ticket during the call.
- The specialist team resolves it, with the recording and transcript on the ticket.
- If the summary or the team was wrong, they flag it in one field.
- Real calls show what to tune.
- The tuned agent goes live and takes the next calls, and the loop repeats.
On 10 September 2026, the client's dashboard showed 40 of 41 tickets needed no correction, across phone calls and WhatsApp messages. It is a small, early number. It counts tickets nobody flagged, not tickets someone marked correct. But it measures the outcome, using people who already read every ticket.
Callers also talked to it in ways our testing never produced. Those calls are where the tuning since launch has come from.
When is a voice agent testing platform not worth it?
Testing platforms such as Vapi's use AI testers that play the caller, then score each conversation. They earn their cost when you have high call volume, frequent releases, many languages or regulated scripts that must be said word for word.
They are not worth it yet if you run one line with a few dozen calls a week. There, the layers above plus a correction field on every ticket are usually enough.
Start a platform when people can no longer read the calls themselves.
The bottom line
Voice agent testing is mostly about the gap between the caller you imagined and the one who rings. The same habit applies to chat agents: our guide to testing a RAG chatbot (one that answers from your documents) starts from questions the client wrote.
If you are about to launch a voice agent, start with one question: who will read the first hundred calls, and where will they record what the agent got wrong?
Want a voice agent tested on your own callers before it answers your line?
See how we built and measured the Talent Carriage voice agent.
Get started: hello@avestalabs.ai
Frequently asked questions
- How do you test voice agents in real-world conditions?
- Use people who know the callers. Before launch, ask staff to play awkward callers: ones who open with a detail, pause, interrupt or switch languages. After launch, have someone read every early call and record what the agent got wrong. Add background noise and poor phone lines to your audio runs, because real calls rarely sound like a quiet office.
- What is word error rate in voice agent testing?
- Word error rate measures how many words the speech recognition got wrong when it turned the caller's speech into text. It adds up swapped, missing and extra words and divides by the number of words in a correct transcript. It is a useful check on the hearing part of a voice agent, but a low score does not prove the agent asked the right questions or logged the right outcome.
- How do you test a voice agent in Hindi and English?
- Test the mix, not just each language on its own. Real callers switch between Hindi and English inside one sentence. Record or script test calls that do this, check the transcript keeps both languages, and check the agent replies in the language the caller is using. Some speech services report only the main language they detected, so check the words themselves.
- How many test calls do you need before a voice agent goes live?
- There is no fixed number. Cover every kind of request the line gets, the awkward callers (pausers, interrupters, language switchers) and every way out, such as asking for a person. Then start with a small group of callers where you can, and make sure every early call is read by someone.
Share this article




