LLM Fallback Strategy: Retries and Failover for AI Agents
What your AI agent should do when its model fails: retry, wait, switch to a backup model, or stop.
Key takeaways
- An LLM fallback strategy sets what an AI agent does when its model fails: retry, wait, switch to a backup model, or stop and tell a person.
- Choose by the kind of error. Retry a server error or an overload, and wait out a rate limit. A spend cap or a bad request fails however often you retry.
- Retry in one place only. Anthropic's SDK and the Vercel AI SDK already retry twice by default, so extra retries in your own code multiply.
- An agent that fails halfway through a task leaves unfinished tool requests behind. Save what was done and mark the rest as interrupted before any retry or backup model carries on.
- A backup model is only a fallback if it passes the same tests as the main one. Run your evaluation set on both.
A refund can go out twice without the AI model making any mistake. The model only has to go quiet for a minute, halfway through the task.
In a typical case, the AI agent looks up the order and asks the payments system to issue the refund. Then the AI model behind the agent stops answering. The company that runs the model, the provider, is overloaded.
What happens next is up to you. The customer might see an error, or wait a minute for nothing. Or the agent might start the whole task again, and the payment goes out a second time.
The model here is a large language model, or LLM: the part of the agent that reads and writes text. An LLM fallback strategy is the set of rules for that minute: when to retry, when to switch provider and when to stop.
What is an LLM fallback strategy?
An LLM fallback strategy is the set of rules for what an AI agent does when its model fails to answer. It says which errors to retry, how long to wait, when to switch to a backup model from another provider (called failover), and when to stop and tell a person. The right move depends on the kind of error, not just on the fact that something failed.
These rules live in the software around the model, the agent harness, not in the model itself.
In one line:
Retry what might pass next time. Switch when the provider is down. Stop when nothing will help.
When should you retry, and when should you fall back to another model?
Read the error first. Providers return different errors for different problems, and each one points to a different move.
| What the error says | What it means | Retry? | Switch to a backup model? |
|---|---|---|---|
| Server error (500) or a dropped connection | Something broke on the provider's side | Yes, after a short wait | Yes, if retries fail |
| Overloaded (529 or 503) | The provider is busy for everyone | Once or twice, with growing waits | Yes, if it persists |
| Rate limit (429) | You sent too much, too fast | Yes, after the wait the provider asks for | Only if the work can't wait |
| Spend cap or quota (a 429, a 400 or a billing error) | Your account is out of budget | No | Only to a provider your budget covers, and tell a person |
| Timeout (no reply within your limit) | The provider is slow or stuck | Once, inside one overall timeout | Yes, if the second try is slow too |
| Request too long for the model's context window | The conversation no longer fits | No | Only to a model that takes longer input |
| Bad request (400) | Something in your request is wrong | No | No: fix the request |
The sources are the providers' own error guides. The same code can mean two things. A 429 can be a rate limit or a spend cap, and a 400 can be a bad request or a spend limit you set. Read the error type and message, not just the number.
Anthropic's API errors page says a spend-cap 429 "keeps failing until access resumes". OpenAI's error codes guide says "Retrying billing, spend, or quota errors won't restore API access."
Retry in one place only
Many libraries already retry for you. As of November 2026, Anthropic's SDK retries connection errors, rate limits and server errors "twice by default". The Vercel AI SDK's default is 2 retries too.
Add your own retry loop on top, and the attempts multiply. Amazon's guide to timeouts, retries and backoff warns that "Retries are 'selfish'", and that when the cause is overload, extra retries "can make matters significantly worse". Its advice: retry "at a single point in the stack".
Before you add a retry, find out who is already retrying.
How does a fallback chain work?
A fallback chain is the order the agent works through when a request fails. Each step gives up more than the one before: more time, more cost or a weaker answer. So the agent moves on only when the step before has failed.

- Retry the same model. Once or twice, with a growing wait plus a little randomness, so many agents don't all retry at the same moment.
- Switch to a backup model (failover). Send the same request to a model from another provider. LiteLLM's reliability docs describe this order: once a call has used up its retries, LiteLLM "falls back to another model group."
- Give a simpler answer. Tell the customer the request is saved and someone will follow up, instead of an error.
- Stop and tell the person. Say plainly that it failed, and record which error it was.
The error table decides where a request enters the chain. A server error, an overload or a timeout starts at step 1. A spend cap or a bad request skips to step 3 or 4, because no retry and no backup will fix it.
One addition saves a lot of waiting: skip a model that keeps failing. LiteLLM puts a model into cooldown for a set time after repeated failures. While it cools down, send requests straight to the backup.
Set one timeout for the whole chain, too. Without it, three steps of retries can leave a customer waiting for minutes.
What happens when an AI agent's model fails halfway through a task?
This is where agents differ from chatbots. A chatbot sends one question and gets one answer. An agent works in steps, and many steps are tool calls: the agent asking another system to look something up or take an action.
Back to the refund. If the model fails after asking for the refund but before reading the reply, the agent's record holds a request with no result.

- Look up the order. The tool replies with the order details.
- Ask for the refund. The request goes to the payments system.
- The model fails. It never reads the payments system's reply.
- Save and mark. Keep the work done so far, and mark the refund request as interrupted.
- Check before redoing. The next attempt, on the same model or a backup, checks the refund status first.
Two things go wrong if you skip step 4. Anthropic's guide to handling tool calls says each tool result "must immediately follow" its request, so a record with a missing result is rejected and the retry fails too. And an agent that simply starts again may issue the refund twice.
Amazon's guide sets the goal for any action with side effects. They should "happen only once no matter how often you retry." Our post on AI agent tool call failures covers how to tell an agent an outcome is unknown.
Will a backup model behave the same way?
Not by default. A backup model reads the same instructions and the same tool descriptions, but it may follow them differently. Some reject settings that others accept. As of November 2026, Anthropic's errors page lists models that return a 400 if you force them to use a particular tool.
So treat the backup as a second release. Run the same evaluation set on it that you run on the main model, and fix what fails before the day you need it. Our post on what AI evaluation is explains how to build that set.
How we handle model failures in Avesta AgentOS
Avesta AgentOS is the platform we build and run our AI agents on. Each agent runs on one model its builder chooses, from providers that include OpenAI, Anthropic and Google. Three patterns from it are worth copying.
1. Every failed request is recorded
When a model request fails, the platform records the provider, the model and the agent, and sorts the failure by kind, such as a server error or a person cancelling. Those records show which provider and model fails, for which agent, and how often: the first thing to read before adding a backup.
Image requests also get a timeout, because a stuck provider would otherwise stall the conversation. When an agent asks an image model for a picture, the request has 120 seconds, and that limit covers the one retry too. A retry never doubles the wait.
Avesta AgentOS doesn't switch a conversation to another model by itself. A failed request is retried, and if it still fails, the person sees a plain message asking them to try again.
2. A half-finished turn is saved, not thrown away
If a person stops a turn, or the model's reply ends with no output, the platform keeps whatever answer was written so far. Every tool request that never got a result is given one that says the tool's work was interrupted. That keeps the record valid, so the person's next message can carry on.
3. One set of tool definitions for every provider
Not every provider accepts every detail in a tool's description, such as format hints, examples or default values. It removes those before sending, so one set of tool definitions can go to each provider it supports.
We still found a gap. Our tools tell every provider not to enforce strict checking of each tool's fields. One provider library didn't pass that setting on, so that route checked strictly anyway. We ran a workaround from September to November 2026, until the library was fixed.
The same tools, sent through a different route, behaved differently. That is why a backup model gets tested, not assumed.
When should you not use an LLM fallback?
A fallback is not free. It needs a second provider account, a second set of tests and a decision about data.
- When your data agreements name one provider. A backup model sends the same customer data to another company. If your privacy assessment or contract doesn't cover that company, don't add the backup. In Metricsense, our call analytics product, a customer who must use their own provider key gets a clear error, never a silent switch to a different key.
- When the system must stay in one cloud account. AskCHAIRS, for CHAIRS Global, runs entirely inside the client's own AWS account. For a system built that way, a backup model has to sit inside that account too, or there is no backup.
- When a different answer is worse than a late one. For regulated wording or approved advice, queue the task for a person instead.
- For batch work. Overnight jobs can wait an hour and run again. Retries and a rerun are cheaper than a second provider.
The bottom line
Start with the error table, retry in one place, and save half-finished agent work before anything runs it again. Our post on moving AI agents from prototype to production covers the other gaps that show up on day one.
Pick one agent that can change something, such as a refund or an email, and stop its model halfway through a test run. Then check whether the task runs twice.
Want AI agents that record every model failure, keep half-finished work and run on the provider you choose? See how Avesta AgentOS handles it.
Get started: hello@avestalabs.ai
Frequently asked questions
- What is a fallback in AI?
- A fallback is the second option a system uses when the first one fails, and the fallback approach is planning that option in advance. In AI applications it usually means sending a request to a backup model or provider when the main model returns an error, times out or is overloaded. A good fallback also covers what happens when every option fails: a plain message to the user and a record of the failure.
- What is the difference between fallback and failover?
- The two words overlap. Fallback usually describes one request moving to a backup model after it fails. Failover usually describes moving all traffic to a backup for a period, for example while a provider has an outage. Either way, the backup has to be tested and allowed under your data agreements before you need it.
- How many times should you retry an LLM request?
- Usually once or twice, with a growing wait between attempts. Anthropic's and the AI SDK's libraries default to two retries. More attempts rarely help with an overloaded provider and add load to it. Set one timeout that covers the request and every retry, so a person isn't left waiting while the attempts stack up.
- Does a backup model need its own prompt?
- Sometimes. Models from different providers may follow the same instructions differently, and some reject settings that others accept. Start with the same prompt, run your evaluation set on the backup, and only write a separate version where the results show it needs one. Keep both versions under the same tests so they don't drift apart.
- What is a circuit breaker for LLM calls?
- A circuit breaker stops sending requests to a model that keeps failing, for a set period. Instead of every request waiting through its own retries, traffic goes straight to the backup until the cooldown ends. LiteLLM, for example, can put a model in cooldown after more than a set number of failed calls within a time window.
Share this article




