AI Insights
LLM provider failover is the practice of automatically retrying a failed AI request, and if it keeps failing, sending it to a backup model or provider so your product keeps working. Every hosted model API has outages, rate limits and overloaded periods, so any AI product with paying users needs a plan for them before launch, not after the first bad morning.
This guide is for founders and technical leads building on hosted models. It covers which errors are worth retrying, how to back off properly, when to fail over to a second model, and the traps that make failover quietly worse than the outage it was meant to solve.
Retry the errors that are temporary and fix the ones that are not. Anthropic’s API error documentation shows the pattern clearly. A 500 is an unexpected internal error and should be retried with exponential backoff. A 529 means the API is temporarily overloaded and is also transient. A 429 means a rate limit was hit, and the documentation advises retrying only if the response includes a retry-after header; a spend-cap 429 has no such header and will not resolve by retrying. Other 4xx errors, such as 400, 401, 403, 404 and 413, mean the request or configuration is wrong and must be fixed first.
The same page notes that the official SDKs already retry connection errors, 429s and 5xx errors with exponential backoff, twice by default, and honour retry-after when present. Check what your SDK does before writing your own loop, or you will multiply retries without meaning to.
Wait a short time, double it on each failure, and add randomness. OpenAI’s cookbook on handling rate limits recommends random exponential backoff: wait briefly after a rate limit error, retry, and lengthen the wait if it fails again, until the request succeeds or a maximum retry count is reached. The randomness, called jitter, stops many clients from retrying at the same instant. The guide also warns that failed requests still count toward your per-minute limit, so resending immediately makes the problem worse.
Set a hard ceiling on total retry time. A user waiting on a chat reply will not tolerate a minute of silent retries, so decide up front how long the app may try before it tells the user something or switches path.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
Fail over when retries on the primary have been exhausted, or when the failure is one retrying cannot fix. Three cases are worth planning for: sustained overload or outage at the primary, a rate limit you cannot wait out, and a request that is too long for the primary model’s context window. Each can have a different backup. An overload might route to the same model family on another cloud, while a too-long request might route to a model with a larger context window.
Routing libraries make this configuration rather than code. The LiteLLM router documentation describes fallbacks, which move to the next model group after a deployment fails, an order setting so lower-numbered deployments are tried first with their own retries, and cooldowns that take a failing deployment out of rotation. By default it allows three failures per minute before a five-second cooldown. These are the documented defaults at the time of writing, so confirm them for the version you use.
Failover fails quietly more often than it fails loudly. The common traps are predictable.
Decide on a graceful degradation path before you need it. Options include a cached answer for common questions, a plain message that explains the delay and offers a human contact, or queueing the request to be processed when service returns. For anything customer-facing, a clear honest message beats a spinner. Our production readiness checklist for AI apps includes this as a launch gate, and our guide to custom AI development in Australia explains how reliability fits into a staged build.
Before you launch, you should be able to answer yes to each of these: Do we know which errors we retry and which we fix? Do retries use backoff with jitter and a total time limit? Is there a named backup, tested on our own examples? Do we know where the backup processes data? Do we log and alert on every switch? Is there a human-readable message if all paths fail? If any answer is no, that is your next piece of work.
LLM provider failover is the practice of retrying a failed AI request and, if it keeps failing, routing it to a backup model or provider so your product keeps working.
Temporary ones. Anthropic’s documentation says 500 and 529 errors should be retried with backoff, and a 429 should be retried only if the response includes a retry-after header. Other 4xx errors mean the request or configuration needs fixing first.
Use random exponential backoff: wait briefly, lengthen the wait on each failure, add jitter, and stop at a maximum retry count. OpenAI’s guidance also notes that failed requests still count toward your per-minute limit.
The backup may answer differently, break structured output, cost more, or process data outside the region you promised customers. Test the backup on your own examples and log every switch.
It can. The OAIC’s guidance tells organisations to check where cloud-hosted AI servers are and whether personal information could leave Australia, so treat each failover target as a disclosure decision.
Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — we design reliability, testing and fallback paths into the AI products we ship for founders and small businesses.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
Planning custom AI development in Australia? See what you pay for, how to stage a build, which rules apply and how to pick a partner. Book a free scoping call.
AI Insights
AI teammate for optometrists and audiologists: answer calls, book visits and hand clinical matters to your staff. Hosted in Australia, from $79/mo. Try NeoMind.
AI Insights
Learn how online evals score live AI traffic: what to check, how to sample, and when to use an LLM judge. Book a free 15-minute scoping call with Neomeric.