Your LLM Fallback Plan Might Be Making Outages Worse

If you build anything on top of an LLM API, the last year has probably taught you the same lesson it taught everyone else: the providers go down, and they go down more than they used to. By June 16, 2026, Claude had logged its tenth significant disruption in twelve days (TechTimes). Third-party trackers counted well over a hundred Anthropic incidents in a rolling 90-day window, and hundreds since the start of 2025 (StatusGator). Treat those exact counts as directional (every tracker defines an “incident” a little differently), but the direction is not in doubt. The usual explanation is boring and probably correct: demand is outrunning the infrastructure. Anthropic’s reported revenue and large-enterprise customer count both roughly tripled in a matter of months, and the servers are feeling it (PYMNTS).

So you do the obvious thing. You add a retry. Maybe you add a second provider and fail over to it when the first one chokes. Ship it, sleep better.

Here’s the uncomfortable part. That reflex comes straight from the microservices playbook, and for LLMs it can quietly make your outages worse instead of better. Retrying harder can turn a thirty-second blip into a ten-minute one. Failing over to a “different” provider can land you right back in the same failure. And retrying a tool call can charge a customer’s credit card twice. Fallback is worth doing. The naive version of it is one of the things breaking in 2026.

Let me walk through what actually holds up, and where the LLM-specific traps are hiding.

Why the microservices playbook breaks

The retry-and-failover patterns most of us learned were designed for a very specific kind of call: fast, cheap, idempotent, and stateless. A remote procedure call that takes fifty milliseconds, costs nothing, can be safely repeated, and carries no memory between attempts. If one fails, you fire it again, and the worst case is a tiny bit of wasted time.

An LLM call is none of those things. It can take eight seconds, or thirty. It costs real money on every single attempt, because you pay for every token you send. It often is not safe to repeat, because the model may have just told a tool to do something in the real world. And a long agent session is nothing but state.

The clearest way to see the mismatch is the math on a naive retry. As one 2026 write-up on the problem put it, “a retry policy designed for 50ms RPCs does not survive contact with an 8-second LLM call” (TianPan, May 2, 2026). Stack three attempts with exponential backoff on top of a call that was already slow, and you can burn thirty-plus seconds before your fallback even gets a turn. The user is long gone. (Those specific figures are the author’s own estimates, so hold them loosely, but the shape is right.)

And every one of those attempts re-sends the entire prompt. In a long agent session that’s thousands of tokens of history, re-billed on each retry. A retry storm is not just a latency problem. It’s a cost problem, arriving at the worst possible moment.

The retry storm nobody means to cause

This is the counterintuitive core, so it’s worth being concrete. Retries do not just fail to help during an outage. They can be what makes the outage bad.

On May 29, 2026, Azure OpenAI had a rough eight hours: elevated latency, timeouts, and 5XX errors, worst in Europe and Australia East. The root cause writeup is the interesting bit. An upstream change altered how certain capacity failures were surfaced, which “led to a rapid and unexpected increase in internal retry traffic” (Azure status history). A change in error handling made clients retry more aggressively, the extra load fed the problem, and the problem generated more errors. That loop is a retry storm, and it is exactly what the TianPan piece warns about when it says naive retries can “extend brief outages from 30 seconds to 10 minutes.”

The fix is to treat your retry budget as a scarce resource rather than a free reflex. One suggested rule of thumb is to cap retry traffic at something like five to fifteen percent of your base load and to fail fast instead of queuing when you blow past it. Take that specific range as a starting heuristic, not an industry standard, because it mostly comes from a single source.

The other half of not causing a storm is telling apart two failures that look similar and need opposite responses. A 429 means you are being rate-limited: the provider is telling you, explicitly, to slow down. The right move is to honor the rate-limit headers, back off, and shift load to another provider or quota pool. Hammering it with immediate retries is the one thing guaranteed to keep you throttled. A hard 5xx is a real server error, where a brief retry before falling back makes sense. Production guidance generally treats 429, 500, 502, 503, and 504 as retryable and 400, 401, 403, and 404 as not worth retrying at all, since a malformed or unauthorized request will fail the same way every time (Maxim AI, Feb 2026). Conflate the rate-limit case with the outage case and you build a storm generator.

Circuit breakers: fail fast, fail forward

If retries are the pattern that can hurt you, the circuit breaker is the one that tends to help. If you have not built one, the idea is borrowed from electrical wiring. You track the failure rate for each provider-and-model pair. When failures cross a threshold, the breaker “trips open,” and for a cooldown period every new request skips that provider entirely and goes straight to the next option in your chain. After the cooldown it goes “half-open,” lets a test request or two through, and closes again only if they succeed (Maxim AI; a clean TypeScript example lives in this walkthrough).

The subtle advantage is easy to miss, and it’s the reason circuit breakers beat mid-stream retries for anything with side effects. A breaker makes its decision before the request goes out. It looks at recent history, decides the provider is unhealthy, and routes around it without ever dispatching the call. A mid-stream retry does the opposite: it re-sends a request that may already have been received and acted on upstream. When your call can trigger a real action, gating before dispatch means you never re-issue an operation that might have already committed. That is the whole game, and the next section is why it matters so much for agents.

One honest caveat, because this pattern gets sold as a silver bullet. Circuit breakers are not free to run well. Tuning the thresholds and windows is fiddly, and it’s telling that Bifrost, a fast Go gateway whose own team wrote one of the better production guides on all of this, does not currently expose configurable circuit breakers as a feature (Maxim AI). When the people who literally wrote the guide haven’t shipped the knob, that tells you it’s harder to operationalize than the diagrams suggest.

The trap that charges the card twice

Here’s the LLM-specific failure that keeps people up at night, and it’s why the “fail before dispatch” property above is not academic.

Picture an agent that just called a tool to charge a customer or send an email. The request reaches the provider, the tool runs, the action happens for real. Then, on the way back, the network hiccups and your client times out before the response arrives. From where you’re sitting, the call “failed.” So your retry logic does its job and fires it again. Now the customer is charged twice, or gets two emails, and nothing in your logs looks obviously wrong (Chanl; Motomtech).

This is what “non-idempotent” means in practice, and it’s the sharp edge that separates LLM agent failover from ordinary API failover. A timeout does not tell you whether the work happened. It only tells you that you didn’t hear back.

There are real mitigations, and none of them are complete on their own. Idempotency keys are the cleanest: you attach a stable key to the operation, and the remote service dedupes on it, so a repeat is a no-op. The catch is that the remote has to support them, and plenty of tools don’t. Local result caching keyed on the exact call arguments works regardless of what the remote supports, because you just refuse to issue a duplicate you already have an answer for. And there’s active research into “effect-aware replay,” where a system replays the recorded outcome of a completed write instead of re-running it (arXiv 2606.19992).

This is also why one tempting latency trick is dangerous for agents. A “hedged request” is a classic move from Google’s Tail at Scale work: when a call is taking too long, fire a second copy to another provider or region and take whichever answers first. It’s genuinely great for cutting the slow tail on read-only calls. But for a tool call that does something, a hedge is just a deliberate duplicate. You are now running the side effect twice on purpose and hoping the second one loses the race. Reach for hedging on reads, keep it far away from writes.

Cross failure domains, not just logos

The most seductive wrong belief in this whole area is that multiple providers automatically equals resilience. Two logos on the architecture diagram, therefore no single point of failure. Right?

October 2025 put that belief in the ground. On the 19th and into the 20th, AWS’s US-EAST-1 region fell over when a latent race condition in DynamoDB’s DNS management cascaded into a roughly fifteen-hour disruption that took EC2, Lambda, ECS, load balancers, and anything depending on them down with it (ThousandEyes; AWS’s own postmortem). If your primary model provider and your carefully chosen backup both happened to run in that region, you had two providers and one failure. The diagram lied to you.

The lesson is that real failover crosses failure domains, not just vendor names. A failure domain is any shared thing whose death takes everything on it down at once: a cloud region, a DNS system, a shared dependency. A good fallback chain is ordered by how independent each rung is. A cheap first hop might be a different region of the same provider. The next rung is a genuinely different vendor. The last rung, for the truly critical path, is something you run yourself (TrueFoundry, June 2026).

And 2026 made this murkier, not cleaner. The same underlying models increasingly run in more than one place. As of April 28, 2026, OpenAI’s frontier models are available on Amazon Bedrock, not just Azure, part of a larger compute deal that ended Azure exclusivity (tech-insider.org). On paper that’s more redundancy. In practice, “the same model on two clouds” only helps if the two clouds don’t share the failure underneath. Whether provider diversification actually buys you independence, in a world where the models span clouds and the clouds themselves cascade, is genuinely unsettled right now. Anyone who tells you it’s solved is selling something.

The parts nobody has fully solved

It would be dishonest to write a “here are the patterns” piece and imply the problem is a checklist. Several of the hardest pieces are open as of July 2026, and you should design knowing that.

Streaming is the ugly one. The moment you start streaming tokens to a user and the provider dies halfway through the response, there is no clean handoff. You are left choosing among bad options: buffer the whole response server-side before showing anything, which gives you clean failover but throws away the snappy feel that made you stream in the first place; let the response restart on the fallback, which is jarring; or pre-flight each provider’s health with a quick non-streamed first token before you commit to streaming from it, which helps but doesn’t cover a mid-stream death (TrueFoundry). Every serious source lands on some version of “pick your compromise.” Nobody claims a clean mid-stream cross-provider handoff, because as far as I can tell nobody has one.

Cross-family fallback is lossy. Falling from Claude to GPT to Gemini is not a drop-in swap. Each family has its own shape for tool calls, its own streaming format, and its own way of carrying reasoning between turns (Zylos; n1n.ai). Something in the middle has to translate live, and the translation is clean for text, mostly workable for tool calls, and genuinely hard for reasoning state. I wrote about that specific wall in more detail in Switching Models Mid-Session Is Easy. Keeping the Reasoning Is the Hard Part. The short version: it works on the common path and frays on the edges, like parallel tool calls and provider-specific reasoning tokens.

A “successful” failover can still be a worse answer. Different models format differently and refuse differently. A prompt tuned for one provider can get outright refused by a stricter one’s safety filters, so teams end up maintaining model-specific prompt variants. Your monitoring says the failover succeeded. The user got a worse or empty answer anyway.

Window sizes don’t match. If you’re passing a big pile of retrieved context and your fallback model has a smaller context window than your primary, the fallback silently truncates or errors (n1n.ai). Your failover target has to be able to hold what you were about to send it, and that constraint is easy to forget until it bites.

The tooling, briefly and honestly

The good news is you don’t have to hand-roll all of this. As of mid-2026, basically every gateway ships some flavor of fallback. LiteLLM offers explicit fallback chains across a hundred-plus providers and is what AWS itself reached for in its own Bedrock resilience demo. Portkey bundles retries, caching, timeouts, and fallback in one layer. Cloudflare AI Gateway added automatic retries in April 2026 and a cross-provider API in May (changelog). Vercel AI Gateway, Helicone, Bifrost, and Kong all play in the same space, Bedrock has native cross-region inference, and LangChain has with_fallbacks() plus a newer fallback middleware, though even that has rough edges (a March 2026 issue reports its fallback wrapper being rejected as a model argument in the high-level agent API, deepagents #2154). OpenRouter routes and load-balances too, though its fallback is more implicit than the explicit chains you configure in LiteLLM or Portkey.

Two honest caveats before you pick one. First, a gateway is itself a new hop in front of every request, which means it’s also a new single point of failure and a bit of added latency. Second, and this matters when you read the comparisons: most of the 2026 coverage of this space is written by gateway vendors. Cross-check any capability claim against the provider’s current docs at the moment you’re evaluating, because these products change monthly and the benchmarks are self-reported.

The deeper limitation is that most of these tools stop at routing the request. They pick which provider gets the call and move on. Very few of them own the context window itself, which is a shame, because the context window is exactly what drives both your bill and that window-mismatch failure mode from the last section. Fallback that ignores the size and cost of what you’re actually sending is only solving half the problem.

Where a gateway that owns the context window helps

This is the gap AJNT is built around, so let me connect it to the patterns above without overselling it, because a few of these map onto AJNT’s design directly.

AJNT’s model routing is the cost-ordered fallback ladder described here, with a live circuit breaker for each provider-and-model pair. The deliberate choice is circuit breakers rather than mid-stream retries, for exactly the reason in the sections above: the breaker gates before the request is dispatched, so it routes around a bad provider without re-issuing a call that might have already committed a side effect. When an upstream degrades, the ladder falls through to the next rung on its own.

Because AJNT sits on the wire as a single gateway speaking every provider’s native protocol, the cross-family translation happens live in the pipeline: a session that started on one model family can be served by another mid-session, with the tool-call shapes and reasoning semantics translated across. That’s the hard, lossy problem from earlier, and it’s genuinely hard, which is the honest framing.

The piece most other gateways skip is the context window. AJNT compresses the resent conversation history before forwarding it upstream, which keeps what actually crosses the wire bounded. That directly addresses the window-mismatch trap: a bounded, compressed context is far more portable across fallback targets with different window sizes, so falling back to a smaller-window model doesn’t quietly truncate your prompt. And a canonical model catalog collapses the many provider-specific aliases of the same model into one entry, which softens the version-pinning problem where a model ID can vanish underneath you.

AJNT is early, so I’m going to resist quoting numbers I can’t stand behind. The point is architectural: these failover patterns are the design, not a feature bolted on the side.

The reframe

If there’s one thing to take from all this, it’s that fallback is not a checkbox you add at the end. It’s a set of decisions. Which of your failures are rate limits versus real outages, and are you treating them differently. Whether your “second provider” actually lives in a different failure domain or just has a different logo. Which of your tool calls can be safely retried and which will charge someone twice if you do. What “success” even means when the fallback model gives a worse answer.

Fail fast and fail forward instead of retrying into a storm. Cross real boundaries, not just vendor names. Know which of your calls have side effects. Everything moves fast enough in this space that any specific claim here has a shelf life measured in months (this is all as of July 2026), but those principles are the part that should still hold when the next round of outages arrives. And there will be a next round.