Switching Models Mid-Session Is Easy. Keeping the Reasoning Is the Hard Part.

Picture the session you’re most annoyed to lose. You’re forty minutes into a coding task with Claude Code or Codex. The agent has read a dozen files, run the test suite twice, and finally has the whole problem loaded into its head. Then your provider rate-limits you, or has an outage, and the run stops cold.

The obvious fix is to keep going on a different model. Someone else’s GPU is up, so route to it and continue. This is the one thing that, right now, mostly does not just work.

Not because nobody has tried. Half a dozen open-source proxies exist specifically to translate one model’s API into another’s, and the simple demos look great. The trouble shows up in the bug trackers: sessions that hang because a tool-call ID has the wrong shape, requests that get rejected with a 400 over a missing cryptographic signature, weaker models that start typing out fake tool calls as plain text. Almost all of it traces back to one thing that translates cleanly for text, sort of translates for tool calls, and does not translate at all for the part that matters most: the model’s reasoning.

That’s the piece worth understanding before you count on failover as a feature. Text and tool calls are the easy 80 percent. Reasoning state is the hard 20, and it’s hard for a reason that isn’t going away.

Three protocols, one conversation

Start with what a “session” actually is on the wire, because the three big families don’t agree on it.

When a coding agent talks to a model, it isn’t sending a chat log. It’s sending a structured document: the system prompt, every prior message, every tool the model is allowed to call, and every result those tools returned. The model reads all of it and appends its next move. The exact shape of that document is the “wire protocol,” and each family designed its own.

Anthropic’s Messages API treats a message as an array of typed blocks. A block is text, or a tool_use (the model asking to run a tool), or a tool_result (what came back), or a thinking block (the model’s reasoning). Everything is explicitly labeled by type. Anthropic’s extended thinking docs describe this layout in detail.

OpenAI’s Chat Completions API is the older, widely-copied shape. The assistant message carries a tool_calls array, each call with an id and a function name plus its arguments as a JSON string. Results come back as separate messages with role: "tool", matched up by that ID. There’s no dedicated slot for reasoning between turns. It gets generated, then discarded.

OpenAI’s Responses API, which is what Codex CLI speaks, is newer and shaped differently again. Instead of a list of messages it’s a list of typed items, including reasoning items that each have their own ID. It can carry reasoning forward across turns, either by referencing the previous response by ID or by passing an encrypted blob of it back. OpenAI’s own reasoning-items cookbook walks through both.

Google’s Gemini uses functionCall parts, and as of the Gemini 3 models it attaches a thought_signature to them, which its documentation calls “an encrypted representation of the model’s internal thought process.”

Same conversation, four genuinely different data models. To move a live session from one to another, something in the middle has to read one shape and re-emit it in another, turn after turn, without the client noticing. For most of the document, that’s tedious but doable. For one part, it’s a wall.

The easy 80 percent: text and tool calls (mostly)

Plain text is nearly free to translate. A paragraph is a paragraph. Move it from a Claude text block into an OpenAI message and you’re basically done.

Tool calls look like they should be almost as easy, because on the surface they’re all just JSON describing “call this function with these arguments.” They are not almost as easy, and the ways they break are instructive.

The IDs alone will get you. Claude requires tool-call IDs to match a specific pattern, ^[a-zA-Z0-9_-]+$, and other models don’t follow that rule. When one open-source harness tried to continue a session from Kimi onto Claude, the Kimi-shaped tool-call IDs didn’t satisfy Claude’s regex and the whole session just hung (openclaw #7107, February 2026). The maintainers closed it “not planned.” The only real workaround was starting over.

The subtler failure is worse because it looks like the model getting dumber. When another user switched a session from a GPT Codex model to a smaller Gemini Flash model, the previous provider’s tool-call formatting leaked into the conversation history. The weaker model saw that formatting, and instead of actually calling tools, it started typing out text that mimicked the tool-call syntax (openclaw #15043, February 2026). It looked like it was working. It was writing fan-fiction of a tool call. Injecting a disclaimer into the prompt telling it not to do that wasn’t enough to stop it.

None of this is unsolvable. Careful translation of IDs, argument encoding, parallel-call structure, and tool schemas handles it. The point is that “careful” is carrying real weight. This is the part a good gateway can actually get right, but “it’s just JSON” undersells how many small, sharp edges are hiding in there.

The hard 20 percent: reasoning that can’t leave home

Here’s the part nobody has solved, and it’s structural, not a matter of someone writing better translation code.

Every frontier family now attaches reasoning to its tool-calling loop as an encrypted, provider-bound artifact. Not readable, not portable, and in some cases mandatory.

Claude’s thinking blocks carry a signature field. A June 2026 writeup of the Messages streaming format describes it plainly: the full thinking content is encrypted into that signature, “which is used to verify that thinking blocks were generated by Claude when passed back to the API.” During tool use you have to hand the complete, unmodified thinking block, signature and all, back on the next turn, or the request is rejected. You can pass it back. You cannot open it, edit it, or forge one.

OpenAI’s encrypted reasoning is the same idea with a tighter lock. The encrypted reasoning content is, in the words of one bug report tracing a failure, “cryptographically bound to the originating organization” (pydantic-ai #4608, 2026). It breaks not just across model families but across OpenAI organizations. A separate proxy project hit 400 invalid_encrypted_content just moving between OpenAI and an Azure-hosted OpenAI endpoint (CLIProxyAPI #1793), because the reasoning couldn’t be verified anywhere but where it was born.

Gemini turns the lock into a hard requirement. Omit the thought_signature on the first functionCall part of a turn and the request fails with a 400, per Google’s docs. That is exactly the error a popular router hit trying to translate a Claude session into Gemini: 400 missing thought_signature, with no way to synthesize a valid one (claude-code-router #1431, June 2026, still open at writing).

Why is it built this way? Honestly, for good reasons that just happen to be your problem. Encrypting reasoning supports zero-data-retention promises, makes it harder to distill a frontier model by harvesting its chain of thought, and lets the provider verify the reasoning actually came from their model and wasn’t tampered with. It’s a feature for the provider. It’s a wall for anyone trying to carry that reasoning somewhere else.

If you want the academic version of “this is unsolved,” it exists. A paper published in April 2026, LLM-Rosetta, built a genuine cross-provider translation layer with a neutral intermediate format, and it treats reasoning as a first-class thing to carry. It still concedes that “provider-specific features like signatures lack universal equivalents.” Its round-trip mode, staying within one provider, is lossless. Its actual cross-provider mode intentionally drops the signatures and encrypted reasoning, because there’s nothing on the other side to translate them into. The current state of the art for crossing a family boundary is not “transfer the reasoning.” It’s “drop it cleanly.”

And dropping it isn’t free. OpenAI’s own cookbook found that keeping reasoning items in the loop between tool calls was worth about 3 percent on the SWE-bench coding benchmark and pushed cache-hit rates way up (their figure, measured within a single provider). So the reasoning is load-bearing. When a cross-family switch throws it away, you’re paying some version of that cost. Nobody has cleanly measured how much when the boundary is crossed, which is its own honest gap.

Two more taxes: streaming and cache

Two smaller problems ride along with the big one.

Streaming is the first. Each family streams its response as a live sequence of events, and the grammars don’t match. Anthropic emits a typed sequence, message_start, then content blocks, then deltas that are themselves typed (thinking_delta for reasoning, text_delta for text, signature_delta for that signature arriving), then stop events. OpenAI streams flatter chunks. The Responses API streams item events. A gateway translating a session live can’t just forward bytes. It has to re-emit the stream in the exact event grammar the client expects, or the client’s parser trips over an event it was never written to handle. It’s invisible when it works and a hang when it doesn’t.

Cache is the second. Prompt caches are scoped to a specific model, so the moment you switch, the cache you’d built up is gone and the new model reprocesses the context from scratch. That’s a cost, not a crash, but it’s real. The tempting fix, summarizing or compacting the history at the switch point to clean up the foreign formats, has its own trap: compaction can itself break the encrypted reasoning-item IDs that a session depends on, which is a failure people have hit in practice. The neat instinct to “just summarize at the boundary” is a landmine of its own.

So should you even switch?

There’s a camp that says the answer is simply no, and they have a case worth taking seriously.

The argument, laid out in pieces like MindStudio’s “Never switch models mid-conversation”, is that a switch throws away the warm cache, feeds the new model a context full of another model’s style and structure that it never would have produced itself, and breaks reasoning continuity, all at once. A March 2026 study put numbers on it (arXiv 2603.03111): switching models mid-conversation swung quality by an amount the authors described as “comparable to upgrading between full model tiers,” and about 70 percent of the variance came down to two things. How constraining the first model’s style was, and how much the second model leaned on context it hadn’t generated. Treat those exact figures as one study’s result via a secondary summary, not settled fact, but the direction is clear enough: a session that starts on a given model tends to beat one that switches into it.

For the voluntary case, that’s basically decisive. Switching mid-task just to reach a slightly better model is usually a bad trade. The consistency of the context matters more than the marginal capability you’re chasing. If the current model is doing fine, leave it alone.

The involuntary case is a different question, and it’s the one that actually comes up. Your provider goes down. You hit a rate-limit wall. You blow through a cost ceiling. Now the choice isn’t “switch versus stay optimal.” It’s “continue on a degraded-but-working model versus stop dead.” A session that keeps going with dropped reasoning and a cold cache is worth a lot more than a session that hard-fails, because the alternative to degraded isn’t perfect, it’s nothing.

One nuance the skeptics mostly leave open: their tests lean on free-form chat, where a model’s prose voice carries much of the context. Coding-agent sessions are heavier on file contents and command output and lighter on one model’s writing style, so the switching penalty might be smaller for agents than the chat studies suggest. That’s plausible. It’s also unproven, and I’d rather say so than pretend the coding case is settled.

What a gateway can actually do about it

Once you accept that reasoning state can’t cross the boundary, the job gets clearer, and more honest. The goal isn’t teleporting Claude’s signed thinking into GPT’s encrypted reasoning. That’s not on offer from anyone. The goal is two things: translate everything that does translate without corrupting it, and make the forced switch survivable instead of fatal.

This is the layer AJNT sits in. It takes the three wire protocols, Anthropic Messages, OpenAI Chat Completions, and the OpenAI Responses format Codex speaks, into a single internal pipeline, and translates reasoning semantics, tool-call shapes, and streaming formats across families live. That’s the whole earlier list handled in one place instead of hand-rolled per project: the tool-call ID formats that hang sessions, the schema shapes, the streaming grammar the client has to receive in its native form.

For the involuntary case, the routing ladder is the actual answer to “my provider had a bad day.” Instead of one hardcoded endpoint that fails when it fails, each request walks a cost-ordered list of provider-model options with a live circuit breaker on each one. When an upstream degrades, the request falls through to the next rung automatically. It uses circuit breakers rather than retrying mid-stream, so a tool call that already ran doesn’t get fired twice with duplicate side effects.

What a gateway does not do, and it matters to say it, is make the reasoning-transfer wall disappear. Nobody’s product does that, because the wall is cryptographic and intentional. What it can do is turn the boundary from a 400 or a hung session into a clean, deliberate handoff, drop the reasoning that can’t travel, faithfully carry the text and resolved tool results that can, and keep the client talking to one stable endpoint the whole time. That’s the realistic win. Anyone promising more than that is selling you past what the APIs allow.

The bottom line

Switching models mid-session is easy to demo and hard to do well, and the hard part has a specific name: reasoning state that was deliberately built not to travel. Text moves. Tool calls move if you’re careful about the sharp edges. Reasoning gets dropped at the border, by design, and it’s load-bearing enough that dropping it costs you something.

So know which switch you’re making. Reaching for a marginally better model mid-task is the move the skeptics correctly shred. Surviving an outage or a rate limit is the move that’s actually worth engineering for, and there the right expectation is a clean, continued-but-degraded session, not a magic one where nothing was lost. Judge any gateway by how gracefully it handles that boundary, not by whether it claims to erase it.

All of this is accurate as of July 2026. Model versions, API shapes, and provider behavior in this space change on something closer to a monthly cycle, so reconfirm the specifics before you build on them.