Why Message 50 Costs More Than Message 5

You are forty messages deep in a coding session. The task is basically done. Your last message was “yes, do that.” And somehow that two-word reply cost you more than the detailed, carefully written request you sent back at message five.

That is not a billing bug. It is the most misunderstood fact about how these tools work, and once you see it you cannot unsee it: the model has no memory. Every single turn, it re-reads the entire conversation from the beginning. So message 50 is not paying for message 50. It is paying for messages 1 through 49, all over again, plus itself.

That one mechanism drives most of what people find baffling about their agent bills. Let me walk through why it happens, why the obvious fixes only half-help, and where the real leverage actually is. I will be honest about the parts that are genuinely fuzzy, because this space moves monthly and a lot of confident numbers floating around are softer than they look.

The model does not remember anything

Here is the part that trips everyone up. When you chat with a coding agent, it feels like a conversation with someone who is following along. Ask a follow-up and it knows what “that function” refers to. It seems to remember.

It does not. A transformer, the kind of model under Claude, GPT, and Gemini, is stateless between API calls. It does not keep your conversation in its head between turns. Each time it responds, the tool bundles up the entire history so far, sends the whole thing back to the model, and the model reprocesses every prior token from scratch to rebuild its working state before it writes the next word. (That working state has a name, the KV cache, and the way processing cost grows with length is well documented in the inference literature.)

So there is no running log the model is quietly remembering. There is only the text, resent in full, re-read in full, every turn. And you pay for it in full every turn, because everything sent up counts as input tokens, and input tokens are billed on every call.

That is the whole trick. Message 50 costs more than message 5 because at message 50, the model has 49 messages of history to chew through first, and you are footing the bill for all of it again.

The math: a little each turn, a lot in total

Let me make the shape of this concrete, because it is worth getting right.

At turn N, you are billed for roughly N turns of accumulated history. So the cost of each individual turn climbs, more or less, in a straight line as the session goes on. Turn 10 costs about twice what turn 5 did. Turn 40 costs about eight times what turn 5 did. Nothing dramatic yet.

The drama is in the running total. Add up the cost of every turn, 1 plus 2 plus 3 all the way to N, and you get the triangular sum, which is close to N squared over two. That is the quadratic feeling people describe. Double the length of a session and you do not double the total bill, you roughly quadruple it. The cost does not grow with how much you typed. It grows with the square of how long you kept going.

One honest caveat, because I do not want to overstate it. On the provider’s side, the raw attention compute genuinely does scale quadratically with length, O(L squared), as a 2025 survey of long-context inference lays out. But that is their compute, not your invoice. What lands on your invoice is a per-input-token price. The quadratic you feel is the running total of a linearly-growing per-turn cost, not the attention math itself. The effect is real either way. I just would rather you understand it than repeat a half-right version of it.

Tool output is the fuel, not your typing

Here is where it gets practical, especially for coding agents. The thing filling up all that history is almost never your prose. It is tool output.

Every file the agent reads, every command it runs, every response from a connected tool gets appended to the conversation in full and never summarized. You asked it to skim a 3,000-line file at turn 8? Those 3,000 lines are now part of the conversation. And they will be resent and re-read at turn 9, and turn 20, and turn 40, right up until something clears them out. You are paying to re-read that file dozens of times even though you only cared about it once.

This is why agent sessions balloon in a way normal chat never does. Vantage modeled a 50-turn coding session in April 2026 and found input tokens outweighed output by something like 25 to 1, with input running around 85 percent of the total cost. A separate audit of 30 engineering teams by LeanOps put re-sent context at roughly 62 percent of the bill, the single largest line item. Treat those as two firms’ findings rather than universal laws (nobody has published a rigorous, peer-reviewed number here, which I will come back to), but they point the same direction: the bill is dominated by history you are paying to resend, and tool output is what stuffs that history.

There is a scale twist worth naming. When you run a coding agent, you are burning your own budget, and a long session is an annoyance you can feel in your invoice. When a company ships an agent inside a product, a support bot or an onboarding assistant, that same cost curve fires once per user, per session, at whatever scale the product reaches. At that point it is not an annoyance. It is cost of goods sold, sitting right against gross margin, and no user is going to remember to clean up the context for you.

”But doesn’t caching fix all this?”

This is the strongest objection, and it deserves a real answer rather than a wave.

Prompt caching is genuine and it is big. The idea is that if the start of your request is identical to one the provider saw recently, it can reuse the work instead of reprocessing from scratch, and it bills you a fraction of the normal rate for that reused portion. As of mid-2026, Anthropic charges cache reads at about 10 percent of the normal input price, a 90 percent discount, and reading a cached block resets its clock so a busy session keeps it warm. OpenAI extended its cache retention from a few minutes to up to 24 hours in May 2026. Google’s Gemini turns implicit caching on by default. For something like a support bot with a fixed 100,000-token system prompt hit thousands of times a day, caching genuinely flattens the curve. That is real, and I am not going to pretend otherwise.

Now the catch. Caching is a prefix match, and it is keyed on the exact bytes of your request plus the exact model version. It only helps for the longest run of text at the start that is byte-for-byte identical to before. The moment the request diverges, everything after that point is a cache miss and gets billed at full price.

And agent conversations diverge constantly. Every turn appends a new tool result in the middle of the history. Edit a single tool’s description and you invalidate the entire tool list that follows it. A timestamp updates, a username changes, the tools come back in a different order, and the cache breaks right there. The stable, cache-friendly prefix a support bot enjoys is exactly what a working coding agent does not have. So the real-world savings on a dynamic, tool-heavy loop land well below the headline 90 percent. How far below? Honestly, nobody has published a rigorous cache-hit study on real agent workloads. It is one of the genuinely open questions here.

The fair summary: caching helps most where the prefix stays still, and least exactly where agents live.

”But the context window is a million tokens now”

The other thing people reach for is the size of modern context windows, and it is true that they got huge. Anthropic dropped its long-context surcharge and now serves its full million-token window at standard rates, with Opus 4.6 hitting general availability at that size in March 2026. Most frontier models are at a million tokens standard now. (One exception to keep in your back pocket: OpenAI still charges a premium, roughly double for input, above 272,000 tokens in a request, so on their models crossing that line makes the whole thing pricier.)

But a bigger window does not repeal any of the math above. It just hands you more rope. Filling the window is precisely what costs money, and it turns out it is also what makes the model worse.

That second part surprises people. Chroma tested 18 frontier models in July 2025 and found every single one degrades as the input grows, often losing 30 to 50 percent of its accuracy well before it reaches the advertised limit. A model that claims 200,000 tokens can start getting noticeably worse at 50,000. The practical rule people have landed on is to plan for maybe 40 to 50 percent of the advertised window as genuinely usable. As one 2026 writeup put it, you end up paying the context tax twice, “once in cash, since every token in context is billed as an input token on every turn, and again in degraded responses.” The session gets more expensive and dumber at the same time, for the same reason.

What actually helps

So what do you do about it? The options fall into two piles, and the difference between them matters.

The first pile is manual habits. Clear the session between tasks. Run a compaction command to summarize and shrink the history. Trim your config file. Switch to a cheaper model for the boring parts. These all work. The problem is that they are things a human has to remember to do, in the moment, mid-task. And they are simply unavailable to a product agent running autonomously once per user. There is no human in that loop to hit the clear button.

The second pile is automatic. Tools like Claude Code, Codex CLI, and OpenCode now fire auto-compaction when the window gets around 85 percent full, summarizing the transcript and replacing the full history with the summary. That helps, though it is lossy: people report a kind of compaction amnesia where an earlier instruction quietly falls out of the summary. Pruning stale tool results is probably the highest-yield move available, precisely because tool output is what dominates. And the research frontier is goal-aware compression, work like Context Codec that tries to deliberately preserve the goals, decisions, and results that matter instead of blindly truncating, though whether that reliably beats plain summarization is still an open question.

There is one structural point worth ending on. A lot of the tooling people assume covers this does not. Gateways and routers like OpenRouter, LiteLLM, and Portkey decide where a request goes, which provider, which model, which fallback. That is useful. But routing does not shrink what gets sent. The context window, the thing actually driving the bill, sails through untouched. Most of the stack sits next to the problem without laying a finger on it.

Where this leaves you

The honest fix is to bound what actually crosses the wire per session, automatically, without asking anyone to change how they work or how their agent’s code is written. The client keeps behaving normally, resending its whole history the way agents inherently do, while what genuinely travels upstream stays bounded. That is the problem AJNT is built around: a wire-compatible gateway that compresses the resent context and routes each request down a cost-ordered ladder, all behind a single endpoint you point your existing tool at. No new SDK, no rewrite. Sessions can run far past a single provider’s context ceiling without you babysitting anything, and because the fix lives at the gateway it applies to every session, including the autonomous product ones where no human is around to run a cleanup command.

But strip away the pitch and the takeaway stands on its own. Message 50 costs more than message 5 because of how the machine is built, not because you did anything wrong. The model has no memory, so it re-reads everything every turn, and the total grows with the square of how long you keep going. Caching softens it where your prompt holds still. Bigger windows just give you more room to make it worse. The cost curve is real and it is structural, which means it belongs to infrastructure to solve, not to you to remember.