How to Benchmark Context Compression Without Fooling Yourself

Somewhere in your feed this month there is a claim that looks like this: “5x smaller context, same quality.” Maybe it is a new prompt-compression library. Maybe it is a gateway, or a coding agent’s release notes, or a research paper’s abstract. The number is big and round and it is doing a lot of work.

Here is the question that claim almost never answers: same quality on what, measured how?

That gap is the whole story. Compressing a context window has an easy half and a hard half. The easy half is the ratio: count the tokens going in, count the tokens coming out, divide. Anyone can report that, and it is the number that ends up in the headline. The hard half is fidelity: did the model still do the job as well as it would have with the full context? That half is slippery, expensive to measure honestly, and easy to fudge without technically lying. So it usually gets a hand-wave.

This post is about how to measure the hard half. By the end you should be able to design a benchmark that a compression scheme cannot game, and to look at someone else’s single-number claim and know exactly what they left out. As of August 2026 there is no shared, standard way to do this, which is part of why the marketing numbers fly around unchallenged. Let us fix the method, at least for ourselves.

First, “compression” is three different things

Before you can benchmark it you have to say which “it” you mean, because the word covers three techniques that live at different layers and fail in different ways. People conflate them constantly, and the conflation is where a lot of the confusion starts.

Prompt or context compression shrinks the text you actually send to the model. The classic approach is token pruning: run the prompt through a small language model, score how surprising each token is, and drop the low-surprise ones on the theory that they carry little information. This is what the LLMLingua family does (that work is from early 2024, so treat it as background rather than the current frontier). Summarizing old history into a shorter recap counts here too, as does retrieval-style trimming where you keep only the chunks relevant to the current question. All of it is lossy. You are throwing text away and betting the model did not need it.

KV-cache compression shrinks the memory the inference server holds while it runs the model, not the text you send. Techniques like eviction (keep the “heavy hitter” tokens, drop the rest) and quantization (store the cache at lower precision) live here. A good survey from April 2026 walks through the families. If you call a model through an API, this layer is invisible to you. It matters enormously to whoever runs the GPU, and not at all to your prompt.

Agent history compaction is the one most people actually feel, because it is what coding agents do between turns. Claude Code, for example, runs a tiered system that has been reverse-engineered in some detail: before each call it clears out old tool results, keeping only the most recent handful and replacing the rest with a placeholder, and when the session gets heavy enough it spins up a helper to summarize the older turns while keeping the recent ones at full fidelity. A protected “root set” (the system reminders, the project instructions, your latest messages) never gets touched. A source-level writeup puts the trigger for a 200K window somewhere around 167K tokens.

One distinction is worth nailing down before we go further, because it trips up even careful people: caching is not compression. Prompt caching (which Anthropic, OpenAI, Google, and DeepSeek all ship as of 2026) reuses the computation for a prefix that has not changed, so you do not re-pay to process the same opening tokens twice. It does not make the context smaller. It discounts reprocessing an identical prefix. Compression changes the content; caching preserves it byte for byte and just skips the recompute. This matters for benchmarking because the two levers fight each other: the moment you compress or edit the history, you have changed the prefix, and the cache you would otherwise have hit is now invalid. A January 2026 study on caching for long-horizon agents found that dynamic tool results breaking the cached prefix was a real and measurable problem. Any honest benchmark has to decide what it is doing about caching, not ignore it.

For the rest of this post, “context compression” means families (a) and (c): the stuff that changes what the model sees.

Why the ratio lies

Take the cleanest possible claim: “we cut the context by 5x and task performance held.” The ratio is real. The problem is that “performance held” is measured on whatever the author chose to measure, and compression has a nasty habit of passing the test you run while failing the ones you did not.

Here are the failure modes a ratio hides.

It can compress away the instructions. The single most dangerous thing a compressor can do is decide your system rules, your style guide, your “always do X, never do Y” constraints are low-information and drop them. On a question-answering benchmark this barely shows up, because the questions do not depend on those rules. In a real agent session it is catastrophic, and you find out in production.

Where you drop tokens matters, not just how many. The “lost in the middle” effect, first documented by Liu et al. in 2023, is that models attend well to the start and end of a long context and poorly to the middle. The comforting story was that million-token models fixed this. They did not. Someone reproduced the dip on a current model in April 2026: strong recall at the edges, a clear sag in the middle. Bigger windows did not remove the middle, they just made it longer. A compressor that happens to preserve edge tokens and prune middle ones will look fine on some tests and quietly lose whatever lived in the sag.

Single-turn success does not predict multi-turn success. Most compression evals are single-shot: one big prompt, one answer. Real agent sessions are dozens of turns. A May 2025 paper took tasks that models handled fine in one shot, split the same information across multiple turns, and watched performance fall by 39% with reliability dropping much further. A benchmark that only tests single-turn compression is measuring the easy case and calling it the hard one.

Aggregate scores hide the reasoning that broke. Chroma’s context rot study from mid-2025 tested 18 frontier models and found every one degrades as the input grows, non-uniformly, sometimes well before the advertised limit. A 200K-window model can start slipping at 50K. One of their more uncomfortable findings: coherent, well-structured input sometimes degraded the model’s attention more than shuffled input did. Multi-hop reasoning, where the model has to chain several facts, fell off even while simple single-fact retrieval still looked healthy. If your fidelity metric is an average over easy and hard questions, the hard ones drown.

There is a related reality check worth internalizing. The advertised context window is not the usable one. Effective length tends to land somewhere in the range of half to two-thirds of the number on the box. You will see specific figures floating around (a widely shared May 2026 post puts several frontier models well under 40% recall at a full million tokens on a hard multi-round retrieval test). Treat those exact percentages as directional, since they come from blog measurements rather than a controlled study, but the direction is not in dispute and shows up in the peer-reviewed benchmarks too.

The point of all this is not that compression is bad. It is that a ratio without a fidelity number, measured on the workload you actually care about, tells you almost nothing. It is a vanity metric.

Why the benchmarks lie too

So you decide to be rigorous and reach for a standard long-context benchmark. Good instinct, but the benchmarks have their own problems, and knowing them is the difference between measuring something real and measuring an artifact.

The famous one, “needle in a haystack,” plants a sentence in a long document and asks the model to fish it back out. It was useful in 2023. It is now saturated: models score near-perfect on it and still fail real retrieval, because finding one exact string is not the same as reasoning over a long context. A near-perfect needle score is not evidence that a compressor preserved anything that matters.

The deeper problem is that no single benchmark predicts the others. The HELMET benchmark (from late 2024, with a good writeup from Princeton) studied dozens of models across seven categories of long-context task and found the categories barely correlate with each other. Doing well on synthetic retrieval told you little about doing well on retrieval-augmented generation, which told you little about summarization. There is no one task you can run as a proxy for “handles long context well.” You need a spread, and you need it to match your real use.

Then there is a subtler trap that a 2026 benchmark called phi-LongBench put a name to. Most long-context benchmarks use fixed input lengths, which hides where along the length curve a model starts failing. Worse, they conflate two very different things: what the model already knew from training, and what it actually retrieved from the long context you gave it. If a model answers correctly because the answer was in its training data, that is not long-context skill, and it certainly is not evidence your compressor preserved the relevant tokens. The phi-LongBench authors built a score that tries to subtract out baseline knowledge, and when they did, the model rankings shifted noticeably. This is new work and not yet validated by other groups, so hold it loosely, but the underlying warning is solid: if your benchmark’s answers are guessable from general knowledge, you are not testing compression, you are testing trivia.

This connects to a quieter risk: train-on-test leakage. Many long-context benchmarks are built on public datasets that frontier models have very likely seen during training. When the model can shortcut the task from memory, your compression scheme looks better than it is, because the model did not actually need the context you compressed.

The harness

Here is the methodology, written as a spec you can lift. None of it is exotic. It is mostly a refusal to let any single number stand in for the answer. Each item defends against a specific way the naive version fools you.

1. Measure the ratio and the task success on the same sessions. This is the whole game. Not the ratio on your sessions and the quality on someone’s QA set. The same inputs, run twice, compression on and off, and you report both numbers as a pair. A ratio without its matching fidelity number on the identical workload is not a result.

2. Report a curve, not a point. Compression is a dial, not a switch. Run it at several levels and plot task success against compression across the range. What you are looking for is the knee: the point where quality falls off a cliff. A single operating point is almost always cherry-picked to the spot where the tradeoff happens to look good. The curve is honest; the point is a sales slide.

3. Use held-out sessions the model did not train on. Otherwise you are measuring memorization, per the leakage problem above. Fresh, private workloads are worth more than any public benchmark here, precisely because no model has seen them.

4. Use multi-turn, tool-heavy, long-horizon workloads. Single-shot QA is the easy case, and it is not the case compression is for. If you are compressing agent history, benchmark on agent tasks: tool-calling suites like tau2-bench, terminal-driven work like Terminal-Bench, coding tasks like SWE-bench, or long research sessions. The failures that compression causes mostly live in the turns, not in the one-shot.

5. Separate three different fidelity questions. “Did quality hold” is too coarse. Break it into: did the compressor preserve the instructions (the rules and constraints), the retrievable facts (the specific details you might be asked for), and the reasoning-relevant state (the intermediate conclusions the model needs to keep going)? A March 2025 paper on information preservation in prompt compression argues for exactly this kind of multi-axis scoring, using exact match for factual recall and softer metrics for summarization-style content. A compressor can ace one axis and fail another, and a blended score hides it.

6. Control for position. Because of lost-in-the-middle, plant your test targets at the start, the middle, and the end, and score them separately. A compressor that protects the edges will flatter itself if all your probes sit at the edges.

7. Isolate long-context skill from baseline knowledge. Prefer questions that cannot be answered from general training knowledge, so a correct answer is evidence the context survived compression rather than evidence the model got lucky. This is the phi-LongBench warning made practical.

8. Hold caching constant, or measure it on purpose. Since compression invalidates cached prefixes, a benchmark that ignores caching can credit compression with savings that a plain cache would have delivered for free, or charge it with latency that caching would have hidden. Decide your caching policy, state it, and keep it the same across the compression-on and compression-off runs. If cost is part of your claim, model caching and compression together, because in production they interact.

9. Make it reproducible, and stamp the date. Fixed seeds, a published task set, and the exact provider and model version for every run. This space moves monthly. A compression result from a model version that shipped in the spring may not replicate on the summer release, and if you did not record which version you used, nobody (including future you) can tell whether the method changed or the model did.

One honest caveat threads through all of this. Scoring fidelity on open-ended agent trajectories, where there is no single gold answer, is genuinely unsolved. Task-success rate is the least-bad proxy we have, and it is coarse: it tells you whether the session reached the goal, not how much cognitive room to spare it had. If someone tells you they have this fully figured out, be skeptical. The right move is to say plainly that the fidelity axis is approximate, and then measure it anyway, because approximate-and-stated beats precise-and-fictional.

What a good result actually looks like

Put the spec together and the shape of a trustworthy compression result stops being “a big number” and becomes something more specific. It is a curve with a clearly located knee, measured on realistic multi-turn sessions the model has not seen, with the fidelity axis broken into instructions, facts, and reasoning state, with position controlled, with the caching policy pinned and stated, and with enough version detail that someone can rerun it next month and know whether they are comparing like with like. The headline stops being “5x” and becomes “here is how far we can push it before the task starts to break, on work that looks like yours.”

That is more work than dividing two token counts. It is also the only version that survives contact with a skeptic. Right now the field does not have a shared harness for this, which is exactly why every new compression paper ships its own task set and its own peak-token numbers that do not compare to anyone else’s. The methodology above will not fix the field, but it will keep you from fooling yourself, and it will let you call the bluff on a claim that will not show you its curve.

Where this leaves AJNT

We build a context-compression layer, so we have to hold ourselves to this too, and it is worth being clear about how.

AJNT is a gateway that sits in front of whatever model your agent already calls, and it compresses the conversation per session, continuously, as the session grows. The design choice that matters for this post is that the compression is measured and billed as savings: the amount we shrink your context is not a marketing figure, it is a line on the invoice, the tokens you did not have to pay a foundation model to reprocess. That is one axis of the curve from the harness above, made concrete.

The honest thing to say is that the savings axis is the easy half, the same easy half everyone can report. The half that decides whether compression is worth doing is the other axis, task fidelity, which is the one this whole post argues gets fudged. Because AJNT is wire-compatible and drops in without touching your agent code, you are well positioned to run the real test yourself: take your own sessions, run them with compression on and off, and measure the difference in whether the agent actually finished the job. Same sessions, both numbers, the way item one of the harness demands.

And the boundary, stated plainly, because the method above cuts both ways: compression does nothing for a short single-turn call, and little for a tight retrieval setup with no accumulated history to trim. The whole premise is that there is a growing conversation worth compressing. When there is not, the honest curve is flat, and you should expect us to say so.

Do not ask a compression scheme how much it compresses. Ask it what that cost you, and make it show you the curve.