What Is Context Rot? Why Bigger AI Context Windows Make Models Worse

Feed a frontier LLM a two-hop reasoning question buried inside a long context, and its accuracy can fall from 99% to 70% just because the surrounding text got longer. Not because the model lost the information. Because it had too much of it. That is context rot, and as of July 15, 2026, it is the single biggest reason AI agents fail in production. This piece explains what context rot is, why it happens, where it bites hardest, and what the teams shipping reliable agents are doing about it.

Context rot, defined

Context rot is the degradation of an LLM’s reasoning quality as its input context grows, even when the model can retrieve every relevant token perfectly. It is not the model running out of window space. A GPT-4.1 or Claude 4 with a million-token window still rots. The window just gets bigger, and so does the region inside it where the model is unreliable.

Chroma’s 2025/2026 Context Rot study tested 18 frontier models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. Performance dropped non-uniformly and unpredictably as input length grew. The finding that matters: expanding context windows does not produce linear improvements in accuracy. It produces a longer runway on which the model can lose focus.

“The longer the context an LLM works with, the less accurate it becomes.” — Redis engineering blog, 2025

Why does context rot happen inside a Transformer?

It happens because attention is a finite resource, and the Transformer spends it on every token in the window whether that token matters or not.

Every token in a Transformer’s context has to compare itself against every other token. That pairwise attention is what lets the model reason across a document. As context grows, the same attention budget gets spread thinner across more comparisons, and the signal for any single important token weakens. Researchers writing at Understanding AI describe this as a real tension between context size and attention focus. You cannot fix it by adding more tokens. You are the reason it is getting worse.

Three mechanisms show up in the literature:

• Attention dilution: the model’s focus gets spread across too many tokens, and important ones stop standing out.
• Entropy growth: as more content enters the window, the distribution of “what might matter next” flattens, and the model’s predictions get noisier.
• Representational collapse: internal token representations start to blur together, so the model treats distinct pieces of information as more similar than they are.

A 2025 arXiv paper put this bluntly: even when models retrieve all the relevant information perfectly, their performance on math, question answering, and coding still degrades as context length grows. Retrieval is not the bottleneck. Reasoning under load is.

The “lost in the middle” problem

A 2023 Stanford study first documented the shape of the failure. Researchers placed 20 retrieved documents (roughly 4,000 tokens) into an LLM’s context and moved a target answer through different positions. Accuracy sat at 70 to 75% when the answer lived at position 1 or position 20. It fell to 55 to 60% when the answer was buried in the middle. A 15 to 20 point drop, driven entirely by position, not content.

That U-shaped curve is why long agent sessions rot in a predictable way. Early instructions and the most recent tool output get attention. Everything the agent decided in the middle of a 30-turn conversation slowly stops mattering to it. The agent contradicts itself. It reintroduces a pattern it rejected 15 turns ago. It forgets a naming convention it set for itself an hour in.

The Needle in a Haystack test lies to you

For two years the industry evaluated long-context models with the Needle In A Haystack (NIAH) benchmark: hide a sentence in a long document, ask the model to find it. Models pass this easily. That is the problem.

NIAH tests lexical retrieval, not reasoning. Chroma’s research and NVIDIA’s RULER benchmark both make the same point: simple needle retrieval masks the actual reasoning collapse happening at longer contexts. The concept that replaces it is the Maximum Effective Context Window (MECW), which is task-specific. A model with a stated 1M-token window might have an MECW of 800k tokens for retrieval, and 40k tokens for multi-hop reasoning. In one experiment cited by Understanding AI, asking a model to chain two facts (Semper Opera House is in Dresden; Dresden is in Saxony) pushed some models from 99% accuracy down to 70% as the surrounding context grew.

If your product involves aggregation, sorting, or multi-hop logic, the number on the spec sheet is not the number you actually get.

Where context rot hits hardest: coding agents and multi-agent systems

Coding agents rot fast because they generate their own noise. Every file read, every tool call, every failed test run, every debug print dumps tokens into a shared window. After 20 to 30 turns the signal-to-noise ratio has collapsed, and the agent starts making the kinds of mistakes a junior developer makes on hour four of a long day: reintroducing deleted code, ignoring constraints set at the start of the session, forgetting naming conventions it defined itself.

There is a structural point here that gets missed. Enterprise codebases were never going to fit in a context window, and pasting more of the codebase in does not help. As CloudGeometry argues, a codebase is a graph. Flatten it to a token sequence and you lose the structure that made it comprehensible. More text is not more understanding.

Multi-agent systems were supposed to be the escape hatch. Split the work across agents, and each one keeps a fresh, focused window. They do, sort of. But per Mem0’s research on multi-agent memory, 36.9% of multi-agent system failures come from inter-agent misalignment, which is a structural memory problem, not a model quality problem. Multi-agent setups can also burn up to 15 times more tokens than a single agent. The rot moves from inside one agent’s context into the handoffs between agents.

Fixes that actually work

The industry name for the fix is context engineering: deciding, at each step, exactly what the model should see. It is broader than prompt engineering and broader than RAG. RAG is one retrieval technique inside it.

LangChain’s Write, Select, Compress, Isolate framework is a decent scaffold:

• Write: move information out of the live context into external memory, so the agent can retrieve it when needed instead of carrying it forever.
• Select: pull only the relevant pieces back in at each step. Not the whole history.
• Compress: summarise or prune older turns before they degrade the window.
• Isolate: hand narrow subtasks to sub-agents with fresh contexts, so pollution stays contained.

Three tactics inside that framework are doing the most work in shipped systems:

1. Just-in-Time Context Retrieval (JIT-CR). Instead of preloading a big document, the agent holds a reference (a file path, a document ID) and fetches content only when it needs it. Anthropic’s Claude Code combines pre-computed retrieval with JIT exploration using primitives like glob and grep, letting the agent navigate its environment rather than swallow it whole. Cursor IDE’s hybrid retrieval reported up to 12.5% higher accuracy using a similar blend.

1. Sub-agents. A sub-agent takes a narrow task, runs in a fresh window, returns a clean result, and terminates. The parent never sees the noise. This is the isolate step made concrete.

1. The scout pattern. A cheap, fast sub-agent pre-screens candidate content for relevance before it enters the main window. Unlike vector similarity, the scout uses actual LLM judgement about utility, which catches cases where semantically close content is still useless for the task at hand.

One caveat on caching. KV-cache optimisation can cut costs by roughly 10x, but it does nothing for rot. It makes the degradation cheaper. Not better.

Algorithmic work is happening too. LongRoPE extensions try to preserve the attention patterns the model learned during pretraining as the window grows. Hybrid global-local attention schemes and “full-complex” attention variants (using the imaginary component of RoPE as parallel attention channels) show gains in benchmark long-context retrieval. These help. They do not repeal the underlying tension.

The security angle: memory poisoning

Once you push information out of the live window into external memory, you have created a new attack surface. Memory poisoning is a persistent attack that corrupts an agent’s long-term memory so that every future session inherits the bad state. It is not prompt injection, which is transient. It is contamination that survives across conversations.

According to MintMCP’s research, attack success rates jump from a 40% baseline to over 80% when an agent consults its memory before responding. In multi-agent systems, poisoned memory in one agent can spread to others through shared knowledge bases.

The mitigation is dynamic, task-scoped least privilege for agents. Traditional role-based access control breaks here because agents are non-deterministic: they infer which tools might be useful and can reach beyond their intended scope. AWS’s Cedar-based multi-agent authorisation design makes the point that authority must survive delegation. When agent A hands to agent B, the originating user’s scope has to travel with the request, or permissions quietly widen every hop.

What to do with this

Stop treating context window size as a proxy for capability. It is not. A model with a 1M-token window that rots at 40k tokens on multi-hop reasoning is, for your agent, a 40k-token model. Measure the MECW for the tasks you actually run. Assume rot starts around turn 20 in a coding agent and design for it: sub-agents for narrow work, JIT retrieval instead of preloaded dumps, a scout to filter what enters the window, and external memory with provenance tracking and short-lived scopes. The teams shipping reliable agents in 2026 are not the ones with the biggest windows. They are the ones deciding, at every step, what not to put in them.