Skip to main content
Zubnet AI › Learn › Wiki › Context Rot
Using AI

Context Rot

Also known as: Long-Context Degradation
The drop in a model's accuracy and reliability as its context window fills up, even well before the official limit.

Why it matters

Context rot is the main reason context engineering exists. It's tempting to solve every problem by stuffing more into the prompt: all the documents, the whole codebase, the entire chat history. But more context can make results worse, not better, and it always makes them slower and more expensive.

It's especially important for agents. A long-running agent accumulates tool outputs, file contents and its own past reasoning. After a few dozen steps, the context can be mostly noise, and the agent starts forgetting instructions, repeating work or losing track of its goal. Much of the craft of building agents is keeping that context clean.

Deep Dive

A model might advertise a context window of a million tokens, but that doesn't mean it uses every token equally well. As the input grows, models tend to miss details, mix up facts, follow early instructions less closely and make more mistakes. The term "context rot" spread in 2025, notably after a report from the vector database company Chroma tested 18 leading models and found that performance dropped as input length grew, even on simple tasks.

It's not a hard cliff. It's a gradual decay, and it starts earlier than most people assume.

Why It Happens

  • Attention is spread thin: every token competes for the model's attention. The more tokens, the harder it is for the relevant ones to stand out, especially when distractors look similar to the answer.
  • Training mismatch: models see far fewer very long examples during training than short ones, so their long-context skills are less practiced.
  • Position effects: research on the "lost in the middle" effect showed models often recall information at the beginning and end of the context better than in the middle.

Why Benchmarks Hide It

The classic needle-in-a-haystack test hides one odd sentence in a long document and asks the model to find it. Many models score nearly perfectly, which made long context look solved. But real tasks are harder: the "needle" resembles the surrounding text, the model must combine several facts, or it must reason over the whole input. Tests built that way show much steeper drops.

How to Fight It

  • Retrieve, don't dump: use RAG or search tools to bring in only the relevant parts instead of everything.
  • Compact and summarize: have agents periodically summarize their progress and drop raw tool output they no longer need.
  • Put key instructions where they're seen: important rules near the start (in the system prompt) and the specific question near the end.
  • Use sub-agents: hand a sub-task to a fresh agent with a clean context and only bring back its result.
  • Keep tool outputs lean: return the useful 20 lines, not a 5,000-line log.
  • Use external memory: write notes to files and re-read them when needed, rather than keeping everything in context.

A Rule of Thumb

Treat the context window as working memory, not storage. The advertised maximum is the most you can fit, not the amount that works best. For accuracy-critical tasks, test your model at the context sizes you'll actually use.

← All Terms
ESC