AI Context Is Not Memory: Why Agents Need to Forget

Dynamic context pruning scores each context piece for relevance, dropping low score content and compressing mid score content. Segment level memory retrieval groups conversation history by topic, preserving coherence. Context rot describes accuracy degradation from irrelevant information. Production systems now filter, rank, prune, and summarise context on every turn to manage costs and improve accuracy.

1. The follow up question

In my previous post I explained why context is not free, and why a smaller AI model carrying a huge context can sometimes cost more to serve than a larger model with a short one. The response I heard most often, in different words each time, was the same question. If context is expensive, why not just keep what is useful and drop the rest.

That question points at a distinction that is easy to blur but worth making explicit before going any further. Memory and context are not the same thing. Memory is the potentially enormous external store of everything an agent has learned or encountered, its documents, past conversations, tool outputs, and working history. Context is the small working set selected from that memory for the task in front of it right now. Treated this way, the emerging engineering problem maps cleanly onto ideas any systems engineer already knows well. Memory is storage. Selecting what to bring in is retrieval, similar to paging. Context itself is the working set. Pruning is eviction. Summarising is compression.

That mapping is where AI context management research and tooling are heading in 2026, and it is the frame this post uses throughout.

2. Dynamic context pruning

One practical implementation looks like this. Every piece of context, whether a past message, a tool output, or a retrieved document, is scored for relevance to the current task before it ever reaches the model. Content that scores low gets dropped entirely. Content that scores in the middle does not get discarded outright, it gets passed through a cheaper model and compressed into a short summary, preserving the essence of the information without carrying its full token cost.

This is not the only way to prune, and the research is not converging on a single architecture. Provence treats pruning as a sequence labelling problem and dynamically determines which portions of retrieved context to remove, rather than applying a fixed low score, medium score, high score pipeline. What the different approaches share is the underlying principle, that relevance should be decided before generation rather than assumed by default.

A useful concrete example of what this buys you in practice comes from Squeez, which looked specifically at coding agents that repeatedly consume large tool outputs such as shell command results or file listings. Its fine tuned pruning model reportedly removed 92 percent of tool output tokens while retaining a recall of 0.86 and an F1 score of 0.80 on its benchmark. In plain terms, the agent did not need the 20,000 tokens a shell command just returned. It needed roughly 1,600 of them. That is context hygiene made concrete rather than theoretical.

3. Segment level memory retrieval

Long running agent conversations rarely stay on a single topic, which is precisely the frustration behind the original question. Two recent approaches, SeCom and DyCP, both address this but from different angles, and the distinction between them is worth keeping straight.

SeCom builds better units of memory. It explicitly constructs a memory bank out of topically coherent segments, so the underlying structure of what gets stored is organised by topic from the outset. DyCP works differently. It operates outside the model and dynamically identifies which sequential stretch of past dialogue is relevant to the current turn, without requiring predefined topic boundaries or offline memory construction. Put simply, SeCom decides how memory should be organised, and DyCP decides which coherent stretch of the past matters right now. Both outperform naive turn level retrieval, which the DyCP research shows can disrupt coherence by ignoring how turns depend on each other within a topic, and both outperform dumping the entire history into the prompt.

4. Context rot is real, and it is a stronger result than it first sounds

The case for managing context well is not purely financial. A June 2026 paper, Diagnosing and Mitigating Context Rot in Long-horizon Search, evaluated four flagship models across three deep search benchmarks and identified what the authors call premature termination. As accumulated context grows, models increasingly give up outright or produce uncertain, incorrect answers, well before they exhaust the context window itself. The degradation is not a hard limit being hit, it is a behavioural failure that gets worse the longer the context runs.

An even more striking result comes from a separate paper, Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. Across five models and math, question answering, and coding tasks, the researchers found performance degrading by between 13.9 and 85 percent as input length increased, even when the model retrieved the relevant information perfectly. The failure held even when irrelevant tokens were replaced with plain whitespace, and even when the model was forced to attend only to the relevant tokens directly. Related work on distracting retrieved passages found accuracy drops of six to eleven percentage points depending on the model, simply from the presence of irrelevant material in the prompt.

Together these results say something stronger than the intuitive claim that irrelevant information confuses the model. They say that length itself, independent of relevance and independent of retrieval quality, can degrade performance. Some teams now refer to this as context rot. The implication is that pruning context well is as much an accuracy strategy as it is a FinOps one, and the two goals point in the same direction rather than trading off against each other.

5. Where this is heading

Across the current research, a clear engineering pattern is emerging. Context is becoming a managed resource rather than an ever growing transcript. That means deliberately filtering, ranking, pruning, summarising, and isolating information on every turn, instead of defaulting to the largest window the model will accept.

For anyone running agents in production today, the practical version of this is a task scoped context window that compacts or resets between distinct tasks, automatic summarisation of resolved threads, and relevance scoring applied to tool outputs before they are appended to the running context. None of this requires retraining the underlying foundation model, though some of the approaches above do involve a separate, purpose trained model for the pruning or compression step itself, Provence and Squeez both work this way. What it requires more fundamentally is treating context the way any performance sensitive system already treats working memory, as a resource to be actively managed rather than a drawer to be filled and never emptied.

We spent several years trying to give models bigger context windows. The harder engineering problem turns out to be deciding what belongs in them at all.

References

1. LogRocket Blog, “The LLM context problem in 2026: strategies for memory, relevance, and scale” – https://blog.logrocket.com/llm-context-problem/
2. “DyCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs” – https://arxiv.org/pdf/2601.07994
3. Zomer Gregorio, “Implementing Dynamic Context Pruning for Long-Context LLM Agents” – https://zoms.vercel.app/blog/implementing-dynamic-context-pruning-for-long-context-llm
4. Zylos Research, “LLM Context Window Management and Long-Context Strategies 2026” – https://zylos.ai/research/2026-01-19-llm-context-management/
5. Chirkova et al., “Provence: efficient and robust context pruning for retrieval-augmented generation” – https://arxiv.org/pdf/2501.16214
6. “Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents” – https://arxiv.org/pdf/2604.04979
7. Xia et al., “Diagnosing and Mitigating Context Rot in Long-horizon Search” – https://arxiv.org/abs/2606.29718
8. Du et al., “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval” – https://arxiv.org/abs/2510.05381