What a context window forgets

Every agent framework ships a knob for this. Keep the last twenty messages, summarise what falls off the end, and move on. The knob is set once, usually to whatever the quickstart used, and after that nobody looks at it again. An assistant that has been booking trips for the same person since April is having a conversation that no longer fits in a prompt, and that knob decides what it is allowed to remember.

So I measured what the knob throws away.

The measurement

LoCoMo is a set of ten conversations that run for 19 to 32 sessions each, released with the ACL 2024 paper on very long-term conversational memory. What makes it useful here is that every question comes with the ids of the turns that answer it. That lets you take a context policy, hand it a budget and a question, and ask a narrow question back: is the turn that carries the answer still in there?

What that measures is a ceiling. A model handed the right turn can still answer wrongly, so a policy that keeps the evidence has only cleared the first bar. The reason to measure the ceiling anyway is that it costs nothing. No API calls, no judge, no variance between runs, and a policy that has already lost the evidence is not worth taking to a full evaluation.

One number sets the scale. An average LoCoMo conversation is 23,056 tokens. The evidence for an average question is 84 of them. Four tenths of one percent of the transcript carries the answer, and every context policy is a guess about which four tenths.

Line chart of evidence recall against context budget on LoCoMo for five context policies. At 2,048 tokens a long-term store reaches 63% and a sliding window 8%. The two search-based policies stay far above the window until 32,768 tokens, where every policy holds the whole conversation. Session summaries flatten at 13%. — Five policies over 1,527 questions. The two that read the question before choosing sit twenty points above the rest until the budget is large enough to keep everything.

Eight percent against sixty

At 2,048 tokens, which is 8.9% of the conversation, a sliding window holds the evidence for 7.7% of questions. Keeping the opening of the conversation as well as the end lifts that to 8.2%. Searching the transcript against the incoming question reaches 59.5%, and a long-term store searched at sentence level reaches 62.7%. Same budget, same conversations, seven times the hit rate.

The other way to read it is in tokens. Search reaches 50% recall on a budget of 1,024. The window needs 16,384 to get there, sixteen times the context for the same result, and every one of those tokens is paid for on every turn.

Where the window actually dies

The averages hide the shape of the failure. Split the questions by how many sessions sit between the evidence and the question, and the window does not decay. It falls off a cliff. In the session in progress it holds everything. One or two sessions back it holds 62.3%. Three to five sessions back, at the same 2,048 tokens, it holds 0.0%, while search is still at 72.1%.

That is zero of the 140 questions in that bucket. Widening the window does not rescue it either: a budget that reaches five sessions back reaches the whole conversation, at which point nothing is being managed and every turn pays for all of it.

The part search does not fix

Broken out by question type at the same budget, the best policy holds the evidence for 76.8% of single-hop questions and 16.2% of multi-hop ones. A multi-hop question needs several turns from several sessions at once, and one query against a store tends to bring back one of them. Partial credit is higher, so it is not that nothing is found; it is that all of it has to be found together and usually is not.

This is the number I would want on the table before anyone claims a memory system works. A single averaged recall figure over a mixed question set moves with the mix, and the multi-hop share is where the difficulty actually lives.

There is a second catch in the same direction. The long-term store wins by 3 points over plain search, and it wins because it stores sentences rather than whole turns, which fits more of the conversation into the same budget. But the complete evidence turn survives only 3.9% of the time. The rest of its wins hand the model one sentence out of several, and whether that sentence is the one carrying the answer is not something this measurement can see. Whichever column you believe, it has to be the same column for every policy.

Running it on your own transcripts

The code is a small package with no runtime dependencies:

pip install retainkit

Conversations go in as JSON, with turn ids, session numbers and the questions you care about:

retainkit eval conversations.json --budget 2048 --budget 8192

It prints recall against budget for each policy with a bootstrap interval, the split by evidence age, and the cheapest budget at which each policy reaches a recall target. A policy is any object with a select method, so a scheme you already run in production goes into the same sweep as the built-in ones and gets compared on the same questions.

Two limits apply before you use the numbers. An abstractive summariser rewrites what it keeps, and a rewritten sentence cannot be traced back to a turn, so the summary policy here is extractive and an LLM summariser is outside what this can score. The token counts are also a four-characters-per-token estimate, which moves a window policy's absolute recall by up to 22 points, though not the ordering. Pass your own tokenizer when the budget has to be exact.

In most stacks that knob still holds whatever the quickstart put there. An afternoon of measurement will say what it is throwing away.