A Diary It Rereads Every Time: Context Windows and LLM "Memory"

A Diary It Rereads Every Time: Context Windows and LLM "Memory"

Teren Bryson 10 min read [ Ai and llms ] Part 5 of 6

“Memory, my dear Cecily, is the diary that we all carry about with us.” — Miss Prism, in Oscar Wilde’s The Importance of Being Earnest (Act II)

Here, Miss Prism is a governess who once wrote a three-volume novel and has no sense of irony about it, so it’s fitting that the line belongs to her and that Cecily, her student, gets the better reply. I’m saving Cecily for the end. For now, take Prism at her word, because “memory as a diary” is, with one important caveat, about the best description I’ve found of how a language model remembers anything. The caveat is that the model doesn’t carry the diary around. Somebody hands it the diary at the start of every request, and it has to read the whole thing before it writes a word.

I closed part four by promising we’d have a conversation about, well, the conversation itself, one of the hungrier tenants of a GPU’s memory, and here it is. A language model has no memory in the sense you’d expect or be used to, and nearly everything that looks like memory is text being resent, summarized, searched, or stored by the software around it. Unfortunately, some of this gets back into that damned math again; I’ll show the math, say what it means, and you’re welcome to skip straight to the explanation, which I almost assuredly would.

The model forgets everything

Remember from part four that once training ends, the weights are frozen, and nothing you type changes them. So when you come back to a chat, and the LLM seems to remember your last question, it isn’t because it learned anything. The application sent the earlier conversation again, along with your new message, and the model read it all as one fresh prompt. Every turn of every chat is a cold start with a very long note stapled to it.

The ceiling on how much fits in that note is the context window, counted in tokens, and it has to hold your prompt and the model’s reply together. Llama 3’s largest model, for example, supports a window of up to 128K tokens . A rough rule of thumb for English is about three-quarters of a word per token, so that’s about 96,000 words, which is roughly the same as a long novel. The model didn’t start out with that capacity, either. According to the Llama 3 paper, the team grew the window gradually in six stages, starting from 8K tokens , training on longer sequences at each step, so a long window is something a model has to be taught to handle, one stage at a time.

Why length costs so much

Attention, from part one , compares every token with every token that came before it. Double the length of the conversation, and you roughly quadruple the comparisons. That alone would hurt, but the cost that bites hardest in practice is memory. To avoid recomputing the keys and values for every earlier token on every step, the system stores them in the KV cache from part four, the ticket rail in our kitchen analogy, and that ticket gets longer with every token in the conversation. The diary, it turns out, lives on the rail.

The math here uses Llama 3’s published dimensions, because Anthropic hasn’t published Claude’s. The paper’s architecture table gives the 405B model 126 layers, 8 key-value heads, and a model dimension of 16,384 , split across 128 attention heads, so each head is 128 numbers wide. For every token, the cache stores one key and one value in every layer, which is 2 × 126 × 8 × 128 numbers. At 2 bytes each, that’s 516,096 bytes, or about half a megabyte per token. At 8,000 tokens, the cache holds about 4 GB; at a full 128,000 tokens, it’s about 66 GB, for one conversation, which is most of an 80 GB GPU before you’ve stored a single weight. Those are my calculations from the paper’s figures, assuming 16-bit storage and ignoring overhead, so check my work, as I’m frequently wrong on these math things.

The 8 shared key-value heads, a design choice I mentioned in part one, come from a trick called grouped-query attention . If the model kept a separate key and value for each of its 128 attention heads, the cache would be 16 times larger, or roughly a terabyte at 128,000 tokens, which is the kind of number that makes a finance department ask very pointed questions, while staring at you like you have two heads. The diary is heavy, which is why nearly every recent model shares or compresses its keys and values somehow. DeepSeek’s latent attention squeezes them further, and Google’s Gemma 3 lets most of its layers look only at recent text: according to its technical report , it uses five local layers, each limited to the last 1,024 tokens, for every one layer that sees the whole window, specifically to keep the KV cache from exploding with long contexts.

Chart of KV-cache size for one conversation with Llama 3 405B: it grows about half a megabyte per token, reaching about 4 GB at 8,000 tokens and 66 GB at 128,000, under one 80 GB GPU. Without shared key-value heads it would fill that GPU at about 10,000 tokens and reach about 1 TB by 128,000.

All of these figures come from open models whose makers have published their designs, and that’s a big caveat. Closed models, Claude included, may use some of these techniques, all of them, or something else entirely, and I really can’t tell you which. Someone smarter than me may be able to reverse engineer the answer somehow, but then again, maybe not. It’s all a bit convoluted without good documentation.

Longer isn’t better

You’d expect a model to use a long window the way you or I would use a long document, reading all of it and pulling out whatever matters. The evidence says it’s patchier than that. The paper Lost in the Middle tested models on questions where the relevant passage sat at different positions in a long input. They did best when the answer was near the beginning or the end, and noticeably worse when it was buried in the middle, even models built for long contexts. It’s a familiar failing; I’ve done the same thing with every terms-of-service agreement I’ve ever scrolled past. Also, I wrote about it recently, so a certain sense of deja vu may be permeating the room.

Chart from the Lost in the Middle paper: GPT-3.5-Turbo answered 75.8% correctly when the useful document came first among 20, 57.2% at fifth, 53.8% at tenth, 55.4% at fifteenth, and 63.2% at twentieth. With no documents at all it scored 56.1%, so in the middle it did worse than with nothing.

That paper is from 2023, and models have been retrained many times since, so I wouldn’t lean on the exact shape of its curve. Later work points in the same direction, though. RULER , a 2024 benchmark, found that every model it tested claimed a window of at least 32K tokens, but only about half of them still performed acceptably at 32K. NoLiMa , from 2025, made the test harder by phrasing questions so they didn’t share obvious words with the answer, and 11 of the 13 models it tested, all advertising 128K windows or more, fell below half their short-context scores at 32K. Benchmarks aren’t your task, and the newest models may do better than anything in those papers; I can’t promise you either way. What all three papers agree on is that a window is a capacity spec: it tells you how much the model will accept, not how well it will use it.

I wrote about the practical side of this in the Prompting Claude series, in a post on long documents that covers the lost-in-the-middle problem and a quote-first trick that helps with it, so I won’t repeat it here.

What stands in for memory

Since the model can’t remember, the software around it does the remembering. Four approaches are fairly common, and most products combine several. The proprietary models may even have more tricks than these. It’s hard to know.

  1. Summarize: when a conversation nears the limit, older turns get replaced with a shorter recap. It’s like tearing out the diary’s early pages and pasting in a one-page summary; you keep the gist of it all, but lose the detail, and the summary itself was written by a model that can get things wrong.
  2. Retrieve: instead of cramming everything into the prompt, the system searches a store of documents and pastes in only the passages that look relevant. The technique, called retrieval-augmented generation, was laid out in a 2020 paper by Patrick Lewis and colleagues , which paired a language model with a searchable index of Wikipedia passages. Think of it as an index at the back of the diary, which is more than most diaries have. I’ll be writing an entire piece on RAG, so stay tuned for that.
  3. Cache: if the start of your prompt is the same every time (a long set of instructions, for instance), the provider can keep the KV cache it already computed for that part and reuse it instead of redoing the work. Anthropic’s prompt-caching documentation describes it as letting the API “resume from specific prefixes in your prompts,” with a cache entry lasting five minutes by default and reads billed at a fraction of the normal input price. This one follows straight from part four: it’s the line cook keeping the onions already chopped for the last order instead of chopping them again, as long as the next order comes in quickly.
  4. Keep notes: some LLMs now keep a persistent store of facts about you and read the relevant parts back into the prompt at the start of each session. Claude’s memory feature works that way from the user’s side: it keeps a summary of your projects and preferences that you can view and edit in your settings. For developers, Anthropic offers a memory tool that lets the model read and write files that outlast a single conversation. Either way, the model hasn’t learned anything; it has been handed a better-organized diary. But the trick is convincing, and it’s easy to start thinking that the model is more aware than it actually is.

Each of these has a failure mode that leads back to the diary. Summaries drop details, retrieval can grab the wrong passage and the model will quite happily use it anyway, and stored notes can be dead wrong, stale, or applied somewhere they certainly don’t belong. So in every case the memory is only as good as whoever or whatever wrote it down.

The effective window

So treat a stated window size as the most the model will accept, and test it on your own work before you trust it. A model that accepts 128,000 tokens might handle your 20,000-token contract well, or it might drop a clause somewhere in the middle; the only way to find out is to try. If the answer matters, put the important material near the start or the end, repeat it if you have to, and check what comes back against the source.

Cecily’s reply

As for Cecily, she answers Prism on the spot with what I think is one of the better descriptions anyone has written of a language model’s relationship with its own diary, and Wilde wrote it more than a century before there were any:

“Yes, but it usually chronicles the things that have never happened, and couldn’t possibly have happened.” — Cecily, in Oscar Wilde’s The Importance of Being Earnest (Act II)

That’s the problem of hallucination in one sentence. The model reads its diary (the prompt) and then writes the next entry (the reply). Sometimes the entry is accurate. Sometimes it’s very fluent, confident, and all about a conversation that never took place. Why that happens, and what these models are good at in spite of it, is the subject of the next post in the series. Stay tuned for that.

Share
comments powered by Disqus

Related Posts