Make It Show Its Work First

Ever been to a really long meeting? You tend to remember the opening bits and the closing action items and conclusions, and the middle blends into a jumbled summary in your memory. As it turns out, LLMs do close to exactly the same thing with a long prompt: they “forget” the middle. It’s one of several foibles the models we’ve built so far share with us humans, and it’s generally referred to as the “lost-in-the-middle” problem. Say you paste three vendor contracts, about ninety pages in all, into one prompt and ask which one has the shortest termination notice period. The clause you need is on page 23 of the middle contract, which is right where the model is weakest.

Somebody wrote a paper about this. In 2023, Nelson Liu and colleagues at Stanford and elsewhere published “Lost in the Middle” , which is one of the more straightforward titles in the field. They found that a model’s accuracy is often highest when the information it needs is hanging around at the beginning or end of a long input, and that it “significantly degrades” when that information lives in the middle, even on models built specifically for long context. Plot accuracy against where the answer sits, and you get a U-shape, with page 23 of the middle contract right at the bottom.

Before anyone points out the obvious: that paper tested 2023-era models, and long-context recall has improved since. How much of the U survives on a current Claude model is something to test on your own prompts, and I’d rather not blindly stab at it from a 2023 paper. Anecdotally, though, the problem still shows up with long prompts.

By long, I don’t mean you typing a long question into the chat box. I mean a prompt with dozens of pages of documents, code, or logs pasted in, to review a codebase or compare contracts or whatever; Anthropic’s docs draw the line at about 20,000 tokens. If the pile is too big to paste in at all, the usual answer is retrieval-augmented generation, or RAG: a system searches a document store and pastes in only the passages that look relevant. I’ll get to RAG in a future post. For now, assume everything fits in one prompt.

Anthropic’s docs recommend a fix for prompts that long, which is the chart below: long documents on top, a name on each one, the relevant quotes pulled out before the answer, and the question at the very bottom. Their own 2023 test is the most concrete number I have: with the relevant quotes pulled into a scratchpad first and examples in the prompt, Claude 2’s accuracy went from 0.939 to 0.961, which Anthropic describes as 36 percent fewer errors. Quoting first helped in every head-to-head comparison they ran, at a small cost in latency.

The quote step is the one I’d keep if I could keep only one, because it gives me something to check. If the quote says ninety days and the answer says thirty, I know which one to distrust, and I didn’t have to reread ninety pages to find out.

I left one number off the chart on purpose. The docs also say that putting the question last can improve response quality “by up to 30 percent in tests,” but they don’t say compared to what, or on which model. The 2023 numbers come from Anthropic’s own test too, on a model that’s long since retired, but at least they come with a baseline.

So make it show its work first. That way, the important stuff from the middle of the meeting stands a better chance of actually getting used. Unless it was a really boring meeting, in which case I can’t really help you. Maybe go with Douglas Adams on this and flag a passing spaceship.