Stable Stuff First, Question Last

Stable Stuff First, Question Last

Teren Bryson 4 min read [ Ai and llms ] Part 4 of 6

“Buy the ticket, take the ride.” ~ Hunter S. Thompson, Fear and Loathing in Las Vegas

Thompson was most definitely not thinking about API pricing when he wrote that, and while most of the advice in that book is best left on the page, this line is the exception. Prompt caching is a ticket: you pay a little extra up front to write the cache, and it only pays off if you come back and ride it.

In last week’s post we talked about where to put things in a long prompt so the model doesn’t lose what’s in the middle: documents at the top, question last. This week is the same layout for a different reason. It also determines how much you pay each time you send the prompt, which is less fun to think about and much easier to measure.

We already know that every call to a model sends the whole prompt, and you pay for every input token every time, including the forty-page manual that hasn’t changed since Tuesday. Prompt caching is a technique that lets the API pick up from a part of the prompt it has already processed instead of starting from scratch. Anthropic’s docs say it “significantly reduces processing time and costs” for prompts with repeated parts, and that the response is identical either way, so it changes the bill and leaves the answer as is.

The pricing is where it becomes more compelling. Writing the cache costs a little more than a normal call, 1.25 times the base input price for the default five-minute cache. However, reading it back costs a tenth of the base price on most models, a twentieth on Opus 5.5 and Sonnet 5.5, and less still on the Fable and Mythos models. Send a 50,000-token prefix to Sonnet 5.5 ten times, and you pay for it ten times over without caching; with it, though, you pay about 1.7 times, putting you ahead by the second call.

The catch is that a cache hit needs the prompt to match exactly, character for character, from the top down to the spot you mark for caching, which the docs call the breakpoint. The API reads a prompt in a set order (tools, then the system prompt, then the messages), and a change at any level tosses out that level and everything after it. Change one word in a tool description and the whole thing misses.

Any network engineer who has built switch configs from a gold master already knows how this works. The base template is the same on every device, and the per-device variables (hostname, management IP, VLANs) go at the end. Edit a line in the base, and you get to rebuild every config downstream of it. The prompt version of that mistake, used by Anthropic’s docs as their example, is a timestamp or request ID sitting before the breakpoint: it changes on every call, so every call pays to write a fresh cache while never reading one. Move the per-request details below the breakpoint and the problem goes away.

That is what we talked about in last week’s post on layout, more or less: the long, stable material on top, and the question and anything else that changes every call at the bottom. The same order that helps the model remember the middle is also the order that makes caching pay off.

Infographic titled Stable stuff first, question last: a prompt stacked top to bottom, with tools, system prompt, documents and examples above the cache breakpoint and per-request data and the question below it, beside the cost of a 50,000-token prefix over ten calls (10x uncached, 1.7x cached) and a recall curve that dips in the middle.

One important detail, and the thing I’d warn you about, is that caching fails quietly. Below a minimum length (512 tokens on current models, more on some older ones), the request just isn’t cached, and no error comes back. The only way to know is the usage block in the response: cache_creation_input_tokens is what got written, and cache_read_input_tokens is what got read. If both are zero, you’re paying full price, and the API isn’t going to mention it.

Before anyone gets too excited, however: the cache lives just five minutes by default, and the clock resets each time it’s read. So, if your calls come in slower than that, each one pays 1.25x to write to a cache nobody reads, which is 25 percent more on those tokens than not caching at all. There’s a one-hour cache for that case, at twice the base write price, so obviously do the math on your own traffic before you assume it’s cheaper. And none of this makes the answers come back any better or more useful; that was last week’s job.

So, stable stuff first, question last. Buy the ticket, and then make sure you take the ride, because a cache you write and never read is just a more expensive way to pay full price.

Share
comments powered by Disqus

Related Posts