Reading the Whole Cookbook to Write One Word: How Inference Works

“Bandwidth problems can be cured with money. Latency problems are harder because the speed of light is fixed—you can’t bribe God.” — “an old network saying,” quoted in David Patterson’s “Latency Lags Bandwidth” (2004)

Anyone who’s ever had to explain to an executive why the shiny new 10-gig circuit to Singapore didn’t make the application one bit faster has lived that saying. You can buy a fatter pipe; you can’t buy a smaller planet. That gap, between the bandwidth you can buy and the latency you can’t, explains a good deal of what’s odd about the way a chatbot answers you, starting with the way the words arrive one at a time.

I closed part three with a million people wanting answers to insipid questions at once, and that’s where we pick up. Training, the subject of part two and most of part three, is one enormous bill, paid once per model. Inference is the meter that never stops running: every question anyone asks the finished model is a small job that has to start quickly and finish before the person waiting on it wanders off to do something else. Some of this gets technical, as usual; I’ll translate as I go, and I’ll tell you where my own understanding gets precariously thin, which it does in more places than I usually care to admit.

Dinner service

Once training ends, the roughly 405 billion numbers we’ve been following since part one are frozen, and nobody touches them inappropriately again. Inference runs the model forward only, the same multiply-and-add loop from part one, with no grading and no correcting. Each token takes far fewer calculations than a training step, which sounds suspiciously like good news, until you notice that the job has changed shape. Training is one huge job, and the only question anyone cares about is how quickly it finishes, or at least, how quickly it appears to finish. Inference is millions of small jobs that arrive anytime someone asks a question, each with a human waiting on the other end with all the optimism of a golden retriever. The provider has to get the first words out fast, keep the rest coming at a steady clip, and do all of it cheaply enough to stay in business.

Back to the kitchen, then. If training was culinary school, all ruined sauces and somebody shouting at you, inference is a Friday-night dinner service with the tickets printing faster than anyone can read them, staffed by a cook with a peculiar handicap. To put each word on the plate, this cook has to walk to the back, read the entire cookbook cover to cover, walk back, and then do it all again for the next word. That’s a hell of a lot of reading for one word, and absurd as it sounds, it’s close to what the hardware does. The analogy has a hole, as they all do: the cook isn’t forgetting the recipes so much as working at a counter far too small to hold the entirety of the book. The chip’s fast memory holds a sliver of the model at a time; the rest sits in slower memory next to it and has to be streamed through, every time. Your conversation rides along too: the model’s running notes on every earlier word get read again for each new one, which is part of why long conversations cost more. (More on that below, and in part five. Yes, that means more math. Sorry.)

Prefill and decode

Every answer comes out of two phases that behave nothing alike. The first, called prefill, is the model reading your prompt. All of those tokens already exist, so the GPU can work through them in parallel and do a great deal of math for every byte of the model it hauls out of memory; this is the cook taking in the whole ticket at a glance. The second, called decode, is the model writing its answer one token at a time, because each new token depends on the ones before it. The chip can’t start on word ten until word nine exists, and every one of those words means another trip through every layer of the model, which is to say another walk to the back for the whole cookbook.

Decode is where Patterson’s saying comes due. The math that follows is mine, not a published benchmark, so check my work, or, you know, nod, smile, and move on to the next bit of non-math, which is what I usually favor. Take a generic 70-billion-parameter model stored at 16 bits per number, which comes to 140 GB of weights. To produce one token, the hardware has to read essentially all of that from memory. Spread the weights across two of the H100 SXM GPUs from part three, each rated by Nvidia at 3.35 TB/s of memory bandwidth, and you have about 6.7 TB/s to work with, so one pass takes roughly 21 milliseconds. That caps a single user at about 48 tokens per second, no matter how fast the multiplication units are. Even that number flatters the hardware, since it ignores the extra memory traffic from the key-value cache (more on that shortly) and assumes the work splits perfectly between the two chips. Real systems do much worse. Helmuth von Moltke the Elder, the Prussian field marshal, made the point in 1871 in a line since boiled down to “No plan survives first contact with the enemy.”

Meanwhile, the math units spend most of that time standing around waiting for data, like line cooks watching the door for a delivery truck stuck in traffic, which happens quite frequently, I am reliably told by people in positions to know these things. Patterson saw the same pattern twenty years ago, at a smaller scale. His keynote from the same year put numbers on it: over 20 to 25 years, by his count, bandwidth improved somewhere between 120 and 2,200 times across four technologies, while latency improved only 4 to 20 times. His four were processors, memory, networks, and disks, not GPUs, so I’m not claiming his numbers carry over. The shape of the problem does, and if you’ve spent any time staring at a slow WAN link, you’ve seen it before.

I learned that lesson the hard way setting up a site in Shanghai over a 256 kb link back in 2010. Well, I learned a few lessons from that episode, and not all were written about. The bandwidth-delay product (BDP) used to bite a lot harder, before TCP window scaling and autotuning became the default. But the technical difficulties of overcoming latency in both long-distance circuits and local systems remain. Pretend you’re a frustrated software developer trying to use SCP to move an 800MB file from somewhere in Europe back to mainland U.S., and wondering why the shiny new DS3 (45 Mbps) is barely an eighth full. With about 100 milliseconds of round trip and only 64 KB allowed in flight at a time (the TCP window without scaling, and older versions of SCP capped their own window at about the same size), the transfer tops out around 5 Mbps, so a file the pipe could move in under three minutes takes closer to twenty. Yeah, I was there, Gandalf, thousands of years ago.

Fewer trips to the back

Because decode is limited by memory rather than math, much of the special sauce in serving these models comes down to making fewer trips to the back, or getting more out of each one. The simplest trick is batching, which amounts to cooking for the whole dining room at once: serve many users together, and one pass through the weights advances every one of their answers by a token, so they split the cost of the trip. The catch is that memory has a second tenant, the key-value (KV) cache. That’s the model’s running notes on each conversation, the keys and values from the attention step in part one, stored for every earlier token so they don’t have to be computed again. In kitchen terms, it’s the ticket rail, and every ticket gets longer as its answer does (again, context windows matter).

The authors of PagedAttention point out that each request’s cache is large, grows as the response does, and gets wasted through fragmentation, which limits how many requests can share a GPU. Their fix borrows paging from operating systems, an old idea put to a new job, and the systems person in me finds that more satisfying than is probably healthy. FlashAttention works on a related problem inside the chip: it computes exactly the same attention result as the standard method while shuffling less data between the chip’s slow memory and its fast memory. How it pulls that off involves rearranging the math in ways that run past where my tourist’s grasp of the language gives out, so I’ll stop at what it does, before I dig myself into a hole.

My favorite is speculative decoding , because it comes with its own sous-chef. A small, cheap model guesses several tokens ahead, and the big model checks all of those guesses in a single pass, keeping the ones it agrees with and tossing the rest. The checking is cheap for the same reason prefill is cheap: the guesses already exist, so the big model can look at them all at once. The original paper reported speedups of two to three times on a large T5 model, with output identical to what the big model would have written alone.

And DistServe argues that prefill and decode get in each other’s way enough that they belong on separate GPUs, each pool tuned for its own job, the way a busy kitchen keeps the prep cooks out from under the line. All of these results come from papers, though, and papers show a technique at its best, on models its authors chose. What a production service actually gets depends on its traffic, and I can’t see anyone’s traffic from where I’m sitting.

Two jobs, same hardware

Set the two workloads side by side, and it’s easier to see why the same GPUs get deployed so differently. These are typical differences, not universal ones.

This is where I’m back on home turf. Training traffic, as part three covered, is enormous and predictable: big synchronized exchanges, with thousands of chips waiting on the slowest link. Inference traffic is closer to the reverse: smaller, burstier, and far more sensitive to delay, much like interactive traffic over the WAN links of old. For mixture-of-experts models, which hand each token to a few specialized sub-networks (the “experts”) that can live on different GPUs, it could also mean a steady patter of small exchanges as tokens hop from one GPU to another. That last part is my own reasoning, not something I’ve seen measured publicly, so treat it as a hypothesis until someone measures it. Like the data centers we built and optimized for WAN/LAN traffic (north-south and east-west) back then, today’s data centers have to be built to optimize the traffic patterns their builders expect to see. You can’t optimize one for training and expect it to be just as good at inference, or vice versa.

Where the savings go

If all these tricks make each token cheaper, shouldn’t the total bill come down? Sometimes the price per token does. The savings have a way of getting spent, though: cheaper tokens invite longer prompts, bigger models, and agents that call the model dozens of times to finish a single task. I don’t have a clean dataset showing how much total inference spending has grown, so all I have is anecdote: among my colleagues and me, the faster and cheaper inference gets, the more work we find to throw at it. Somebody noticed the same thing about software in general three decades ago, and they get the last word.

Latency can’t be bribed

So an old network saying describes a chatbot about as well as it ever described a transatlantic circuit. Add GPUs, and the service as a whole can handle more people at once. Nothing you can buy makes the next token arrive before the previous one exists, though, or gets a memory chip to hand over its data faster than physics allows. Money cures the bandwidth problem, more or less; the latency problem stays wherever God left it. And whatever speed the engineers manage to win back, the rest of us are already finding ways to spend, which is the point Niklaus Wirth made in 1995, when he offered, “with a touch of humor,” a law he credited to his colleague Martin Reiser:

“Software is getting slower more rapidly than hardware becomes faster.” — Martin Reiser, as quoted by Niklaus Wirth in “A Plea for Lean Software” (1995)

Next up is the conversation itself, one of the hungrier tenants of that memory. The longer the context, the bigger the cache, and the more it costs the model to remember what you said five minutes ago. It’s also a good place for a shameless plug for my Prompting Claude series: short, picture-heavy posts on managing the conversations you’re having with your LLM of choice. Most of it carries over to other models, and it applies to API, SDK, and CI pipeline work as much as to live chat sessions.