Fail Better, Fifteen Trillion Times: How an LLM Is Trained

“Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.” — Samuel Beckett, Worstward Ho (1983)

Silicon Valley apparently adopted Beckett’s line as a motto, their north star, somewhat like putting a tombstone on a welcome mat. As Colin Marshall points out in a piece for Open Culture , the original passage is much darker than the startup reading, and Beckett likely saw his own work as a failure. I’m using it anyway because it is the most accurate one-sentence description of how a language model gets built that I’ve found, and it’s still much better overall than whatever methods I use to learn, never mind whatever quotes would come out of that basket of sadness.

In part one of this series, we ended with a model whose ~405 billion numbers were arranged correctly and meant nothing. Before training, those numbers are random, and the model’s output is noise. Training attempts to turn the noise into something useful through one rather boring move: guess the next token, get graded, nudge every number a hair in the direction that would have made the guess better, and repeat. Iteration in action. The cleverness is in the scale, the grading, and what happens after the basic lessons. I’ll endeavor to tackle them in that order. As in part one (minus the eye-watering math), I’m writing this all for a reader who is comfortable with numbers but not necessarily with the kind of math a physics professor might have on their blackboard, and that still leaves some people out; I apologize, truly I do, but the floor on how far this stuff can be simplified is real.

The ingredients

Start with text, a lot of it—preferably not racist, misogynistic, or otherwise against modern sensibilities. That’s a choice, though, and each company does it slightly differently. Meta’s paper on Llama 3 says its flagship model was pre-trained on 15.6 trillion tokens, each token being around 0.7 of an English word , the pieces of words we met in part one. Nobody is enough of a masochist to feed that in raw. The data gets collected, duplicates are removed, and filters hopefully toss out the crap, a job done in part by classifier models that score the text. Anthropic’s latest system card describes the same ingredients this way: public internet data—a major point of contention, and the subject of lawsuits—public and private datasets, and synthetic data generated by other models, cleaned with “deduplication and classification.”

The weak point here is the word “crap.” Somebody decides what counts as good text, and those decisions (made by people and by classifiers trained by people) shape what the model knows and how it writes. So, model biases are very real. The labs treat their filtering recipes as competitive secrets, so outside observers can see the size of the pile and little of the curation. That limits how much I, or any outsider, can tell you about the pile that built Claude specifically. I keep coming back to Claude in examples, by the way, because I use it extensively and it’s ubiquitous enough that most people reading this will recognize it.

One job: guess the next token

Think of the model as a student taking an enormous fill-in-the-blank test. It sees a stretch of text, covers up the next token, and predicts what comes next. The grade is called the loss, and the usual version, cross-entropy, is a measure of surprise: if the model gave the true next token a high probability, the loss is small, and if it gave the true token almost no probability, the loss is large. A model with random weights spreads its bets evenly over the whole vocabulary, so with 128,000 possible tokens it assigns the right one a probability of about 1 in 128,000. In the natural-log units most papers use, that’s a starting loss of roughly 11.8. The goal of training is to drive that number down.

The nudging comes from calculus, but the idea is the same as the children’s game of hot and cold. After each batch of text, an algorithm called backpropagation works out, for every one of the 405 billion numbers, whether turning it up a little would make the loss go up or down, and by how much. An optimizer then takes a small step in the “less surprised” direction for all of them at once. A widely used choice is AdamW , which adjusts the step size for each number separately and carefully pulls the weights toward zero so none grows out of hand. The game is played in 405 billion dimensions instead of two, and it is played trillions of times. Yeah, big numbers. And now you’re starting to get an idea of why these systems use so much computing power (and electrical, water, etc.)

You’re right to be suspicious that this works at all. Why should predicting the next word force anything resembling knowledge? The honest answer is that you can’t predict the end of “The capital of France is” without something that works a little like knowing it, and by extension, you can’t predict the next line of a legal brief or a block of Python code without absorbing a good deal about law or code. That’s an argument from the results, though, not a proof, and researchers still disagree about how much of what the model does deserves the word “understanding.” I’ll return to that in part six, much to the inevitable delight of those who think AI is going to kill us all (the popular zeitgeist fuels panic in disturbing ways).

How much compute: the scaling laws

Why do labs keep building bigger? Because the improvement is, within limits, predictable. A 2020 paper by Jared Kaplan and colleagues, Scaling Laws for Neural Language Models , found that the loss falls smoothly as you increase model size, data, and compute, across more than seven orders of magnitude. Two years later, a DeepMind team showed in the Chinchilla paper that many models of the era were too big for the data they’d seen: a 70-billion-parameter model trained on about four times more data beat Gopher, at 280 billion parameters, on the same compute budget. The lesson was to grow data and model together, which, in retrospect, seems self-evident, but hindsight being what it is, someone had to write it down.

A common rule of thumb says that training costs about six floating-point operations per parameter per token (okay, I lied; there is some math in this part as well). Plug in Llama 3’s numbers, and you get 6 × 405 billion × 15.6 trillion, or about 3.8 × 10²⁵ operations, which matches the 3.8 × 10²⁵ figure in the paper . I did the multiplication myself (well, I used my laptop calculator, and it still took several tries to get it right), and I was pleased it matched the published number. We’ll turn that figure into racks, power, and network traffic in part three (all you datacenter-haters rejoice—we’re going to eat Robin’s minstrels).

The concession here is that loss is a grade on the fill-in-the-blank test, and a lower loss does not automatically mean a more useful assistant. Scaling laws tell you how the test score moves. They say nothing about whether the model writes a coherent or cogent email.

Finishing school

A model that has only finished pre-training is an enormously well-read stranger with no manners, a bit of a blowhard as it were. Ask it a question, and it may continue with three more questions, because that is what the text it saw tended to do. Turning it into an assistant is the job of post-training, and the recipe has several layers. First comes supervised fine-tuning on example conversations written to show the desired behavior. Then comes learning from preferences. The approach OpenAI described in the InstructGPT paper has people compare candidate answers, trains a second model to predict which answers people would prefer, and uses reinforcement learning to push the main model toward them. So you can see how easy it is for a model to get skewed this way or that based on all of the training methodologies we’ve talked about so far, to say nothing of people’s inherent biases as the process is developing.

Anthropic published a variation in 2022 called Constitutional AI , in which written principles guide an AI model’s feedback on outputs so that less of the labeling has to come from humans. Whether and how that exact method is in use today, I have no clue. The system card says post-training aims to make the model “an assistant whose behavior aligns with the values described in Claude’s constitution,” and its table of contents includes a section on reward hacking in reinforcement-learning environments, which tells you reinforcement learning is in the mix. It does not tell you the method, the data, or the amount. One lesson here is that the models want to please you, to get a result that looks like success, even if it’s dead wrong (we’ll talk about hallucinations in another part).

What I can’t tell you

Two things I’d like to give you and can’t. The first is a dollar figure for training a frontier model; labs don’t publish theirs, and the estimates floating around are guesses built on guesses. The second is a list of what Claude’s training data contained. What the public record supports is the order of magnitude: trillions of tokens, a compute budget around 10²⁵ operations for a flagship open model, and a long post-training process whose details stay mostly private. I do find it interesting, however, that Anthropic has on staff a “chief ethicist” to help Claude “learn” empathy and other desirable traits. We’ll leave that one alone for now, because it invites an uncomfortable conversation about these models, their training, and just how far they’ve come.

A record of mistakes

Oscar Wilde had Mr. Dumby say it in Lady Windermere’s Fan in 1892: “Experience is the name every one gives to their mistakes.” A trained model is a very literal realized version of that. Its 405 billion numbers are a record of trillions of small errors (much like my brain), each one corrected by a tiny amount, and then a second round of corrections applied by people and principles that decided what a good answer looks like (not at all like my brain). Beckett wanted us to fail better. The training run does that on a schedule, and the thing that comes out is not wisdom so much as accumulated, well-organized failure.

Next up is the part I have historically known best, coming from a network and systems engineering background: the chips, the cables, and the network that make fifteen trillion tries possible in a matter of weeks instead of centuries.