Two Strong Oxen or 1,024 Chickens: The Hardware and Network Behind LLM Training

“If you were plowing a field, which would you rather use? Two strong oxen or 1024 chickens?” — attributed to Seymour Cray

I couldn’t find a primary source for that line; the collection I checked lists it as attributed , noting that it shows up in several books without citation. It’s a fitting quote I didn’t want to leave out, so I’m leaving it in with the caveat attached. Anyone who’s read anything I’ve written for years now knows that I’m fond of wrapping articles with quotes, so there’s that. Anyhow, Cray built single, enormous, fast machines, and the quote implies the self-evident answer is oxen. The AI industry has chosen the chickens, however, in quantities Cray would have found alarming. That raises an interesting engineering question the quote skips: how do you get 16,000 chickens to pull in the same direction at the same time?

One disclosure before we go forward. Several of the companies in this post sell networking gear, and several are founding members of an industry effort I discuss near the end. I should point out that I was involved with Cisco as a Cisco Champion for several years and have used their products extensively throughout my career; I’ve also participated in many, many Tech Field Day events since early on in that organization’s origins. I believe I’m as neutral as anyone these days, but I wanted to be upfront in case someone starts picking nits. Moreover, I’ll treat vendor claims as claims, and I’ll tell you if I’m leaning on a source that has something to sell.

This is part three of six. If you’ve read the first two, you know what the model is (a stack of matrices) and how it learns (educated guessing, grading, nudging, and much repeating). Now we get to the part where I’m much more comfortable: a frontier training run is a networking problem dressed up in a math costume. As before, I’m writing for a reader who follows Ars Technica, which leaves some people out; where I remember, I’ll endeavor to translate network jargon into plain terms instead of assuming anyone knows what the hell I’m yammering on about.

Why one chip doesn’t cut it

Let’s start with memory. Nvidia lists the H100 SXM, a workhorse training chip over the last few years, at 80 GB of memory, 3.35 TB/s of memory bandwidth, and 900 GB/s of NVLink bandwidth per GPU . Those are the vendor’s figures, so they describe the best case. Meanwhile, a 405-billion-parameter model stored at 16 bits per number needs 810 GB just for the weights (2 bytes times 405 billion), more than ten of those chips before you’ve stored a single intermediate result. Training also has to keep the gradients and the optimizer’s bookkeeping around, which pushes the total several times higher.

Now on to compute. Apologies in advance for tossing more math over the fence. In part two we got to about 3.8 × 10²⁵ operations for Llama 3’s flagship run. Let’s assume one GPU sustains 400 trillion operations per second, a round number that happens to be what Meta actually measured per GPU at that scale. One chip would need roughly 3,000 years. Spread the same work over the 16,000-odd H100s Meta says it used and you land near two months, assuming nothing ever breaks. The scaling isn’t free, as we’ll see. It only works if the chickens (you didn’t think we were done with chickens, did you) can talk to one another.

Whose chips are these?

While the chips most people picture are Nvidia’s GPUs, and they do dominate, they aren’t alone. Google builds its own tensor processing units (TPUs), and Amazon builds Trainium. Anthropic (known for its Claude LLM, and a company that I spend a decent amount of money with) has been public about spreading its bets: Data Center Frontier’s reporting describes access to up to one million Google TPUs under an expansion announced October 23, 2025, an AWS cluster called Project Rainier with roughly 500,000 Trainium2 chips, and continued “significant use” of Nvidia GPUs. Anthropic has since said as much itself: its April 6 post says Claude is trained and served on Trainium, TPUs, and Nvidia GPUs.

What I don’t know, and neither source says, is which model ran on which chip. Treat the mix as a procurement fact, not an architecture diagram.

Slicing up the model

Even with enough chips (can you ever have enough), you have to decide how to divvy up the work. The Llama 3 team used four kinds of parallelism at once , and each one asks the network to do something different. Tensor parallelism splits individual matrices across GPUs, an approach popularized by Megatron-LM , so every layer’s multiplication is a team effort. Pipeline parallelism assigns different layers to different GPUs, like an assembly line. Data parallelism makes copies of the model, each processing different text, then averages what they learned. And context parallelism splits a long input across GPUs.

The paper lists these in the order tensor, context, pipeline, data. According to the paper, that ordering is optimized for network communication, with the chattiest groups kept inside a single server where the links are fastest. Either way, the point for a network engineer is that these aren’t one traffic pattern. They’re several, layered on top of each other, with different message sizes and different tolerance for delay.

What the network looks like, at least for today

The details Meta publishes are refreshingly concrete. Each server holds eight GPUs connected by NVLink. Each rack holds 16 GPUs in two servers behind a single top-of-rack switch. Between servers, the links run at 400 Gbps per GPU. For the 405B model specifically, Meta trained over a RoCE network (RDMA over Converged Ethernet, which lets one machine write straight into another’s memory without hassling the CPU, which is completely similar to the way my brain works, except in every possible way), while the smaller models ran on Nvidia’s Quantum2 InfiniBand. In other words, the largest model in that family was trained on Ethernet.

Why does any of this matter? Because of the way training steps work. At the end of each step, the GPUs have to combine their results before any one of them can start the next one, so the whole cluster has to wait for the slowest participant. One congested link, one flapping optic, or one dropped packet that triggers a retransmit stalls thousands of expensive chips. As a longtime network engineer in a past life, I know that lossless Ethernet, priority flow control, and tail latency are all difficult in their own right. Getting this stuff working mostly flawlessly takes real engineering talent—which is not cheap. I’d rather you hear that from someone who has debugged it than from a summary of a paper.

Things break

Here’s the number I’d put on a slide. During a 54-day stretch of Llama 3’s pre-training, Meta recorded 419 unexpected interruptions . Do the division, and that is roughly one every three hours. A cluster that size isn’t a machine that occasionally fails; it’s a machine that is always in the middle of failing, and the software has to treat that as the normal case. The usual answer is checkpointing: saving the model’s state at regular intervals so a failure costs minutes instead of days, plus tooling to spot the bad node and swap it out. I’m confident in the first part and less in the specifics of how any given lab does the second.

The paper’s breakdown of failure categories contains a figure that doesn’t agree with the rest of its own table (148 faulty-GPU events listed as 30.1% of 419 doesn’t compute, even with creative math, though the other rows do), so I’m leaving the percentages out and sticking to the total. It’s a reminder that even the best-documented public run has rough edges. This is all emerging and rapidly evolving technology, and as with anything being developed fast and as it’s being used, mistakes happen, in both actual practice and in reporting.

The scale-up fight

Everything above is “scale-out,” the network between servers. The newest argument is about “scale-up”: the ultra-fast fabric inside a rack, where Nvidia’s proprietary NVLink is currently dominant. An industry group has been trying to change that. The Open Compute Project’s ESUN initiative (Ethernet for Scale-Up Networking) launched October 13, 2025, with 12 founding members: AMD, Arista, Arm, Broadcom, Cisco, HPE Networking, Marvell, Meta, Microsoft, NVIDIA, OpenAI, and Oracle, most of which, for full transparency, I’ve done business with at some point in my career. It has since grown to more than 175 companies. Its first base specification, published in March 2026, calls for lossless Ethernet using priority and credit-based flow control, link-level retry, and a compact 4-byte header in place of the usual IP and UDP stack, plus congestion control for multi-hop topologies.

On October 7, Arista announced rack-scale Ethernet architectures for both scale-up and scale-out, including a scale-up design it calls Etherlink SU-144, and positioned them against Nvidia’s NVLink and InfiniBand. Those are Arista’s claims. Nvidia is itself an ESUN founding member, so this isn’t a clean story of open Ethernet against a proprietary villain. It’s a market in which everyone is hedging. Plus, I’ve never bought the argument that just because something is proprietary, it’s automatically a bad thing.

One caveat here: ESUN’s first release is a requirements specification, not a shipping fabric, and I haven’t seen independent measurements comparing these Ethernet designs with NVLink under real training loads. Until I do, I’d file the vendor performance claims under promising, but to be taken with the proverbial grain of salt.

Station wagons and chickens

The line I keep coming back to is one that network people know by heart: “Never under-estimate the bandwidth of a mini-van filled with DLT tapes, on the freeway at 65 mph.” It’s popularly credited to Andrew Tanenbaum’s Computer Networks, though people on the NANOG list say they heard versions of it decades earlier (one recalls 1969), so I’d treat the attribution as folklore. The salient point works either way: a system’s real capacity depends on what connects its pieces.

So, oxen or chickens? The industry chose chickens, and the work didn’t disappear; it moved to the farmer. The model is the part we talk about, and the network decides whether a training run takes two months or never finishes. And, yes, we did beat that farming metaphor within an inch of its life, but where else can I write about farm animals and computer networks in the same breath, if not my own blog?

Next up: what happens after training ends and a million people want answers to insipid questions at once. Inference, as it happens, puts wildly different demands on the same hardware.