Wiki / concepts / wiki

LLM training pipeline

How a chat model like ChatGPT is built, per andrej-karpathy's Deep Dive. There are three stages, mapped onto a textbook: exposition (pre-training), worked examples (supervised fine-tuning), practice problems (reinforcement learning).

1. Pre-training: next-token prediction on the internet

  • The data is FineWeb-style filtering of Common Crawl down to about 44 TB, or 15 trillion tokens. Tokenization turns text into IDs from a vocabulary of 100,277 (GPT-4).
  • Training loop: take a random window of tokens (up to the context length, e.g. 8,000) and have the network output a probability for all 100,277 candidates for the next token. The true next token is the label. Then adjust the model-parameters so that token becomes slightly more likely. This runs for every position of every window, in large parallel batches.
  • Researchers watch the loss, a single number for how badly the model predicts; lower is better. In Karpathy's GPT-2 reproduction each update covers 1 million tokens and takes about 7 seconds. 32,000 updates is about 33 billion tokens. At 1% done the samples are local-grammar gibberish; after a day or two, coherent English.
  • Compute means GPUs such as the H100, rented at about $3 per GPU per hour, 8 to a node, nodes stacked into data centers. Reproducing GPT-2 cost about $40,000 in 2019 and about $600 in 24 hours on one 8ร—H100 node for Karpathy ("could bring it down to $100"), thanks to better data, hardware and software. Frontier pre-training runs take months and tens of millions of dollars, which is why they happen rarely and models have a knowledge cutoff.
  • The output is a base model, an internet-document simulator (lossy-compression-of-the-internet) that Karpathy calls "glorified autocomplete".

2. Supervised fine-tuning: becoming an assistant

The internet data gets swapped for conversations. Human labelers, following long instruction documents, write ideal assistant replies (modern sets are largely synthetic, human-seeded). Training continues on those, and the model "very rapidly" takes on the assistant persona. It's much cheaper than pre-training: about 3 hours against months. The result imitates the labelers, which is why Karpathy says you're talking to a statistical simulation of an OpenAI labeler.

3. Reinforcement learning: practice problems

For problems with checkable answers (math, code), sample thousands of attempts and train on the ones that reach the right answer. This is where reasoning emerges, as in DeepSeek-R1's "wait, let's re-evaluate" chains (models-need-tokens-to-think), and where a model can exceed its human examples (move-37). For unverifiable tasks, RLHF trains against a learned reward model, which gets gamed after a few hundred steps (rlhf-is-not-rl).

Training vs inference

Everything above is training. Using the model is inference: feed in tokens, sample the next one from the predicted distribution, append it, repeat. Sampling is random, so outputs are remixes of the training data: statistically similar, rarely verbatim. The weights stay fixed throughout.

The same pipeline from a Stanford classroom

yann-dubois's CS229 lecture fills in details the Deep Dive skips:

  • The loss is cross-entropy on the next token, which equals maximizing the text's log-likelihood. Perplexity, 2 to the average loss, reads as "how many tokens the model is hesitating between"; it fell from about 70 to under 10 between 2017 and 2023.
  • Data work goes from Common Crawl (about 250B pages, 1 PB) through HTML extraction, safety and PII filters, deduplication, heuristic filters, a quality classifier trained on pages Wikipedia links to, and domain reweighting (code and books up, entertainment down). Training ends on high-quality data at a low learning rate. Dubois puts it at about 15 of Llama's 70 core people.
  • Post-training is ordinary training started from the pre-trained weights with a higher learning rate. SFT barely improves past a few thousand examples (LIMA), because it only picks which kind of internet writer to imitate. RLHF uses around a million preference pairs.
  • RLHF moved from a reward model plus PPO (the original ChatGPT recipe, "such a mess" in practice) to DPO, which raises the probability of preferred answers directly and reaches the same optimum. After PPO the model stops being a calibrated distribution, so perplexity no longer evaluates it; Chatbot Arena and LLM judges like AlpacaEval take over.
  • How much to train is set by scaling-laws.

Source: Deep Dive ยท CS229 lecture

Linked from

Lossy compression of the internetOpen weightsScaling lawsYann Dubois