Scaling laws

Plot test loss against training compute, dataset size or parameter count on log-log axes and you get close to a straight line (OpenAI's 2020 scaling-laws paper). Bigger models trained on more data keep getting better, and by a predictable amount. yann-dubois: "overfitting doesn't happen with large language models." As of mid-2024 the lines showed no plateau, and nothing in theory says when one would appear.

How labs use them:

• To choose a recipe before the expensive run. Tune hyperparameters on small models of several sizes, fit a curve, extrapolate, then spend most of the budget (27 of 30 days in his example) on one large run. The old way was to train many large models briefly and keep the best.
• To compare architectures. Transformers beat LSTMs on both the slope and the intercept of the curve. Small architecture tweaks mostly move the intercept, which a longer run or the next GPU generation makes up for; better data improves the curve itself.
• To split compute between size and data. Chinchilla's iso-FLOP curves put the training optimum at about 20 tokens per parameter. Counting inference cost, production models use about 150, because a smaller model is cheaper to serve for its whole life.

The compute estimate behind all of this is FLOPs ≈ 6 × parameters × tokens. For Llama 3 405B on 15.6T tokens that's 3.8e25 FLOPs, about 40 tokens per parameter and roughly half the 1e26 threshold in the 2023 US executive order. Dubois estimates about 70 days on 16,000 H100s and around $75M including salaries. Each new generation aims for about 10x the FLOPs.

Related: model-parameters, llm-training-pipeline, and the bitter lesson (Sutton): with compute always growing, the architectures that use it well win.

Source: CS229 lecture