Quadrillion-parameter transformers and how to train them

TL;DR

Bigger models on repeated data need much more weight decay. Our hyperparameter law stays near-optimal at 16× the fitted compute.

With scaled hyperparameters and ensembling, loss keeps falling with compute on fixed data. Prior recipes get worse.

1.Introduction

The 1,000,000x sample efficiency gap between AI and humans is one of the central mysteries of the field, with no good explanation. One clue is model size: Chinchilla’s prescription for 30T tokens is a ~1.5T-parameter model, a tokens per parameter of 20:1 (Hoffmann et al., 2022). Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000. The tokens per parameter gap between AI and humans is thus 10,000,000x, and an important step forward for the sample efficiency gap may be to close it. On 30T tokens, we should be training quintillion-parameter neural networks. Our current networks are way too small, by seven orders of magnitude.