AI engineering

Chinchilla: What a Smaller, Better-Trained Model Showed

DeepMind matched Gopher’s training budget with a smaller model and more tokens. The result explains why parameter count alone is a poor guide.

Listen to the audio analysis

Create your own

Written for builders and creators.

Most of the massive AI models we've been hearing about are actually starving for data. The industry's been wasting billions of dollars building these massive "skyscrapers" with no way to finish the upper floors.

Editor's note

Why this matters now

DeepMind's 2022 Chinchilla paper asks how to divide a fixed training budget between model size and training data. Its answer challenged a familiar comparison: a model with more parameters need not be the better use of the available compute.

Chinchilla used 70 billion parameters, against Gopher's 280 billion, and trained on 1.4 trillion tokens. It outperformed Gopher across the paper's evaluations at approximately the same training compute. For an engineer reading model specifications, the useful question is therefore how the model was trained, alongside how large it is.

The source

What it says

Distilled from the original. The notes above and below are the editor's own.

A Fixed Budget Forces a Choice

A training run can spend compute on a larger model, more training tokens, or some combination of the two. “Compute-optimal” in this paper means selecting the combination expected to achieve the lowest language-modeling loss for a given training-compute budget.

It does not mean maximizing intelligence per dollar in every deployment. It also does not mean feeding a model tokens until it is fully “saturated.” The objective is narrower: find the best allocation under the paper's training setup and budget.

Earlier scaling estimates favored increasing model size faster than the amount of training data. The Chinchilla authors found that several prominent models were too large for the number of tokens they had seen under this compute-optimal criterion. Additional parameters were not the most effective place to spend the next unit of training compute.

The Gopher Comparison

The authors trained Chinchilla to test their revised allocation. Gopher and Chinchilla came from the same research team, making this a concrete comparison rather than an abstract claim that smaller models always win.

ModelParametersTraining Tokens
Gopher280 billion300 billion
Chinchilla70 billion1.4 trillion

Chinchilla had one-quarter of Gopher's parameters and saw about 4.7 times as many training tokens. The paper describes the data increase in round terms as about 4 times. Their total training-compute budgets were approximately equal.

On MMLU, a benchmark spanning academic subjects, Chinchilla reached 67.5% average accuracy, more than 7 percentage points above Gopher. That result is stronger than a claim of similar quality in a smaller file: the revised training allocation produced better measured performance in this comparison.

It remains a comparison of these models and evaluations. It does not establish that any 70-billion-parameter model will beat any 175- or 280-billion-parameter model.

How the Scaling Rule Was Estimated

The researchers used 3 approaches:

  • Train fixed model sizes for different amounts of time and compare the resulting loss curves.
  • Hold compute constant while varying model size, producing IsoFLOP profiles that expose the best allocation at each budget.
  • Fit a parametric relationship among model size, training tokens and loss, then estimate the optimum.

All 3 approaches pointed toward scaling parameters and training tokens in roughly equal proportions as compute grows. The estimates were approximate, rather than an exact law that every training run must obey.

A useful consequence follows from the training cost relationship. Doubling both parameters and tokens requires roughly 4 times the compute. “Scale them equally” does not mean that doubling the compute doubles both inputs.

The authors also flag uncertainty when extrapolating beyond the budgets studied. A fitted frontier is evidence about a measured range, not a guarantee at arbitrarily large scale.

Why Training Optimum and Serving Cost Differ

A smaller model generally reduces the parameter memory and computation needed for each inference compared with a larger model under similar conditions. Chinchilla therefore offers a reason to care about training allocation even if the reader only serves models.

But the paper's optimization target is the training run. A product that serves a very large number of requests has a different total-cost question: training expense plus repeated inference expense. The Chinchilla ratio alone does not solve that problem.

Nor does a 70-billion-parameter count establish that the model fits on consumer hardware. Memory requirements depend on numerical precision, runtime overhead, context and concurrency. Claims about a particular server or latency need a deployment measurement.

Better Benchmark Results Do Not Settle Safety

The paper reports improved TruthfulQA performance alongside mixed results in toxicity and bias evaluations. These findings need to stay attached to their tests. A gain on questions designed to elicit common misconceptions does not establish that the model is generally truthful.

Likewise, improving language-modeling quality did not automatically remove undesirable generated text. The training-allocation result is useful without turning it into a guarantee about safety or a claim that more data always removes bias.

What the Comparison Changes

The GPT-3 paper showed how performance and in-context learning could improve with scale. Chinchilla asks how to spend that scale efficiently. The two questions are related, but a parameter count alone answers neither.

For model evaluation, the consequence is to compare observed task performance and operating requirements, with training information as context. For training research, it is to estimate the balance between model size and data under the actual budget. The original paper supplies the empirical methods and the Gopher comparison behind that argument.

Editor's note

What to do with this

Chinchilla's result is a correction to an allocation rule: at the studied budget, fewer parameters trained on more tokens produced a better model. The useful habit is to inspect both sides of that trade. Maximizing parameter count ignores the training data, while maximizing tokens per parameter ignores the fixed budget and model capacity. The paper earns its influence by measuring a better balance, rather than by declaring one input universally more valuable than the other.

The original

Training Compute-Optimal Large Language Models

Hoffmann et al. · 29 March 2022

  • AI engineering

    GPT-3: What Examples in a Prompt Can Teach a Model

    The 2020 paper tested one model across many tasks without fine-tuning. Its gains and failures explain the promise and limits of in-context learning.

    4 min read7 min listen

  • AI engineering

    The Bitter Lesson: What Sutton Means by Scale

    Rich Sutton argues that search and learning eventually beat hand-built knowledge in AI research. The distinction matters when deciding which rules to keep.

    3 min read4 min listen

All posts