AI engineering
GPT-3: What Examples in a Prompt Can Teach a Model
The 2020 paper tested one model across many tasks without fine-tuning. Its gains and failures explain the promise and limits of in-context learning.
Listen to the audio analysis
Create your ownWritten for builders and creators.
Imagine being able to teach a machine a brand-new skill just by showing it three examples, without ever having to rewrite a single line of its code.
Editor's note
Why this matters now
The 2020 GPT-3 paper tested a practical idea at scale: could one language model perform many tasks when given instructions and examples, without a separate training run for each task? Brown and colleagues compared zero-shot, one-shot and few-shot prompts across language and reasoning benchmarks.
For an engineer deciding what belongs in a prompt, the distinction is useful. Examples change the context the model uses to answer. They do not update its weights or permanently teach it a new skill. GPT-3 made that approach more convincing across a broad set of tasks, while documenting substantial failures alongside the gains.
The source
What it says
Distilled from the original. The notes above and below are the editor's own.
Adapt the Context, Keep the Weights Fixed
Fine-tuning changes model weights using task-specific training data. In-context learning supplies demonstrations in the input and leaves the weights unchanged. The model continues from the examples when it reaches the new question.
The paper compares 3 settings:
| Setting | What the Prompt Provides |
|---|---|
| Zero-shot | A task description, with no worked example |
| One-shot | A task description and 1 example |
| Few-shot | A task description and several examples |
The number of examples varies with the task and available context. “Few-shot” is an evaluation setup, not a promise that a fixed number of examples will solve every task.
Prompt-based task adaptation preceded GPT-3. The paper's contribution was a systematic study across model sizes and tasks, culminating in a 175-billion-parameter model. It did not invent all prompting or eliminate the value of fine-tuning.
What Scale Added
The researchers found that larger models generally performed better and often gained more from examples in context. That made one pretrained model useful across a broader range of tasks without task-specific weight updates.
The result changes the cost of trying an idea. A builder can express a task through examples before deciding whether a separate training process is warranted. It does not prove that scale replaces specialization on every problem, or that a prompt is necessarily cheaper once deployment quality requirements are included.
The Chinchilla paper later asks a complementary question: how should a training budget be divided between model size and training tokens? GPT-3's scale results and Chinchilla's allocation results concern different choices.
A Concrete Result: Closed-Book Question Answering
TriviaQA gives a clear example of the prompting comparison. GPT-3 answered from its parameters and the prompt, without retrieving documents at inference time.
| GPT-3 Setting | TriviaQA Accuracy |
|---|---|
| Zero-shot | 64.3% |
| One-shot | 68.0% |
| Few-shot | 71.2% |
The few-shot score was 6.9 percentage points above the zero-shot score. The weights were unchanged between those settings. This supports the claim that context can improve task performance, rather than showing that the model permanently learned the supplied facts.
The paper also studies translation, arithmetic, word manipulation and other tasks. Results vary. Natural-language inference and some reading-comprehension tasks remain weak points, so a strong question-answering score should not stand in for general reliability.
Fluent Prose and Factual Prose Are Different Results
The paper asked human participants to distinguish short generated news articles from human-written ones. For the largest model, reported detection accuracy was around 52%, close to chance in that experiment.
That measures how difficult the samples were to identify as machine-written. It does not show that the articles were factually correct, or that the model passed a general Turing test. The paper separately describes factual errors, incoherence and repetition.
For a product using generated prose, those are different evaluation questions. A convincing style can make unsupported content harder to notice. Assessing whether text sounds human does not establish whether its claims deserve to be published.
What the Benchmarks Could Not Settle
The training corpus could contain material overlapping the evaluation benchmarks. The authors attempted to remove such overlap, but a filtering bug left contamination concerns. That limits how confidently a score can be interpreted as performance on unseen material.
Overlap does not prove that every successful answer was memorized. It does mean that a benchmark result alone cannot establish where a capability came from. The paper's contamination analysis is part of the result, rather than a footnote to discard after quoting the score.
The bias analysis also found uneven associations across gender, race and religion. Those observations establish problems in the tested model and prompts. They do not supply a simple remedy such as an instruction to “be neutral,” and the paper does not validate that instruction as a safeguard.
What the Examples Actually Contribute
An example can show the task, the expected answer format and the kind of relationship the model should continue. It may also supply information needed for a particular answer. Those effects can overlap, so calling every improvement “learning a new skill” says more than the experiment can establish.
The later chain-of-thought study changes what the examples contain: it adds intermediate reasoning before the answer. That work provides a more specific test of whether a worked solution helps on multi-step tasks.
GPT-3's original paper remains useful because it keeps the broad promise beside task-level results and limitations. A single model can support many prompted tasks, but each task still needs its own evidence of quality.
Editor's note
What to do with this
Examples in a prompt are a way to adapt a model's behavior for the current context. GPT-3 showed that this could produce strong results across several tasks without updating the model, and that larger models often benefited more. The same study also recorded failures, bias and evaluation uncertainty. Its lasting contribution is a practical interface worth testing, not a guarantee that enough examples can replace training or make a fluent answer correct.
The original
Language Models are Few-Shot Learners
Brown et al. · 28 May 2020
Read next
AI engineering
When Chain-of-Thought Prompting Helped, and When It Hurt
Wei and colleagues found gains from worked examples on difficult tasks, but losses on smaller models. What their 2022 experiments actually establish.
4 min read8 min listen
AI engineering
Chinchilla: What a Smaller, Better-Trained Model Showed
DeepMind matched Gopher’s training budget with a smaller model and more tokens. The result explains why parameter count alone is a poor guide.
4 min read6 min listen