AI engineering
The RAG Paper: What Retrieval Added to Generation
The original RAG model combined document retrieval with generation. Its factuality gains and index-swap experiment show what that split can and cannot do.
Listen to the audio analysis
Create your ownWritten for builders and creators.
A language model's stored knowledge is difficult to inspect and expensive to update. The original retrieval-augmented generation paper explores a way to give a generator an external collection of documents it can consult.
Editor's note
Why this matters now
Lewis and colleagues' 2020 retrieval-augmented generation paper combines a language model with a searchable document index. For a builder deciding whether retrieval is doing useful work, it offers both an architecture and a measured comparison against generation without that external evidence.
In a Jeopardy question-generation evaluation, human judges preferred RAG for factuality in 42.7% of comparisons and the BART baseline in 7.1%. That is a substantial difference on a specific task. Understanding how the model used retrieved passages, and where retrieval failed, helps keep that result separate from a general promise of factual answers.
The source
What it says
Distilled from the original. The notes above and below are the editor's own.
Two Sources of Information
The model combines parametric memory, held in learned weights, with non-parametric memory, held in an external document index. Retrieval supplies relevant passages, and the generator predicts an answer conditioned on both those passages and the input.
This creates an additional way to supply knowledge without changing the generator's weights. It does not erase what those weights already contain, or guarantee that the model will use a passage correctly. The generator still has to connect the retrieved material to the question.
The paper's implementation pairs a Dense Passage Retriever with a BART generator. It studies a particular trained architecture, rather than every pipeline that searches documents and appends the results to a prompt.
Retrieval Is Part of the Model
The retriever encodes the query and documents as vectors. It finds the top-ranked passages using the inner product between the query vector and stored document vectors.
The generator then predicts output conditioned on the input and each retrieved passage. The model combines those predictions, weighting passages through the retrieval distribution. This is more specific than taking the top search result and treating it as the single source of truth.
RAG-Sequence and RAG-Token
The paper tests 2 ways to combine retrieved passages with generation:
| Variant | How Passage Predictions Are Combined |
|---|---|
| RAG-Sequence | Compute a probability for the complete output under each passage, then combine those sequence probabilities. |
| RAG-Token | Combine passage-conditioned predictions at each output token, allowing different passages to contribute at different positions. |
RAG-Sequence holds the latent document fixed for a whole candidate sequence before marginalizing over documents. It does not simply choose one verified source and discard the others. RAG-Token makes the mixture at the token level. Neither formulation guarantees that every generated claim has a faithful citation.
What the Evaluations Showed
On Natural Questions, the paper reports exact-match scores of 44.5 for RAG-Sequence and 44.1 for RAG-Token, compared with 34.5 for the listed closed-book T5-11B baseline. The comparison supports retrieval's value for this question-answering task.
It is not a pure model-size comparison. The systems differ in architecture, training and access to external information. A smaller generator with a document index also has retrieval and index-maintenance costs. Parameter count alone does not establish the total cost of serving either system.
The Jeopardy experiment addresses a different question: when given an answer and asked to generate a question, which system produces the more factual result? RAG's 42.7% preference rate versus BART's 7.1% is a pairwise human judgment. It is not a claim that only 42.7% of RAG outputs were factual, nor that the remaining outputs were all wrong.
Updating the Index Can Change the Answer
The paper demonstrates changing the external index without retraining the generator. Its world-leader experiment uses Wikipedia snapshots from different years, showing that the retrieved information can change which answer the model gives.
This is useful for knowledge that changes over time. It also creates a requirement to match the index to the question. A current officeholder and the officeholder in a specified historical year are different targets.
Replacing an index does not instantly replace the model's entire world knowledge. The new material still needs to be indexed, retrieved and used correctly. Conflicts between retrieved text and information encoded in the weights remain possible.
Retrieval Can Stop Helping
In the paper's story-generation experiment, the retriever collapsed toward returning similar documents regardless of the input, and the generator learned to ignore them. The machinery remained present while its intended contribution disappeared.
That failure explains why a retrieval pipeline needs evidence of use, not just a successful search request. A relevant-looking passage can be ignored. An irrelevant one can distract the model. An outdated one can supply an answer for the wrong period.
The paper's results therefore support checking both retrieval and the final output. The Transformer explainer describes how a generator connects positions in its input, but those connections do not establish the truth of the text they carry.
Where the Architecture Helps
Retrieval gives builders a separate place to manage source material. An application can update, remove or restrict documents without retraining the generator every time. That is an architectural benefit even when answer quality still requires evaluation.
The original RAG paper supplies evidence for factuality and question-answering gains in its tested settings. It also documents the failure modes that keep “we added retrieval” from being an adequate answer to “does the model use the right evidence?”
Editor's note
What to do with this
RAG separates part of the knowledge problem from the generator's weights. That makes external evidence available and makes some updates possible through the index. The paper shows real gains from the combination, while also showing that retrieval can become unhelpful or point to the wrong period. The contribution is a way to improve and inspect access to evidence. Correct generation still depends on which evidence arrives and how the model uses it.
The original
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al. · 22 May 2020
Read next
AI engineering
When Chain-of-Thought Prompting Helped, and When It Hurt
Wei and colleagues found gains from worked examples on difficult tasks, but losses on smaller models. What their 2022 experiments actually establish.
4 min read8 min listen
AI engineering
GPT-3: What Examples in a Prompt Can Teach a Model
The 2020 paper tested one model across many tasks without fine-tuning. Its gains and failures explain the promise and limits of in-context learning.
4 min read7 min listen