AI engineering
Attention Is All You Need: Why Drop Recurrence?
The 2017 Transformer traded sequential processing for attention between tokens. See how that speeds training and creates a long-context cost.
Listen to the audio analysis
Create your ownWritten for builders and creators.
In 2017, eight researchers at Google published a paper proposing something that sounded almost like a provocation.
Editor's note
Why this matters now
Attention Is All You Need proposed a different way to build a translation model: remove the recurrent backbone and let tokens attend directly to one another. For an engineer trying to understand the Transformer, that trade explains both its training advantage and the cost of comparing positions across a long sequence.
The 2017 paper by Vaswani and colleagues also makes its choices inspectable. It compares attention heads, tests learned against sinusoidal position encodings, and reports where a better language-modeling metric produced worse translations. Those details are more useful than treating every part of the original design as an inevitable choice.
The source
What it says
Distilled from the original. The notes above and below are the editor's own.
The Sequential Bottleneck
Recurrent networks such as LSTMs and GRUs build a sequence of hidden states. Computing the next state depends on the previous one, which restricts parallel work within a training example. A signal connecting distant tokens also has to pass through many intermediate states.
Attention had already helped sequence models connect a decoder to relevant input positions. The Transformer's proposal was to use attention as the main way positions exchange information, dispensing with recurrence and convolution.
In self-attention, positions can be processed in parallel during training, and any pair can connect within one layer. That shortens the path for long-range dependencies. It does not make autoregressive generation parallel: a decoder generating new text still needs earlier output tokens before it can produce later ones.
Queries, Keys and Values
Each input representation is projected into a query, key and value. A query identifies what a position is looking for. Keys determine how strongly other positions match it. Values supply the information combined into the result.
The model compares a query with the keys using dot products, converts the scores into weights with softmax, then takes a weighted sum of the values:
Attention(Q, K, V) = softmax(QKᵀ / √dk) V
The division by √dk controls the scale of the dot products. Under the paper's simplifying assumption of independent components with mean 0 and variance 1, an unscaled dot product has variance dk. Large scores can push softmax into regions with very small gradients. Scaling helps avoid that problem.
This gives a token a direct way to use information from a distant position. The mechanism does not require information to survive a chain of recurrent states first.
Why Use Several Attention Heads?
One weighted average can mix together relationships the model needs to keep distinct. Multi-head attention uses different learned projections so several attention operations can represent different relationships in parallel. Their outputs are concatenated and projected back into the model representation.
The base Transformer uses 8 heads, each with 64-dimensional keys and values. Splitting a fixed representation width among heads keeps the main computation comparable to a single full-width head. That is an architectural budget choice, not a promise of identical runtime on every implementation.
The paper's Table 3 shows the effect on the English-to-German development set:
| Attention Heads | BLEU |
|---|---|
| 1 | 24.9 |
| 4 | 25.5 |
| 8 | 25.8 |
| 16 | 25.8 |
| 32 | 25.4 |
The single-head model performed worse, but 8 was not a uniquely winning count. The reported 16-head variant tied it. These results support multiple heads in that experiment, rather than a universal optimum.
Three Places Attention Appears
The original Transformer is an encoder-decoder model. Its encoder builds representations of the input sentence. Its decoder generates the translation.
- Encoder self-attention connects input positions to one another.
- Masked decoder self-attention prevents a position from seeing future output tokens during training.
- Encoder-decoder attention lets the decoder query the encoder's representations of the source sentence.
Both stacks have 6 layers. Attention sublayers sit alongside position-wise feed-forward networks, with residual connections and layer normalization. For the base model, the feed-forward network expands the 512-dimensional representation to 2,048 dimensions, applies ReLU, then projects it back.
The title's emphasis on attention should not obscure these other parts. The proposal removes recurrence and convolution from the sequence model. It still needs feed-forward computation, normalization and a way to represent position.
Position Has to Enter Somewhere
Self-attention alone does not encode the order of the input tokens. The paper adds positional encodings to the token embeddings at the bottom of the encoder and decoder stacks.
Its sinusoidal encoding uses different frequencies across dimensions. The authors also tried learned position embeddings and obtained nearly identical results. They selected the sinusoidal version because it might extrapolate to sequence lengths beyond training, rather than because it clearly won the reported translation comparison.
That distinction separates a measured result from a design hypothesis. The paper establishes comparable scores in its experiment. It does not establish unrestricted length generalization.
The Cost of Connecting Every Position
For a sequence of length n and representation width d, the paper compares a self-attention layer's O(n²d) complexity with a recurrent layer's O(nd²). Self-attention has a constant-length path between positions, while the recurrent path can grow with sequence length.
The quadratic term comes from considering pairs of positions. Doubling n creates roughly 4 times as many pairs. For the sentence-length inputs studied in the paper, this was an attractive trade for parallelism. Long inputs make that part of the computation more expensive.
This is a statement about the original attention computation. It is not an API pricing formula or a complete latency model. Projection layers, feed-forward work, decoding, hardware and implementation all contribute to the system a user actually runs.
The same distinction helps when reading about retrieval-augmented generation. Selecting a small set of relevant passages changes what reaches the generator. It does not remove the need to assess whether those passages contain the evidence the task needs.
What the Translation Results Established
On WMT 2014 English-to-German, the big Transformer reached 28.4 BLEU, exceeding the best results cited in the paper, including ensembles, by more than 2 points. Table 2 reports 41.8 BLEU for English-to-French, with training cost below a quarter of the previous best single model's cost. Section 6.1 instead gives 41.0 for that result, so the paper's table and prose disagree on the exact score.
The base model trained for about 12 hours and the big model for 3.5 days, each on 8 P100 GPUs. These are results for the paper's configurations and translation tasks, not expected training times for an arbitrary Transformer.
One ablation is especially useful for evaluation. Label smoothing worsened perplexity while improving BLEU. A model could become less confident about its token predictions while producing translations that scored better. Optimizing an easy-to-measure training metric did not automatically optimize the task metric.
The attention visualizations also show heads tracking apparent syntactic relationships. Those are observations about the learned weights. They do not by themselves prove that a particular head causally performs a named linguistic function.
Read the Original Design
The paper develops the architecture in Section 3, compares computational paths in Section 4, and gives translation results and ablations in Section 6. The accompanying tensor2tensor repository records the original implementation context.
Editor's note
What to do with this
The Transformer makes a clear exchange: direct connections between positions and parallel training, paid for with pairwise attention work. The paper's experiments show why that exchange was worthwhile for translation. Its ablations also leave room for alternatives, including the position encoding and head count. Reading those choices together explains the architecture more faithfully than treating attention as either a free shortcut or a complete account of every model's cost.
The original
Vaswani et al. · 12 June 2017
Read next
AI engineering
GPT-3: What Examples in a Prompt Can Teach a Model
The 2020 paper tested one model across many tasks without fine-tuning. Its gains and failures explain the promise and limits of in-context learning.
4 min read7 min listen
AI engineering
The Bitter Lesson: What Sutton Means by Scale
Rich Sutton argues that search and learning eventually beat hand-built knowledge in AI research. The distinction matters when deciding which rules to keep.
3 min read4 min listen
AI engineering
What Is Jev? A Model Built to Make Typed Decisions
Jev returns choices and probabilities for software to act on. Its decision interface explains both the speed claims and the jobs it cannot take over.
4 min read7 min listen