3-8 Attention & Transformers — Look It Up Again Instead of Remembering

1Remaining problems

Two problems left in sequence models — the bottleneck and waiting in line

An encoder–decoder RNN packs a whole sentence into one fixed-length vector and processes positions one at a time, in order. As sentences get longer, two things become problems.

Sentence length

Explained for beginners

Bottleneck: an encoder–decoder translation model shrinks the source sentence into one fixed-length vector and then produces the translation from it. Bahdanau et al. saw this fixed-length vector as the performance bottleneck and proposed letting the model look up the relevant parts of the source by itself each time it produces a word of the translation [R1]. This is attention. Waiting in line: an RNN must know h_t before it can compute h_t+1, so positions cannot be computed at the same time. The Transformer removed recurrence entirely and computed all positions at once with attention alone, which made training much faster [R2].

2Attention

The idea of attention — look at the source again for every word you produce

We translate "초파리가 사과를 먹었다" into "The fly ate the apple". Each time an English word is produced, attention is shared among the Korean words. The thicker the line, the larger the attention.

Word being produced

Explained for beginners

The Korean order is "초파리가 (the fly) · 사과를 (the apple) · 먹었다 (ate)", while the English order is "The fly · ate · the apple". The verb sits in a different place. So when producing "ate", the model must look hard at "먹었다" at the very end. The attention values in this figure are numbers set for explanation. A real translation model finds such correspondences by itself through training, and Bahdanau et al. reported that the learned attention agrees well with human intuition [R1]. All attention values are between 0 and 1 and sum to 1 (softmax).

3Q·K·V

Q·K·V with numbers — finding books in a library

Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · VVaswani et al., 2017

Question (Q)

Symbol	In the library	In a neural network
Q Query	A note saying what I am looking for	A vector expressing "what the current position needs". Input × W_Q
K Key	The index card attached to each book	A vector expressing "this is the kind of information I am" for each position. Input × W_K
V Value	The actual content of the book	The information each position actually hands over. Input × W_V
Q·K	How well the note matches the card	The dot product of the two vectors. The more similar their directions, the larger it is
÷ √d_k	—	Prevents the dot product from growing with vector length and making softmax pile onto a single position [R2]
softmax	Dividing reading time among the books	Turns the scores into attention that sums to 1
× V	Mixing the contents in proportion to time spent reading	Adds up the values using attention as weights. This is the output

4Self-attention

Self-attention — a sentence looks itself up

In section 3 the question and the books were separate. In self-attention, every word of the sentence is a query, a key and a value at once. Each word shares its attention among all the other words.

Query word

Where do the numbers in this example come from?

The word vectors were set by hand with four slots: [animal, food, action, pointer word]. The query transform W_Q was set so that "pointer words look for food, actions look for animals". The key and value transforms are left as they are. A real Transformer learns the word vectors and W_Q·W_K·W_V through training, and they are hundreds of slots wide. The formula is exactly the same as in this example. Code 1 produces the same numbers.

5Multi-head

Multi-head attention — looking at several relationships at once

A sentence holds several layers of relationships. "Who ate?" and "What was eaten?" are different questions. Each head uses its own W_Q·W_K·W_V to look at a different relationship, and the results are concatenated.

Query word

Model width (parameter count)

6Position

Position information — attention does not know order

Attention compares all positions at once. So even if the word order is shuffled, each word's result is exactly the same. To tell it the order, a "positional encoding" that differs for each position is added to the word vectors.

Positional encoding

Sentence order

Sine–cosine positional encoding

The original paper put sin(pos / 10000^2i/d) in the even slots for position pos and cos(…) in the odd slots [R2]. The front slots oscillate quickly and the back slots slowly, so each position gets a different pattern. It is like reading the second, minute and hour hands of a clock together to fix a single time. Learning the positional encoding through training is also widely used. The small Transformer in code 2 uses a learned position embedding.

7Block

Assembling the block — attention + feedforward, plus shortcuts

One Transformer block consists of multi-head attention and a feedforward neural network (the MLP from 3-2). Each gets a shortcut that adds its input back unchanged (residual connection) and normalization, and these blocks are stacked in many layers.

Block size

8Encoder·Decoder

Encoder·decoder and GPT-style — what each position is allowed to see

The same block serves different purposes when you change "which position can see which" (the mask).

Architecture

9Strengths & limits

Why it became today's foundation, and its limits

The reasons the Transformer came into wide use can be summed up in three points. They come with costs. The biggest cost is attention computation that grows with the square of the length.

Length n

Why it became the foundation

Computed all at once: with no recurrence, all positions are computed at the same time, so in the same amount of time it can train on far more data [R2].
Distant positions connected directly: the first and last positions are linked by a single attention step. Unlike an RNN, the signal does not fade by passing through every position in between. Code 2 checks this directly on the task from 3-7 (below section 9).
Scaling up the same block: it is easy to grow the scale by stacking blocks in more layers and widening them. BERT, which uses only the encoder [R4], and GPT-3, which uses only the decoder (175 billion parameters) [R5], were built by stacking this block.

Limits and costs

Square of the length: for length n there are n² attention scores. If a sentence becomes twice as long, attention computation and memory grow about 4 times. So there is a limit on the length that can be handled at once, and research to reduce this cost continues.
Data and computation: because its built-in assumptions about order and neighbors (the locality of CNNs, the sequential nature of RNNs) are weak, it must learn these from data. With small data, RNNs and CNNs can do better.
Lengths outside the training range: on inputs much longer than those seen in training, the positional encoding is unfamiliar and performance can drop.
Attention ≠ explanation: an attention map only shows "where the model looked while computing"; it does not guarantee why the judgment is right.

Experiment of code 2 — the task where LSTM failed in 3-7, now with a Transformer

Same task as 3-7 code 2 (every step is ±1, the answer is the first step) and same training settings (Adam 0.005, 30 epochs, 2,000 training and 1,000 test samples, 2 seeds). The Transformer uses one self-attention block (width 16, 2 heads) and a learned position embedding. "Attention to first step" is the mean of the attention that every position gives to the first position.

Scope of this series

This part covers the Transformer up to its architecture. Generative models that use this architecture to produce text and images (pretraining of large language models, instruction tuning, diffusion models, etc.) are outside the scope of this series. The following parts (3-4 Speech, 3-6 Time Series & Stock Prices, 3-5 Text) focus on the process of turning data into numbers.

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · Computing attention by hand with numpyLinked to sections 3–6 · numpy, matplotlib · CPU under 1 second

Computes the library example, five-word self-attention, two heads, and order shuffling with positional encoding, using the same numbers as the page. To use your own data, change BOOK_K·BOOK_V·QUERIES or the word vectors X.

Run output (run on 2026-10-03, numpy 2.4 · matplotlib 3.10 · CPU). The calculation has no randomness, so it is the same whenever you run it.

Code 2 · A small Transformer vs. LSTM in Keras — 3-7's "remember the first signal"Linked to section 9 · tensorflow (Keras), numpy, matplotlib · CPU about 8 minutes

Solves again, with a Transformer of a single self-attention block, the task on which LSTM failed at long lengths in 3-7. It also measures how much attention the trained Transformer gives to the first step, where the answer is. The table below section 9 is built directly from this output.

Run results (run on 2026-10-03, TensorFlow 2.21 · Keras 3.15 · CPU). Numbers may change with seed, version and hardware.

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

Concept	Everyday analogy	Fruit fly / brain analogy	Where the analogy breaks down
Attention	Highlighting the important pages when studying for an exam	Partial Human selective attention (focusing on one voice in a noisy place)	Attention in the brain is a phenomenon tangled up with senses, intentions, fatigue and more. A model's attention is only a weighted average set by dot products and softmax
Q·K·V	The library's question note, index cards and book contents	None	A library brings you one book, but attention mixes all the books in proportion
Softmax attention	Dividing reading time among the books	None	Time is used up however you divide it, but even a position with attention 0.01 is still fully computed (which is why it costs n²)
Multi-head	Several editors reading the same document from different viewpoints	None	A head's "viewpoint" is not set by people. When interpreted after training, many heads show no clear role
Positional encoding	Reading the second, minute and hour hands of a clock together fixes the time	None	A clock comes back around after a day, but positional encoding is familiar only within the lengths it was trained on
Residual connection	Always keeping the original pinned next to the edited manuscript	None	The key point is that the original is "added back as is". If the edits are small, the original passes almost unchanged to the next layer
Mask (GPT-style)	Covering the answer sheet and solving the next question by looking only at the earlier ones	None	During training all positions are computed at once. Only the "visible range" is covered

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

[R1] Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR 2015. arXiv:1409.0473 — hero, sections 1·2
[R2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30. arXiv:1706.03762 — hero, sections 1·3·5·6·7·8·9 (d_model 512, 8 heads × 64, d_ff 2048, division by √d_k, sine–cosine positional encoding)
[R3] Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. EMNLP 2014, 1724–1734. doi:10.3115/v1/D14-1179 — section 1 (encoder–decoder RNN)
[R4] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019, 4171–4186. doi:10.18653/v1/N19-1423 — sections 8·9
[R5] Brown, T. B., Mann, B., Ryder, N., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems 33. arXiv:2005.14165 — sections 8·9 (GPT-3, 175 billion parameters)
[R6] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. doi:10.1162/neco.1997.9.8.1735 — comparison experiment in section 9

E-3 Figure sources

The translation attention values in section 2 are numbers set for explanation (not learned values)
The block structure in section 7 follows Figure 1 of [R2] (encoder side)

Tips for Educators

A guide for instructors, on-the-job (OJT) trainers, and tutors. Times are estimates; adjust them to your actual class.

Audience	University students and working professionals who have finished 3-7. Practitioners who want to know "what parts language models are built from"
Prerequisites	Dot product of vectors (multiply matching positions and add), softmax and MLP from 3-2, encoder–decoder from 3-7
Learning objectives	① Compute the attention formula softmax(QKᵀ/√d_k)V with numbers ② Explain why self-attention, multi-head attention and positional encoding are each needed ③ Explain both why the Transformer became the foundation and its n² cost
Estimated time	Reading 45–55 min · animation demos 25 min · Colab practice 30 min (code 1 10 min, code 2 run 8 min + interpretation 12 min)

Three lesson plans

Lecture, 50 min

Concept-focused

Intro 5 min (section 1 bottleneck) → sections 2·3 library 15 min → sections 4·5 10 min → section 6 order shuffling 10 min → section 9 n² and code 2 results 5 min → check questions 5 min.

Workplace OJT, 90 min

Focused on applying to work

Sections 1·3 15 min → section 8: link the three architectures to work cases (classification, summarization, translation) 20 min → section 9 cost calculation (with your own document lengths) 15 min → code 2 run and interpretation 25 min → prompt cards 15 min.

1:1 tutoring

Pace by level

Beginners: sections 1·2·3·9 only. Intermediate learners: have them compute the result vector of section 3 by hand and compare it with code 1, and derive the 4d² + 4d of section 7 themselves.

Teaching points by section

Section	Questions to pause and ask	Hands-on tips
1 Remaining problems	"If a 40-word sentence is packed into an 8-slot vector, how many slots does each word get?"	Switch from 5 to 40 words and watch the numbers change
2 Attention	"Why does the model look at the last word when producing ate?"	Stop at the "ate" button
3 Q·K·V	"Couldn't we just fetch the book with the largest attention?"	With "Curious about both", watch the result mix into [5.5, 5.5]
4 Self-attention	"What does 'it' refer to? How did the model know?"	Press it → sweet in that order, and also see the limits of hand-set rules
5 Multi-head	"If we increase the heads to 8, do the parameters grow 8 times?"	Check the parameter count at model width 512
6 Position	"If the result is the same even after shuffling the order, what is the problem?"	Press no positional encoding → shuffled order → add, in that order
7 Block	"Is most of a block's parameters in attention or in the feedforward network?"	Read the "feedforward share" in the base model
8 Encoder·Decoder	"Why does the GPT style have to hide future positions?"	Press Decoder only and point to the × cells of the mask
9 Strengths & limits	"If a document becomes twice as long, how many times more attention computation is needed?"	Switch 4,096 → 32,768 to compare memory, then move on to the experiment table below

Top 5 common misconceptions

Misconception	Correct explanation	Scene to show
Attention picks the single most important word	It mixes all words in proportion. Words with small attention also enter the result	Section 3 "Curious about both"
The Transformer knows order by itself	Without positional encoding, shuffling the order gives the same result	Section 6
People assign a role to each head	Roles emerge through training. The two heads on this page were set by hand for explanation	Section 5 and the explanation below it
The Transformer is better than an RNN in every way	It is strong at long-range connections and parallel computation, but it costs the square of the length and can be at a disadvantage with small data	Section 9
The attention map is the reason for the model's judgment	It only shows where the model looked while computing; it does not guarantee why the judgment is right	"sweet" in section 4

Check questions (click to expand the explanation)

Q1. With q = [2, 0.4], k = [2, 0] and d_k = 2, what is the score q·k / √d_k?

q·k = 2×2 + 0.4×0 = 4. 4 / √2 ≈ 2.828. This equals the score of book A in section 3.

Q2. When the scores are [1.98, 1.98, 2.772], what are the softmax attention and the result of mixing the values V = [10,0], [0,10], [6,6]?

Attention ≈ [0.238, 0.238, 0.525]. Result = 0.238×[10,0] + 0.238×[0,10] + 0.525×[6,6] ≈ [5.53, 5.53]. The output of code 1 is [5.525, 5.525].

Q3. If only self-attention is used without positional encoding, what happens to each word's output when the word order is shuffled? Why?

It stays exactly the same. Each word's output is determined only by "comparison with all other words", and the set of things compared is the same regardless of order. In code 1 the difference was 1e-16 (the level of rounding error).

Q4. How many parameters does a multi-head attention layer with model width d = 512 have? What if the heads change from 8 to 16?

W_Q·W_K·W_V·W_O are each 512 × 512, with 512 biases each: 4 × 262,144 + 4 × 512 = 1,050,624. Changing the number of heads only shrinks the head width; the matrix sizes stay the same, so the count is unchanged.

Q5. If a document grows from 4,096 to 8,192 in length, how many times more attention scores are there?

4 times. (2n)² = 4n². It goes from 4,096² = 16,777,216 to 8,192² = 67,108,864.

Colab assignments

Basic

In code 1, compute how the attention in the library example changes if you do not divide by √d_k. Explain in two sentences how the effect of the division grows when the vectors are scaled up 10 times.

Challenge

In code 2, remove the Transformer's position embedding (AddPosition) and train again. Report how the results and the "attention to first step" change with 3 or more seeds, and write the reason, linking it to section 6.

Adjusting difficulty and further learning

Beginners	Sections 5·7 can be skipped without breaking the flow. Make sure to calculate just the library of section 3 by hand
Advanced	Have learners research and present ways to reduce the n² cost of section 9 (looking at the input in segments, low-rank approximation, etc.). Verify the papers they find in the original
Further learning	Sangsun Ahn, “Data Science for Business” lecture notes (data science in general) · STEP [K-Digital] AI Natural Language Processing (linked to 3-5 and 3-8) · STEP [K-Digital] Fundamentals of Fintech Data Analysis (linked to 3-6)
Materials for instructors	Grading criteria for the check questions and sample answers for the assignments are on the answer key page for educators (3-8)

Two problems left in sequence models — the bottleneck and waiting in line

Explained for beginners

The idea of attention — look at the source again for every word you produce

Explained for beginners

Q·K·V with numbers — finding books in a library

Self-attention — a sentence looks itself up

Where do the numbers in this example come from?

Multi-head attention — looking at several relationships at once

Position information — attention does not know order

Sine–cosine positional encoding

Assembling the block — attention + feedforward, plus shortcuts

Encoder·decoder and GPT-style — what each position is allowed to see

Why it became today's foundation, and its limits

Experiment of code 2 — the task where LSTM failed in 3-7, now with a Transformer

Scope of this series

Colab Code — Run This Page Yourself

AI Prompts — Using This Code as a Starting Point

Glossary

Analogies — and Where They Break Down

About the Author and References

E-1 About the author

Courses

Papers and presentations

E-2 Academic references

E-3 Figure sources

Tips for Educators

Three lesson plans

Concept-focused

Focused on applying to work

Pace by level

Teaching points by section

Top 5 common misconceptions

Check questions (click to expand the explanation)

Colab assignments

Basic

Challenge

Adjusting difficulty and further learning

Well done!