Deep Learning A to Z · 3-7

Sequence Models — Walking One Step at a Time, Carrying Memory

In sentences, sounds, sensor logs and stock prices, order creates meaning. Give a network that looks at one frame at a time a loop that remembers "what it just saw," and you get an RNN. Add doors (gates) so that memory does not fade, and you get LSTM and GRU. We follow with numbers whether memory really survives, and confirm by experiment that even with gates, learning does not happen by itself.

Written by Sangsun Ahn (CEO, M-Robo Inc.) · About the author

1990Elman’s simple recurrent network [R1]
1997LSTM published [R3]
3 · 2Number of gates in LSTM and GRU
1,152 : 912Recurrent-layer parameters of LSTM and GRU with 16 hidden units (Keras)
+0.86 → −0.88LSTM memory that flips when the first signal is flipped (Section 5)
2Code examples you can run right away in Colab
1One frame

One frame at a time, in one direction — what a single scene cannot tell you

In 3-1, the fruit fly recognized whether a shadow was "approaching." But what if it received only a single photo of the shadow?

What the network receives

Explained for beginners

The neural networks of 3-2 and 3-3 take one input, produce one output, and are done. They do not remember what they saw before. Such a network is called a feedforward (flowing only forward) neural network. A single shadow of a given size may be approaching or receding. The answer lies not in one frame but in the order of the frames. That is also why the experiment in 3-1 fed in two frames (earlier and later) together.

2Feedforward

Two ways to feed order into a feedforward network, and their limits

The sequences "100 → 110 → 120" (rising) and "120 → 110 → 100" (falling) are made of the same three numbers. There are two ways to feed them into a feedforward neural network.

Input method
Sequence length
3Loops & unrolling

Why a loop appears, and unrolling it

ht = f(xt, ht−1)Reuse the same weights at every time step

View
Length

Explained for beginners

The idea that solves the problem of Section 2 is simple. When processing one step, the network also receives the summary made at the previous step (the hidden state). Drawn as a picture, an arrow leaving the box returns to the box itself and forms a loop. Unroll this loop along the time axis, and it looks like a feedforward neural network with the same box repeated once per step. All the boxes use the same weights. Just as the CNN in 3-3 used the same filter at every "position," an RNN uses the same weights at every "time step." So the number of weights stays the same even when the length changes.

4RNN

RNN — hidden state, BPTT, and vanishing gradients

ht = tanh(wx·xt + wh·ht−1)Elman, 1990

Loop weight wh
Input

BPTT and vanishing gradients

Applying backpropagation on the unrolled view gives BPTT (backpropagation through time) [R8]. For the error at the last output to reach the first step, it is multiplied by wh × the tanh gradient (1 − h²) at every step it goes back. If this value is less than 1, it is raised to the power of the number of steps and approaches 0 (vanishing gradient); if it is greater than 1, it can explode. In 1994, Bengio et al. showed that this makes it fundamentally hard to learn long-term dependencies with gradient descent [R2].

Making wh large keeps memory longer (1.5), but tanh sticks near ±1 and the gradient 1 − h² approaches 0. Holding on to memory and correcting that memory through learning collide with each other. It is the same root problem as "the sigmoid gradient is at most 0.25" in 3-2.

5LSTM

LSTM — the cell state and three doors

ct = ft ⊙ ct−1 + it ⊙ gt · ht = ot ⊙ tanh(ct)Hochreiter & Schmidhuber, 1997 · forget gate Gers et al., 2000

Forget gate bias bf
First signal
SymbolNameRangeWhat it does (weights set for this cell)
ctCell stateUnboundedA conveyor belt carrying long-term memory. It passes to the next step through just one multiplication (f) and one addition
ftForget gate0~1 (σ)How much of the past memory to keep. Held constant at σ(bf)
itInput gate0~1 (σ)How much of the new value to write in. σ(10 × start flag − 5): opens only at the first step
gtCandidate value−1~1 (tanh)What to write in. tanh(3 × xt)
otOutput gate0~1 (σ)How much of the memory to let out. Kept always open at σ(5) ≈ 0.993
htHidden state (output)−1~1ot ⊙ tanh(ct). Goes to the next layer and to the gates at the next step
⊙Element-wise productMultiplies matching positions of vectors. With a single number, as in this example, it is just multiplication

Explained for beginners

The weights of this cell were set by hand for explanation. The rule is "write only at the first step, and hardly erase after that." So when the first signal is flipped, the memory 20 steps later flips exactly too. For the RNN in Section 4, flipping the first signal of the same input left the last value identical to four decimal places. This means LSTM is a structure that can hold long-term memory. Because the cell state is carried on only by one multiplication (f ≈ 1) and an addition, the gradient passed backward barely shrinks either [R3].

Two additions. First, the original 1997 LSTM had no forget gate. Gers et al. added the forget gate to fix the problem of the cell state only growing on endlessly continuing input [R4], and today "LSTM" usually means this version. Second, the fact that the structure can hold memory does not guarantee that learning will find those weights. The experiment in Code 2 shows that difference (below Section 6).

6GRU

GRU — an LSTM slimmed down to two doors

ht = (1 − zt) ⊙ ht−1 + zt ⊙ h̃tCho et al., 2014

Hidden size
Input size

What did GRU merge?

GRU merges LSTM’s forget gate and input gate into a single update gate z. It keeps (1 − z) of the past memory and fills the freed space with the new candidate h̃ by z. It is a balance scale: the more it keeps, the less it writes, and the more it writes, the less it keeps. On top of this it adds a reset gate r, which decides how much of the past memory to use when computing the candidate. Since it does not keep a separate cell state and hidden state, it has no output gate either [R5].

The weight groups shrink from LSTM’s 4 to 3, so by the textbook formula the parameters are exactly 3/4. Keras’s default GRU keeps one extra set of biases, so it has slightly more (912 vs. 1,152 with 16 hidden units and input size 1, about 0.79×). Which one is better depends on the task.

The Code 2 experiment — a structure being able to hold memory is different from learning actually finding it

A sequence in which every step is ±1 and the answer is the first step (all later steps are identical-looking distractors). 16 hidden units, Adam 0.005, 30 epochs, 2,000 training and 1,000 test samples. Results for 2 seeds are listed separately. Around 50% is guessing level, i.e. a learning failure.

7Autoencoder

Autoencoder — squeeze through an hourglass, then restore

It learns to squeeze the input into a narrow middle layer (the bottleneck) and then restore it to the original. No answer labels are needed: the input itself is the answer. The fruit fly mushroom body does the exact opposite and expands.

Bottleneck size
View

Explained for beginners

A narrow bottleneck forces it to keep only what matters most. The sequences in this figure are 16-step pieces made by mixing two kinds of sine waves. It has been proven that the best result a (linear) autoencoder without activation functions can reach by learning is the same as principal component analysis (PCA) [R6], so this figure computes and shows that optimal result directly. Stacked, nonlinear autoencoders can compress better than PCA [R7].

In sequence models, this was used as a structure (encoder–decoder) in which an RNN encoder compresses a sentence into a fixed-length vector and an RNN decoder unfolds it into another sentence [R5]. The limitation that a long sentence is hard to fit entirely through one narrow bottleneck gives rise to attention in the next part, 3-8.

8Notation & history

Diagram notation and history — how to read the figures, and who did what when

Figure symbols differ a little from book to book and paper to paper. Here we summarize this series’ conventions and look, in chronological order, at what problems sequence models solved as they developed.

SymbolMeaningFeedforwardRNNLSTMGRUAutoencoder
BoxOne layer (weight multiply + activation function)✓✓✓✓✓
→Direction values flowForward onlyForward + next time stepForward + next time stepForward + next time stepForward only
↺ LoopReceives the value from the previous time step again—hh and ch— (sequence versions use RNNs)
σDoors (gates) that squash to 0–1——3 (f, i, o)2 (z, r)—
tanhActivation function that squashes to −1–1Optional✓Candidate g and outputCandidate h̃Optional
⊙ · ⊕Element-wise product · addition——✓✓—
HourglassLayer sizes that narrow, then widen————✓
9Model picker

Model picker — what should I try first on my data?

Answer three questions and it picks a model to start from. It is a first candidate, not the answer. The final decision is always made by comparing on validation data (3-2, Section 9).

① Data
② Sequence length
③ Processing mode

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · Computing an RNN and an LSTM step by step in NumPyLinked to Sections 4 and 5 · numpy, matplotlib · under 1 second on CPU

Computes the one-number RNN of Section 4 and the hand-set LSTM of Section 5 on the same sequence. The hidden states, the gradients passed backward, and the change when the first signal is flipped come out as the same numbers as on the page. To use your own data, change only SEQ.

Run output (run on 2026-10-03, numpy 2.4 · matplotlib 3.10 · CPU). The calculation has no randomness, so it is the same whenever you run it.
Code 2 · Comparing SimpleRNN · LSTM · GRU in Keras — "Remember the first signal"Linked to Section 6 · tensorflow (Keras), numpy, matplotlib · about 5 minutes on CPU

Trains three models on sequences in which every step is ±1 and the answer is the first step. Lengths 10 · 50 · 100, 2 seeds. The table below Section 6 is built directly from this output.

Run output (run on 2026-10-03, TensorFlow 2.21 · Keras 3.15 · CPU). Success and failure can change with the seed, versions and hardware. This variability itself is one of the things this experiment shows.

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔 〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

ConceptEveryday analogyFruit fly / brain analogyWhere the analogy breaks down
Hidden stateThe plot summary you build in your head as you read a bookPartial Sustained activity of groups of neuronsWhich circuits maintain short-term memory in the brain differs by region and is still being studied
Unrolling · weight sharingThe same employee working by the same work rules every dayNoneThe brain does not make identical "copies" for each moment. Unrolling is a picture for computation
Vanishing gradientThe first words getting blurred in a game of telephoneNone Backpropagation itself has not been confirmed in the brainIn telephone the forward-moving signal blurs, but a vanishing gradient is the learning signal going backward getting smaller
Cell stateA factory conveyor beltNoneItems on a belt travel unchanged, but the cell state shrinks a little at each step by the forget gate
GateA faucet (how far to open it, 0–1)Partial Neuromodulators that regulate synaptic transmission (e.g. dopamine in the fruit fly mushroom body)Dopamine is involved in learning (changes in connection strength). LSTM gates are computations that set the flow at each step, and are not learning in themselves
GRU’s update gateA balance scale: the less old memory you keep, the more new content you holdNoneUnlike a real scale, it uses a different ratio at every step and for every number
Autoencoder bottleneckSummarizing a long meeting in three lines, then writing it out againPartial Counterexample: the mushroom body expands instead of narrowingUnlike a human summary, an autoencoder does not let a person decide "what matters"; it decides only by the reconstruction error

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

  • Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
  • Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
  • Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
  • Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
  • Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
  • Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
  • Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
  • Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

  1. [R1] Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211. — Hero, Sections 4 and 8
  2. [R2] Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. doi:10.1109/72.279181 — Sections 4 and 8
  3. [R3] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. doi:10.1162/neco.1997.9.8.1735 — Hero, Sections 5 and 8
  4. [R4] Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10), 2451–2471. doi:10.1162/089976600300015015 — Sections 5 and 8 (forget gate)
  5. [R5] Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. EMNLP 2014, 1724–1734. doi:10.3115/v1/D14-1179 — Sections 6, 7 and 8 (GRU, encoder–decoder)
  6. [R6] Baldi, P., & Hornik, K. (1989). Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1), 53–58. Author’s copy — Sections 7 and 8
  7. [R7] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. doi:10.1126/science.1127647 — Sections 7 and 8
  8. [R8] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. deeplearningbook.org — Sections 3 and 4 (Chapter 10, recurrent neural networks, BPTT)
  9. [R9] Caron, S. J. C., Ruta, V., Abbott, L. F., & Axel, R. (2013). Random convergence of olfactory inputs in the Drosophila mushroom body. Nature, 497(7447), 113–117. doi:10.1038/nature12063 — Section 7
  10. [R10] Dasgupta, S., Stevens, C. F., & Navlakha, S. (2017). A neural algorithm for a fundamental computing problem. Science, 358(6364), 793–796. bioRxiv preprint — Section 7

E-3 Figure sources

  • Approaching/receding scenes in Section 1 — same setup as the scenes in Sections 3 and 4 of 3-1 “How AI Works”
  • Mushroom body figure in Section 7 — same rough figures as 3-1 Section 5 (about 50 glomerulus types, about 2,000 Kenyon cells)

Tips for Educators

A guide for instructors, on-the-job (OJT) trainers, and tutors. Times are estimates; adjust them to your actual class.

AudienceUniversity students and working professionals who have finished 3-2. Practitioners handling sensor, log, text and time-series data
PrerequisitesBackpropagation, vanishing gradients, sigmoid and tanh from 3-2. Knowing weight sharing from 3-3 makes Section 3 easy
Learning objectives① Explain, with two methods (averaging and concatenation), why feedforward networks struggle with order ② Show the RNN hidden state and vanishing gradients in numbers ③ Explain what the gates of LSTM and GRU do, and distinguish "the structure can hold memory" from "learning finds it"
Estimated timeReading 45–55 min · animation demo 25 min · Colab practice 30 min (Code 1: 10 min, Code 2: 5 min run + 15 min interpretation)

Three lesson plans

Lecture, 50 min

Concept-focused

Intro 5 min (the same scene of Section 1) → Sections 2 and 3, 10 min → Section 4 vanishing gradients, 10 min → Section 5 flipping the first signal, 15 min → discussion of the experiment table below Section 6, 5 min → check questions, 5 min.

Workplace OJT, 90 min

Focused on applying to work

Classify team data with the Section 9 model picker, 15 min → Sections 4 and 5, 20 min → run and interpret Code 2, 30 min (how to report "it depends on the seed") → Section 7 anomaly-detection ideas, 15 min → prompt cards, 10 min.

1:1 tutoring

Pace by level

Beginners: Sections 1, 3, 5 and 9 only. Intermediate learners: have them trace the c computation of Section 5 by hand for three steps, compare it with the lstm function in Code 1, and derive the parameter formula of Section 6 themselves.

Teaching points by section

SectionQuestions to pause and askHands-on tips
1 One frame"Can you tell it is approaching from the third frame alone?"Pause at "Current frame only," then switch to "Sequence"
2 Feedforward"If sentence length differs every time, what happens to concatenation?"Toggle length 3 and 5 and watch the number of input slots change
3 Loops & unrolling"How many sets of weights do the five unrolled boxes have?"Toggle between the folded view ↔ unrolled view
4 RNN"If increasing w_h keeps memory longer, why does learning get harder?"Change from 0.9 → 1.5 and compare h12 and the gradient numbers
5 LSTM"If you flip the first signal, what happens to the memory 20 steps later?"At b_f = 5, switch +1/−1, then change to b_f = 1 and see the difference
6 GRU"Why did LSTM and GRU, which have gates, fail at T = 100?"Read seeds 0 and 1 of the experiment table side by side
7 Autoencoder"Why does the error suddenly drop when the bottleneck is 4?"Increase the bottleneck from 1 → 4 watching the green curve, then switch to the mushroom body view
8 Notation & history"How does the 1997 LSTM differ from today’s LSTM?"Click through the timeline to the 2000 forget gate
9 Model picker"Where does our team’s data fit?"Have each participant choose answers, then discuss where the recommendations diverge

Top 5 common misconceptions

MisconceptionCorrect explanationScene to show
Using LSTM solves the long-memory problemLSTM is a structure that can hold long-term memory. Whether learning finds those weights depends on the task, training time and initializationExperiment table below Section 6 (T = 100)
SimpleRNN always forgets after just a few stepsDepending on initialization and the task, it can remember for quite long. In this experiment, both seeds succeeded up to T = 50Experiment table below Section 6 (T = 50)
LSTM had a forget gate from the startIt was not in the original 1997 paper; Gers et al. added it in 2000Section 8 timeline
GRU is always faster and better than LSTMIt only has fewer parameters, about 3/4; performance depends on the task. At T = 50 in this experiment, only GRU failedSection 6
Autoencoders imitate how the brain stores memoryIt is a computational technique that summarizes by narrowing. The fruit fly mushroom body actually expandsSection 7 mushroom body view

Check questions (click to expand the explanation)

Q1. If you average a sequence and feed it to a feedforward neural network, what information is lost? Explain with an example.

Order is lost. 100 → 110 → 120 (rising) and 120 → 110 → 100 (falling) both average to 110, so the network cannot tell them apart.

Q2. If you feed 1, 0, 0 into the one-number RNN h_t = tanh(x_t + 0.9·h_(t−1)), what are h_1 and h_2?

h_1 = tanh(1) ≈ 0.762, h_2 = tanh(0.9 × 0.762) = tanh(0.686) ≈ 0.595.

Q3. If the forget gate is constant at f = σ(5) ≈ 0.9933 and the input gate is fully closed, how many times its initial value is the cell state written at the first step by the 20th step?

It is multiplied over 19 steps, so 0.9933 to the 19th power ≈ 0.880×. The actual value in Section 5 (0.988 → 0.864, about 0.875×) is slightly smaller because the input gate is not fully closed (0.007), so the distractors leak in little by little. This sequence’s distractors are ten −1s and nine +1s.

Q4. With 16 hidden units and input size 1, how many recurrent-layer parameters do SimpleRNN, LSTM, and GRU (textbook formula) have?

One set = 16 × (16 + 1) + 16 = 288. SimpleRNN 288, LSTM 4 × 288 = 1,152, GRU 3 × 288 = 864. Keras’s default GRU adds 48 biases, giving 912.

Q5. In Code 2, LSTM failed at T = 100 with both seeds. Can we conclude that "LSTM cannot do long-term memory"?

No. With properly set weights, as in Section 5, an LSTM can hold long-term memory. The failure means gradient descent did not find those weights within 30 epochs. You can draw a conclusion only after re-running the experiment with different training time, learning rate and amount of data, and repeating it with several seeds.

Colab assignments

Basic

In the lstm function of Code 1, change the input gate to always stay open (i = σ(5)). Compute what happens to the cell state 20 steps later when the first signal is flipped, and write the role of the input gate in two sentences.

Challenge

In Code 2, try at least one way to turn the T = 100 failure into a success (more epochs, a different learning rate, more data, etc.). Report the results in a table with 3 or more seeds, and write with evidence what worked.

Adjusting difficulty and further learning

BeginnersSkipping the gradient graph of Section 4 and Sections 7 and 8 does not break the flow. Just make sure to show "flipping the first signal" in Section 5
AdvancedGeneralize the gradient product formula of Section 4 to multiple hidden units (matrices), and discuss the relationship between eigenvalues and vanishing/exploding gradients [R2]
Further learningSangsun Ahn, “Data Science for Business” lecture notes (data science in general) · STEP [K-Digital] AI Natural Language Processing (linked to 3-5 and 3-8) · STEP [K-Digital] Fundamentals of Fintech Data Analysis (linked to 3-6)
Materials for instructorsGrading criteria for the check questions and sample answers for the assignments are on the answer key page for educators (3-7)

Well done!

You’ve read “3-7 Sequence Models” all the way through. The lab members each have a word to sum it up.

the Professor
the ProfessorWe met models that walk one step at a time, carrying memory.
  • An RNN holds memory in its hidden state and passes it to the next step
  • LSTM’s doors (gates) can hold long-term memory
  • Being able to hold it is different from learning actually finding it
Assistant
the Assistant50% again at T = 100!
the Fly Researcher
the Fly ResearcherThe first signal… what was it again…