5-3 Encoder · Decoder — Folding and Unfolding Motion

1Mission

Third order — "Find the knobs in the data"

Generator D in 5-2 won because it knew the three knobs (speed · position · size) in advance. Real data does not come with knobs attached. This mission is to find the knobs on our own, looking only at 600 scenes.

[SYSTEM] 04:55 · 5-2 completion report received. Next order: motion compression. Attached: 600 scenes of six types, one funnel blueprint.

The DoctorDo all 1,196 numbers really move independently? No. When the elbow moves, the wrist follows, and this frame is almost the same as the last one. So we can fold it into far fewer numbers.

The AssistantThe folding side is the encoder, the unfolding side is the decoder! If we fold and unfold and get the original motion back, that means the numbers in the middle hold the whole motion, right?

The Fruit Fly ResearcherThose numbers in the middle are exactly the knobs we were looking for in 5-2. And there's a bonus. Jitter gets wiped out, occluded elbows get filled in, and littering detection gets better too.

The Lab RobotI have prepared funnel sizes from 1 to 16. The 16 directions computed by Code 1 are loaded on this page.

In the upper screen, four stick figures gather into one small capsule and then unfold into three rows of walking stick figures; in the lower part, a blurry stick figure becomes sharp and a bar chart rises. The Assistant clenches a fist in delight and the robot holds up a sign with a rising graph — Folding and unfolding. In section 4 you will create the scene where a blurry input becomes a sharp output yourself.

2Funnel

The funnel — fold 1,196 into k, then unfold again

The encoder folds the original on the left into k numbers (the purple bars in the middle), and the decoder unfolds them back into 1,196 (right). Shrink the bottleneck size k and see how far it can be folded.

Bottleneck k

Scene

Code 1 results — error after folding and unfolding 300 test scenes

Bottleneck k	Compression ratio	Explained variance	Reconstruction error (RMSE)	Bone-length error
1	1,196x	85.72%	0.1021	6.16%
2	598x	95.07%	0.0598	7.65%
3	398.7x	97.84%	0.0395	2.78%
5	239.2x	99.02%	0.0266	1.95%
8	149.5x	99.46%	0.0195	1.86%
16	74.8x	99.80%	0.0123	1.50%
32	37.4x	99.90%	0.0088	1.43%

The test scenes contain pose-estimation jitter (σ = 0.008). The error at k = 32, 0.0088, is almost the same as that jitter. Beyond that point, it starts memorizing the jitter rather than the motion.

Explained for beginners — how this funnel was made

We collected 600 scenes and found, one after another, "the directions in which the motion differs the most" (principal component analysis, PCA). The encoder subtracts the average scene from a scene and projects it onto these k directions to get k numbers. The decoder adds the k directions, each scaled by its number, to the average scene. It is the simplest kind of autoencoder, using only addition and multiplication. An autoencoder built from a neural network uses layers and activation functions to capture curved relationships too (3-7). The 16 directions on this page are the values computed by Code 1, loaded as is.

3Latent map

The latent map and knobs — knobs the data found on its own

This map plots the 600 scenes by knobs 1 and 2. Click a point to choose a scene, then turn the four knobs below. The knobs have no names. Watch what changes and give them names yourself.

Knob 1 Knob 2 Knob 3 Knob 4

Scenes of the same type cluster close together on the map. Nobody told it "this is a throw"; we only folded the data, and still this happens. Knob values are in units of the training scenes' standard deviation (σ). Outside ±2σ is a region that barely existed in the training data, so the unfolded shape looks awkward.

Knob	Where it moves the most	Nickname from observation	Average per scene type (Code 1)
1 (85.7% of the motion)	x of all joints, late in the scene	"Where is it at the end" — does it stay in front of the bin or walk off to the right?	Put in −15.3 · Throw +8.3 · Pass by +6.4 · Drop +6.7
2 (9.4%)	x of the right wrist · elbow and the ankles, early in the scene	"Where does it start, and which side is the right arm on at the start?"	Pass by −4.4 · Wave +4.1
3 (2.8%)	y of the right wrist · elbow · head	"Does it raise the arm or bend at the waist?"	Wave +2.6 · Pick up −2.1
4 (1.2%)	x of the right wrist and the ankles, mid-scene	"Side-to-side swing of the arms and legs in the middle of the scene"	Pick up +1.3 · Throw −0.9

How is this different from the knobs in 5-2?

The knobs in 5-2 (speed · position · size) were set by people when making the scenes. The knobs here are the directions in which the data differs the most, picked one after another. So they have no names, and one knob mixes several meanings (knob 1 is both position and "did it walk off?"). In exchange, they can be found without anyone telling the model anything. This is the latent space.

4Jitter · occlusion

Performance boost ① — wipe out jitter, fill in an occluded elbow

In real pose estimation, joint points jitter, and when an arm is hidden behind the body, the point is lost. The funnel only lets through "things that look like human motion", so when you fold and unfold, jitter is filtered out and blanks are filled in plausibly.

Jitter σ

Scene

Removing jitter (Code 1, 120 scenes)

With jitter σ = 0.01 the error goes 0.0100 → 0.0082, with σ = 0.03 it goes 0.0300 → 0.0090, and with σ = 0.05 it goes 0.0501 → 0.0104. The bigger the jitter, the bigger the effect. Jitter points in random directions, so it barely registers on the funnel's 16 directions.

Filling in an occluded elbow (Code 1, 120 scenes)

If we fit the 16 latent numbers using only the visible entries with least squares and then unfold them with the decoder, even the blanks get filled in. The position error is 0.0230 for "patching with the midpoint of shoulder and wrist" versus 0.0108 for "filling in with the decoder", about 2.1 times more accurate. This is how we fix the problem mentioned in 5-1: "when an arm is hidden by the body, the joint can't be found."

5Littering detection

Performance boost ② — catching littering with the folded numbers

In 5-1 we said, "learning the coordinates as is requires much more data." This time we first fold the coordinates of the joints and the can (1,288 numbers) with the encoder, and then train the simplest logistic regression on the k folded numbers.

Training scenes (per type)	All 1,288 as is + MLP	Fold to k = 8 + logistic regression	Fold to k = 16 + logistic regression
25 (150 total)	96.7%	95.0%	99.7%
50 (300 total)	97.0%	95.3%	100.0%
100 (600 total)	98.0%	98.0%	100.0%

k = 16 + logistic regression: with only 25 per type (150 total), it got 299 of the 300 test scenes right. "Where the can ends up" and "where the person went" are folded together into the 16 numbers, so even a single straight line can separate them.

k = 8 is not enough: small movements such as the can's final position live in the 9th and later directions, so folding into 8 loses part of them. Folding as tightly as possible is not always the answer.

Why this differs from 92.3% in 5-1: 5-1 marked "no object" with the coordinate −9; here it is marked with (0, 0). Just changing how the input is marked lifts the MLP to 98%. All three columns of this table were compared with the same input.

Why does folding help?

The encoder was built by looking only at scenes, without labels (it used all 600, but never saw the littering/normal answers). The resulting "summary of the motion" has its noise filtered out and has few numbers, so the classifier does not get lost even with few labeled scenes. Learning a representation first from unlabeled data and finishing with a few labels is the same basic flow used to train today's large models. Note, though, that these numbers come from synthetic data. With real video they are lower.

6Generation

Performance boost ③ — creating motion without knobs (generator E)

The Evaluator from 5-2 is back. This time, generator E does not know the knobs set by people. It looks at where the throws gather on the latent map (mean and standard deviation), draws 16 new numbers, and unfolds them with the decoder.

Side by side with the four generators from 5-2 (Code 2 · 5-2 Code 2, 20 generated each)

Generator	Knows the knobs in advance?	Bone-length error	Jitter	Max wrist speed	Diversity	Novelty
A One average	—	1.7%	0.0045	1.36	0.000	0.033
B Dice per coordinate	—	44.2%	0.2659	10.00	0.149	0.109
C Pick one and copy	—	4.7%	0.0300	2.75	0.125	0.000
D Sample 3 knobs	Yes (set by people)	0.0%	0.0125	2.71	0.123	0.022
E Draw from 16 latent numbers	No (found by the data)	3.7%	0.0122	1.76	0.114	0.047

What E achieved: even without knowing the knobs, it does not fall apart like B (bone error 44.2% → 3.7%, jitter 0.27 → 0.012). It is varied (0.114), and it did not copy (0.047).

What E could not do: the maximum wrist speed is 1.76, so the throw lost its force. Because it draws near the mean in the latent space and simply adds straight-line directions, the brief moment of a fast swing gets smeared out. Bone lengths are also not as accurate as the real thing (0%).

A better decoder

A linear decoder only adds directions in proportion, so it cannot capture curved relationships like "the arm swings around in a circle." That is why real motion-generation research uses nonlinear encoders and decoders with stacked layers [R2], and trains them so the latent space takes the shape of a normal distribution, creating a space where "whatever you draw unfolds plausibly" (VAE [R1]). There are also motion VAEs that handle the time axis as well [R4].

7Limits

The limits of folding, and the next mission

The encoder · decoder boosted both detection and generation. Still, some things remain.

What we learned in this part

1,196 numbers fold into 16: 99.8% of the motion remains. Motion data is far simpler than it looks.
Folding filters: jitter is wiped out (0.030 → 0.009) and blanks are filled in (2.1 times more accurate).
Folding lets it learn from few labels: 99.7% with 25 per type.
Drawing from the folded space creates motion: generation works even without knobs set by people.

Open problems

The limit of linearity: a forceful throw gets smeared. Nonlinear decoders and VAEs are needed.
Unseen motions: the funnel only knows the shapes of the six types it learned. A motion it has never seen (e.g., falling down) gets pulled toward a known motion when folded and unfolded. Conversely, these "scenes with large reconstruction error" can be used as an alarm for abnormal motion.
How to choose the motion you want: for now, we picked "where the throws gather" ourselves. Couldn't a person choose "waving" with words? Just put words on the same map too.

Heart · star · music note · exclamation mark · moon icon cards on the left are connected by glowing lines to stick-figure motion cards on the right; the Assistant presses the top card while the Doctor watches, chin in hand — Next mission preview · Pairing words and motions on the same map

The DoctorWe turned motions into points on a map. So what if we also turn the words "wave your hand" into a point on the same map? Put in words and the motions near them come out!

The AssistantSo we fold words with an encoder too! If we line up the text funnel and the motion funnel on the same map…

The Fruit Fly ResearcherThat's 5-4. You'll watch tangled lines find their partners as they learn.

Colab code — run this page yourself

Both code notebooks use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab via File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · Folding motion into k numbers and unfolding it (linear autoencoder)Linked to sections 2–4 · numpy, matplotlib · about 3 s on CPU

Builds encode · decode from 600 scenes and measures the reconstruction error for each k, jitter removal, and filling in the occluded elbow (fill). The 16 directions on this page come from here. To use your own data put your data shaped (number of scenes, number of values) in place of S. Sensor and log data can be folded the same way.

Output (run on 2026-10-05, numpy 2.4 · matplotlib 3.10 · CPU). The seed is fixed, so it gives the same result whenever you run it. Two graphs (explained variance by k, and the latent map) are also produced.

Code 2 · Boosting littering detection and motion generation with the folded numbersLinked to sections 5 · 6 · numpy, scikit-learn · about 40 s on CPU

Redoes 5-1's littering detection, comparing "MLP on the raw input" with "fold, then logistic regression" while varying the number of training scenes, and measures generator E with 5-2's Evaluator scoring. The tables in sections 5 · 6 are made directly from this output.

Output (run on 2026-10-05, numpy 2.4 · scikit-learn 1.8 · CPU). The data seed is fixed, but the MLP numbers may change slightly depending on the library version.

AI prompts — using this code as a starting point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. When you press "Copy," the selected code is attached after the prompt. Just replace the parts in 〔〕 with your own situation.

Code to attach

The AI's answer is only a starting point. Always check suggested papers, figures, and library functions against the original sources and official documentation. When using real people's motion data, first check filming consent and the dataset's terms of use.

Glossary

The same dictionary as the yellow dotted underlines in the text. Click a section name to jump to where the term first appears.

Analogy collection — and where the analogies break

Analogies are a first step toward understanding, but they can also plant misconceptions. Analogies to human perception are marked by strength. Direct there is evidence that it actually does the same thing · Partial only some properties are similar · None no known counterpart

Concept	Everyday analogy	Human perception analogy	Where the analogy breaks
Encoder · decoder	Folding clothes into a suitcase and unfolding them at the hotel	Partly Remembering a long dance as a few words like "left foot, clap, spin" and then dancing it again	Clothes stay the same when folded, but the funnel throws away small wrinkles (jitter). What is thrown away does not come back when unfolded
Bottleneck size k	The character limit of a summary	None	In a summary, a person picks what matters, but this funnel picks "what varies a lot." It can miss small but important differences (the can's final position)
Latent space	Plotting restaurants on a map with two axes: "price · spiciness"	None	People choose the axes of a restaurant map, but the axes of a latent space are chosen by the data and have no names
Occlusion recovery	Reading erased letters by guessing from the surrounding context	Partly Human perception that guesses a hidden arm from the posture of the whole body	The guesses are right only within the range of motions it has learned. For a posture it has never seen, it fills in wrongly, toward a posture it knows
Sampling from the latent space	Picking "somewhere near this neighborhood" on a map and going there	None	Pick an empty area of the map (where there was no training data) and you end up somewhere strange

About the author and references

E-1 About the author · E-2 Academic references · E-3 Sources of figures and numbers

E-1 About the author

Ahn Sangsun (CEO, M-Robo Co., Ltd.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Co., Ltd., a spin-off venture of Maeil Business Newspaper, where he currently serves as CEO. M-Robo began as an in-house venture at a news company and is a media venture that has provided a robot reporter service (StockBot) to Kookmin Ilbo and carried out numerous projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. While serving as CEO, he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of "AI Crime and Self-Defense" and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a Ph.D. in Economics (Dongguk University, microeconomics and mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Lectures

Ahn Sangsun (2025). "Data Science for Business" lecture notes. Department of AI Convergence, Open Cyber University of Korea.
Ahn Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn Sangsun (2024). [K-Digital] Fintech Data Analysis Basics. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work because of a site redesign, please visit www.step.or.kr.

Papers and presentations

Ahn Sangsun (2025). A study on a household expenditure prediction model using the Korean Retirement and Income Study panel data: An AI model-based approach. Future Society, 16(2). Full text
Ahn Sangsun (2025). A study on a household expenditure prediction model using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn Sangsun (2025). A study on an algorithm for detecting suspected drug transactions based on SNS text: Focusing on pattern and context analysis. Future Society, 16(1). Full text
Ahn Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. Future Society, 15(3). Full text
Ahn Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation at the 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation at the 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
Ahn Sangsun (2024). Analysis and forecasting of virtual asset time-series data: Comparing time-series models and AI models, and suggesting improvements. Future Society, 15(1). Full text

E-2 Academic references

[R1] Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. ICLR 2014. arXiv:1312.6114 — sections 6 · 7
[R2] Holden, D., Saito, J., & Komura, T. (2016). A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics, 35(4), Article 138. doi:10.1145/2897824.2925975 — section 6 (autoencoders for motion)
[R3] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. doi:10.1126/science.1127647 — section 2 (neural-network autoencoders compared with PCA)
[R4] Petrovich, M., Black, M. J., & Varol, G. (2021). Action-conditioned 3D human motion synthesis with Transformer VAE. ICCV 2021. arXiv:2104.05670 — section 6
[R5] Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2(11), 559–572. — section 2 (the origin of principal component analysis)

E-3 Where the pictures and numbers come from

The scene illustrations (encoding funnel, decoding and performance boosts, text–motion matching) are pixel art made by the author
The encoder · decoder in sections 2–4 · 6 uses the average scene and 16 directions computed by Code 1, loaded as is (32-bit floats)
The tables under sections 2 · 4 · 5 · 6 are the results of actually running Code 1 · 2. The on-screen jitter and generated results are redrawn in the browser the same way, so the numbers differ slightly
The knob nicknames in section 3 were given by the author after looking at which joints and times each direction moves the most

E-4 Notes

[note] Knob (latent variable) — the analogy this series uses as an easy name for a latent variable. It means a small set of numbers that change the whole data together when you turn them. English-language explanations also often call each dimension of a latent space a "knob," "dial" or "slider" as a figure of speech. The formal term in papers is latent variable, and each knob the data finds in 5-3 is a principal component. Unlike a real knob, a single one can mix several meanings.

Tips for educators

A teaching guide for instructors, on-the-job (OJT) trainers and tutors. Times are estimates, so adjust them to your actual class.

Audience	University students and working adults who have finished 5-1 · 5-2. Practitioners who handle large, noisy data such as sensor, video, or log data
Prerequisites	"Motion is a table" and generator scoring from 5-2; the dot product of vectors (multiply matching positions and add them up)
Learning goals	① Explain the encoder · decoder and what the bottleneck k means, using the funnel picture ② Give examples of how folding and unfolding is used for jitter removal, occlusion recovery, learning from little data, and generation ③ Explain the limits of a linear encoder and why a VAE is needed
Estimated time	Reading 35–45 min · animation demo 25 min · Colab practice 30 min (Code 1: 10 min, Code 2: 1 min to run + 19 min to interpret)

Three ways to run it

Lecture, 50 min

Concept-focused

Introduction 5 min (section 1, the homework from 5-2) → section 2, shrinking the funnel's k, 12 min → section 3, turning the knobs, 10 min → section 4, jitter · occlusion, 8 min → sections 5 · 6, reading the tables, 10 min → check questions 5 min.

On-the-job training, 90 min

Focus on applying data compression

Section 2, 15 min → discussion of "how many numbers" vs. "how many truly independent numbers" our own data (sensors, logs) has, 15 min → sections 4 · 5 as cases of noise removal and learning from few labels, 20 min → putting our data into Code 1, 30 min → wrapping up the limits in section 7, 10 min.

1:1 tutoring

Pace by level

Beginners: sections 2 · 4 only. Intermediate learners: have them trace encode · decode in Code 1 by hand, then plot Vt[0] to check why knob 1 in section 3 is "the position at the end."

Teaching points by section

Section	Question to pause and ask	Hands-on tip
2 Funnel	"How many numbers does it take for a throw to look like a throw?"	Raise k = 1 → 3 → 8 and watch the result on the right
3 Latent map	"Nobody told it, so why did scenes of the same type gather together?"	Pick a red dot (throw) and turn knob 1 to −2σ
4 Jitter · occlusion	"How does the funnel tell jitter apart from motion?"	Turn on σ = 0.05 and elbow occlusion together
5 Littering detection	"Why is k = 8 worse than k = 16?"	Read the 25 row of the table first
6 Generation	"What does E know less than D, and what does it do less well?"	Press "Generate again" and look at the max wrist speed cell

Top 5 common misconceptions

Misconception	Correct explanation	Scene to show
Compression loses information, so performance drops	If what is thrown away is mostly noise, it actually gets better. Jitter removal and learning from little data are examples	Section 4, section 5
The bigger k, the better	A large k captures even the jitter. A small k misses small but important movements. Choose it to fit the purpose	k = 32 in the section 2 table, k = 8 in section 5
Every latent number has a meaning people can understand	The directions are set by the data and have no names. Nicknames are interpretations people add afterward	Section 3 table
An autoencoder has to be a neural network	The simplest linear autoencoder does the same job as PCA. Neural networks capture curved relationships as well	Explanation under section 2
Any point drawn from the latent space gives a plausible motion	Only near where training data existed. Making that space even is the job of a VAE	Section 3 outside ±2σ, section 6

Check questions (click to expand the answers)

Q1. If 1,196 numbers are folded with k = 16, what is the compression ratio? How much of the motion is captured?

1,196 ÷ 16 = 74.75, about 74.8x. In Code 1 the explained variance is 99.80%.

Q2. With two directions v₁ = (1, 0, 0), v₂ = (0, 1, 0) and mean 0, what do you get by folding x = (3, −2, 0.4) with k = 2 and unfolding it?

Encoder: z = (x·v₁, x·v₂) = (3, −2). Decoder: 3v₁ − 2v₂ = (3, −2, 0). The third value, 0.4, does not register on either direction and disappears. This is the principle behind "jitter getting wiped out."

Q3. Why is it wrong to patch an occluded elbow with "the midpoint of the shoulder and wrist"? What more does the decoder know?

The elbow is bent, so it lies outside the midpoint. The decoder uses the relationship learned from 600 scenes, "in this posture, the elbow is usually here" (the 16 directions), and fits it together with the other visible joints. In Code 1 the error dropped from 0.0230 → 0.0108.

Q4. The encoder never saw the littering/normal answers when it was built. Why did littering detection still improve?

The encoder learned only the structure of motion (what moves together) and turned 1,288 numbers into a 16-number summary. The summary has little noise and few numbers, so a simple classifier does not get lost even with only 150 labeled scenes.

Q5. Why is generator E's max wrist speed 1.76, lower than the training data (2.55)?

Because it draws near the mean of the throws in the latent space, and the decoder simply adds directions in straight lines. Fast swings, each with slightly different timing, cannot be expressed by a few straight-line directions and get smeared out. A nonlinear decoder or a model that handles the time axis is needed.

Colab assignments

Basic

In Code 1, change the occluded joint from the right elbow to the left knee (joint 9 → joint 10), and lengthen the occluded span from 21 frames to 40 frames. Report in a table how the error of filling in with the decoder changes.

Challenge

Create a motion not in the training data, "falling down" (a scene where lean in the pose function is raised to 1.4), and measure how much larger its reconstruction error is than for the six scene types. Propose, with a threshold value, whether reconstruction error can be used to build an "unseen motion" alarm.

Adjusting difficulty and going further

Beginners	The k buttons in section 2 and the three pictures in section 4 are enough to get "folding and unfolding filters" across
Advanced	Have them replace the linear encoder in Code 1 with a Keras autoencoder (1,196 → 64 → 16 → 64 → 1,196) and compare the reconstruction error and generator E's wrist speed
Next episode	5-4 From Words to Motion — turning words into points on the same map too