5-4 From Words to Motion — Placing Words and Gestures on the Same Map

1Mission

Fourth order — "Make it move when told in words"

With generator E in 5-3, we had to pick "the spot where throws gather" ourselves. This time, people pick it with words. For that, words must also become points on the map.

[SYSTEM] 06:20 · 5-3 completion report received. Next order: choose motions with sentences. Attached: a seventh scene, "dancing", and 4 sentences attached to each scene.

the DoctorI attached a description sentence to each motion. "캔을 멀리 던진다" (throws a can far), "반갑게 손을 흔든다" (waves a hand happily)… We tie each sentence and its motion into a pair: partners close together on the map, strangers far apart!

the AssistantBut sentences are letters. Don’t letters have to be turned into numbers too before they go into a funnel?

the Fruit Fly ResearcherLet’s start with the easiest way. If a word piece from the dictionary is in the sentence, 1; if not, 0. That’s a bag of words. It’s crude, but perfect for seeing where things go wrong.

the Lab RobotWith the dancing scene added, there are now seven kinds of motion. Each kind has four sentences, and one of them is hidden away for testing.

A stick figure types on a glowing keyboard; word pieces stream out of a speech bubble and become a line of five stick figures dancing more and more excitedly, while the Assistant, the Doctor and the robots cheer — A sentence scatters into pieces and becomes a motion. Sections 2–5 build this path step by step.

2Words are numbers

Words are numbers too — a 40-slot bag of words

Type a sentence or click an example. Of the 40 word pieces in the dictionary, only those found in the sentence become 1. These 40 numbers are the input to the text funnel.

A text-prompt stick figure among speech bubbles and glowing square pieces: it throws the pieces, lines them up in a row, lifts them in its hand, and a small stick figure dances — In charge of text prompts, splitting sentences into pieces

Explanation for beginners

A bag of words sees a sentence only as "which words are in it". It throws away word order, particles and context. So "버린" (threw away) in "길에 캔을 던져 버린다" (throws a can away on the street) and "버린" in "쓰레기통에 쓰레기를 버린다" (throws trash into the bin) switch on the same slot — even though one is littering and the other is normal. This crudeness shows up as real mistakes in Section 6. Ways to put a sentence’s order and context into numbers as well are covered in 3-5 and 3-8.

3Pair matching

Pair matching — tangled lines find their partners

Sentences are on the left, motions on the right (deliberately shuffled). The thickness of a line is the probability that the sentence picks that motion. Press "Play training" or drag the slider to advance training.

Training step0

Left sentences

Results of Code 1 — by training step

Training step	Pair-matching loss	7 test sentences → own kind among the 5 nearest test motions	140 test motions → kind of the nearest sentence is correct
0	—	2.9%	10.0%
10	1.763	28.6%	55.0%
25	1.317	65.7%	72.1%
50	0.395	71.4%	96.4%
100	0.090	71.4%	100.0%
400	0.010	71.4%	100.0%

The training on screen above re-runs the same computation in the browser (with different random numbers), so the numbers at each step differ slightly. 5 of the 7 test sentences found their own motion. The 2 that did not are examined in Section 6.

What does contrastive learning fix?

Pick one scene and one sentence for each kind. That makes 7 pairs.
Fold the 7 sentences with the text funnel (40 → 8) and the 7 scenes with the motion funnel (the 16 numbers from 5-3 → 8).
Multiply the two (dot product) to make a 7 × 7 score table. The diagonal holds the true pairs.
Turn each row and each column into probabilities with softmax, and nudge the weights of both funnels so the diagonal probabilities move toward 1. Repeat this 400 times.

CLIP [R1], which aligns words and images this way, and MotionCLIP [R2], which aligns words and motions, work on the same principle. Real models use a transformer as the text funnel and hundreds of millions of pairs.

4Same map

Words and motions on the same map

The points on the 8-slot map were scaled to length 1 (so only direction counts) and then flattened onto the two directions in which they spread the most. Circles are motion scenes, diamonds are sentences. Move the training step from 0 to 400 and see whether each sentence moves next to its own cluster of motions.

Training step400

After training, circles (motions) and diamonds (sentences) of the same color gather together. Words and motions now live in "the same neighborhood of the same map". Now, if you put one sentence on the map, you can pull out a motion near it or draw a new motion from that neighborhood.

5Words → motion

Type a sentence and the stick figure moves

Type a sentence and press Enter. "Find" pulls out the nearest scene on the map as is, and "Make" draws a new motion from the spot where the 5 nearest scenes gather. Join two sentences with "~다가" (while …) or "~고 나서" (after …) and the two motions are stitched one after the other.

Mode

Results of Code 2 — is the generated motion really that kind?

Mode	21 training sentences (×5 each)	7 test sentences (×5 each)
Find	100.0%	71.4%
Make	100.0%	71.4%

The kind of each generated motion was judged by which kind’s center is nearest in the 5-3 latent space. "춤춰" (Dance!) → the generated motion was judged as dancing, with a bone-length error of 5.4%. Stitching: "걸어가다가 손을 흔든다" (waves a hand while walking) → first half walking past · second half waving; "손을 흔들고 나서 춤춰" (wave, and then dance!) → waving · dancing; "캔을 줍고 쓰레기통에 넣는다" (picks up a can and puts it in the bin) → picking up · putting in the bin.

To make "Make" better

In this part, "Make" approximates the spot where the five nearest scenes on the map gather with a mean and a spread, and unfolds it with the linear decoder from 5-3. Real text-to-motion models instead use diffusion models that take the sentence as a condition and strip noise away to reveal a motion (MDM [R4]), or turn motion into tokens and keep writing the next token (T2M-GPT [R5]). For training data, HumanML3D [R3], in which people wrote sentences for every motion, is widely used.

6Limits

Where the bag of words fails, and the final mission

2 of the 7 test sentences found the wrong motion. In both cases the reason is clear.

Test sentence	Correct answer	Motions found (5 nearest)	Why it failed
"길에 캔을 던져 버린다" (throws a can away on the street)	Throw far	putting in the bin × 5	"던져" (throw) is a piece that never appeared in the training sentences, and "버린" (threw away) is a piece learned from a binning sentence ("쓰레기통에 쓰레기를 버린다", throws trash into the bin). Without context, it got dragged toward "버린다 = binning"
"손을 펴서 떨어뜨리고 걸어간다" (opens the hand, drops it and walks away)	Drop at feet	walking past × 5	"걸어" (walk) and "간다" (goes) pulled hard toward walking past. A bag of words does not know which word is the main action (dropping)
"걷다가 손 흔들기" (walking, then waving) (Section 5)	walking past → waving	First half is wrong	"걷" (walk) is in the dictionary but never appeared in a training sentence (the training sentences all use "걸어"). A slot that was never learned does not point anywhere on the map

The "Motions found" in the table above are the results of Code 1. In Section 5, which re-trains in the browser, "길에 캔을 던져 버린다" (throws a can away on the street) sometimes goes toward picking up instead. Where it gets dragged depends on the random numbers in training, but the reason it cannot find throwing (a bag of words that does not know context) stays the same.

What we learned in this part

Words become points too: turn a sentence into numbers and fold it with a funnel, and it lands on the same map as motions.
Learning from pairs: contrastive learning aligns two funnels with nothing but "partners close, strangers far". Loss 1.90 → 0.01.
Pull out and draw from the map: find and make, plus stitching in sentence order.

Open problems

The text funnel is too shallow: handling order, context and unseen words takes a text encoder such as a transformer (3-8).
A stick figure is still a stick figure: we made it dance, but it is still just a skeleton. Dressing this skeleton in a person’s (the Assistant’s) appearance and turning it into a video is the final mission.

Musical notes floating around a stick figure standing in front, side and back views; in the bottom row, four dance poses: swinging the arms, spinning, pointing and lifting a knee — Next mission preview · the stick figure’s dance move cards

the DoctorWith the single word "춤춰" (Dance!), the stick figure danced. The last step is dressing this skeleton as a person. Assistant, ready?

the Assistant…So I’m the one dancing after all. You mean when the stick figure dances, out comes a video of me dancing exactly the same way?

the Fruit Fly ResearcherWe’ll open a real video in 5-5. There was a number called "motion match score" on the screen, too.

Colab code — run this page yourself

Both code notebooks use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab via File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · Bag of words and contrastive learning — words and motions on the same mapLinked to Sections 2–4 · numpy · about 5 s on CPU

Adds the dancing scene, turns sentences into a bag of words and motions into 16 numbers, then aligns the two funnels with contrastive learning. Prints the loss at each training step and the hits for the test sentences. To use your own data put description sentences for your motions in CAPTIONS and the word pieces to use in VOCAB.

Output (run on 2026-10-05, numpy 2.4 · CPU). The seed is fixed, so it gives the same result every time.

Code 2 · Type words, get a motion — find, make, stitchLinked to Section 5 · numpy · about 5 s on CPU

Uses the two trained funnels to find or newly make a motion from a sentence, and stitches sentences joined with "~다가" (while …) one after the other. Judges the kind of each generated motion in the latent space. The table in Section 5 is built directly from this output.

Output (run on 2026-10-05, numpy 2.4 · CPU). The seed is fixed, so it gives the same result every time.

AI prompts — using this code as a starting point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. When you press "Copy," the selected code is attached after the prompt. Just replace the parts in 〔〕 with your own situation.

Code to attach

The AI's answer is only a starting point. Always verify suggested papers, figures, and library functions against the original sources and official documentation. When using footage of real people, first check filming consent and personal data processing rules.

Glossary

The same dictionary as the yellow dotted underlines in the text. Click a section name to jump to where the term first appears.

Analogy collection — and where the analogies break

Analogies are a first step toward understanding, but they can also plant misconceptions. Analogies to human perception are marked by strength. Direct there is evidence that it actually does the same thing · Partial only some properties are similar · None no known counterpart

Concept	Everyday analogy	Human perception analogy	Where the analogy breaks
Bag of words	Cutting a sentence into word cards, putting them in a bag and shaking it	None	Cards taken out of the bag have no order. "The dog bites the man" and "The man bites the dog" become the same
Contrastive learning	A pair-matching card game: stick matching pairs together, pull the others apart	Partly a child who learns what words mean by hearing them while watching the action	A child learns in a few tries, but a model has to see the same pair hundreds of times, and a word with no pair stays unknown forever
Same map	Putting two languages together in one dictionary	None	In a dictionary, people decide the correspondences, but on this map the correspondences emerge from pair data alone. Words with few pairs end up in the wrong place
Find vs. make	Borrowing a book from a library vs. reading similar books and writing something new	None	New writing has the author’s intent behind it, but “make” in this part only draws something near the average of the five nearest scenes
Stitching	Choreography instructions like "first do ~, then do ~"	Partly moving the body in the order of the sentence	A choreographer makes the transitions flow naturally, but here we simply blend 10 frames

About the author and references

E-1 About the author · E-2 Academic references · E-3 Sources of figures and numbers

E-1 About the author

Ahn Sangsun (CEO, M-Robo Co., Ltd.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Co., Ltd., a spin-off venture of Maeil Business Newspaper, where he currently serves as CEO. M-Robo began as an in-house venture at a news company and is a media venture that has provided a robot reporter service (StockBot) to Kookmin Ilbo and carried out numerous projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. While serving as CEO, he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of "AI Crime and Self-Defense" and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a Ph.D. in Economics (Dongguk University, microeconomics and mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Lectures

Ahn Sangsun (2025). "Data Science for Business" lecture notes. Department of AI Convergence, Open Cyber University of Korea.
Ahn Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn Sangsun (2024). [K-Digital] Fintech Data Analysis Basics. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work because of a site redesign, please visit www.step.or.kr.

Papers and presentations

Ahn Sangsun (2025). A study on a household expenditure prediction model using the Korean Retirement and Income Study panel data: An AI model-based approach. Future Society, 16(2). Full text
Ahn Sangsun (2025). A study on a household expenditure prediction model using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn Sangsun (2025). A study on an algorithm for detecting suspected drug transactions based on SNS text: Focusing on pattern and context analysis. Future Society, 16(1). Full text
Ahn Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. Future Society, 15(3). Full text
Ahn Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation at the 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation at the 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
Ahn Sangsun (2024). Analysis and forecasting of virtual asset time-series data: Comparing time-series models and AI models, and suggesting improvements. Future Society, 15(1). Full text

E-2 Academic references

[R1] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. ICML 2021. arXiv:2103.00020 — Section 3 (contrastive learning, CLIP)
[R2] Tevet, G., Gordon, B., Hertz, A., Bermano, A. H., & Cohen-Or, D. (2022). MotionCLIP: Exposing human motion generation to CLIP space. ECCV 2022. arXiv:2203.08063 — Sections 3, 4
[R3] Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., & Cheng, L. (2022). Generating diverse and natural 3D human motions from text. CVPR 2022. — Section 5 (HumanML3D dataset)
[R4] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., & Bermano, A. H. (2023). Human motion diffusion model. ICLR 2023. arXiv:2209.14916 — Section 5
[R5] Zhang, J., Zhang, Y., Cun, X., Huang, S., Zhang, Y., Zhao, H., Lu, H., & Shen, X. (2023). T2M-GPT: Generating human motion from textual descriptions with discrete representations. CVPR 2023. arXiv:2301.06052 — Section 5

E-3 Where the pictures and numbers come from

The scene illustrations (text–motion matching, from prompt to motion, text prompt, dance move cards) are pixel art made by the author
All scenes and sentences are synthetic data. The 28 sentences and 40 word pieces were chosen by the author
The tables under Sections 3 and 5 are the results of actually running Code 1 and 2. The training and generation on screen re-run the same computation in the browser, so the numbers differ slightly

Tips for educators

A teaching guide for instructors, on-the-job (OJT) trainers and tutors. Times are estimates, so adjust them to your actual class.

Audience	University students and working adults who have finished 5-3. Planners and practitioners who want to know what goes on inside generative AI where "you type a prompt and get a result"
Prerequisites	The latent space from 5-3; dot product and softmax (3-2, 3-8)
Learning goals	① Explain the simplest way to turn a sentence into numbers (bag of words) and its limits ② Describe how contrastive learning aligns two funnels by "partners close, strangers far" ③ Give examples of the difference between find, make and stitch on a shared map
Estimated time	Reading 30–40 min · animation demos 25 min · Colab practice 25 min (Code 1 12 min, Code 2 13 min)

Three ways to run it

Lecture, 50 min

Concept-focused

Intro 5 min (Section 1) → Section 2 bag of words 8 min → Section 3 training playback 15 min → Section 4 map 7 min → Section 5 trying sentences 10 min → Section 6 failed sentences 5 min.

On-the-job training, 90 min

Prompt-design focus

Sections 2–3, 20 min → discussion of "words the model knows and words it does not" using the Section 6 table, 15 min → in Section 5, teams look for sentences that work and sentences that fail, 20 min → add new sentences to Code 1 and retrain, 25 min → wrap-up 10 min.

1:1 tutoring

Pace by level

Beginners: Sections 2, 3 and 5 only. Have intermediate learners work out by hand, with a 2 × 2 table, why the single line G = (P − I + Q − I) / 14 in Code 1 "raises the diagonal and lowers the rest".

Teaching points by section

Section	Question to pause and ask	Hands-on tip
2 Words are numbers	"Why do '캔을 던진다' (throws a can) and '던진다 캔을' (same words, swapped) get the same numbers?"	Click the two examples in turn
3 Pair matching	"How many steps did it take for the lines to find their partners? And for the test sentences?"	After playing the training, switch to "Test sentences"
4 Same map	"Where were the diamonds before training?"	Drag the step slider from 400 → 0
5 Words → motion	"How do the results of find and make differ?"	Press "춤춰" (Dance!) several times in Make mode
6 Limits	"What would we have to change so that '던져 버린다' (throws away) is found correctly?"	Type that sentence in Section 5 and check for yourself

Top 5 common misconceptions

Misconception	Correct explanation	Scene to show
AI understands the meaning of a sentence and then picks a motion	It only turns the sentence into numbers and finds a nearby spot on the map. In this part, the numbers only record whether each word piece is present	Section 2, Section 6
Contrastive learning needs answer labels	All it needs are pairs saying "this sentence and this motion go together". Kind names are not used	Section 3, explanation
If the training loss is close to 0, it will also get unseen sentences right	The loss is measured on training sentences. Only 5 of the 7 test sentences were right	Section 3 table
Longer, more detailed sentences always work better	If words outside the dictionary or words from other motions creep in, the sentence can get dragged somewhere completely wrong	Section 6, "손을 펴서 떨어뜨리고 걸어간다" (opens the hand, drops it and walks away)
"Make" creates entirely new motions that were not in the training data	It draws near the spot where nearby scenes on the map gather. It can hardly go beyond the kinds of motion it learned	Section 5, "날아올라" (Fly up!)

Check questions (click to expand the answers)

Q1. If the dictionary has four slots [캔, 던지, 흔들, 손] (can, throw, wave, hand), what is the bag of words for "손을 흔들며 캔을 던진다" (throws a can while waving a hand)?

[1, 1, 1, 1]. All four pieces are in it. This sentence ends up somewhere between throwing and waving.

Q2. If the pair score table is [[3, 1], [0, 2]], what are the softmax probabilities of the first row? How does training try to change this row?

e³ / (e³ + e¹) ≈ 0.881, e¹ / (e³ + e¹) ≈ 0.119. Training adjusts the two funnels to push the diagonal (0.881) toward 1 and the non-pair cell (0.119) toward 0.

Q3. Why is test motion → sentence 100% while test sentence → motion is only 5 out of 7?

On the motion side there are 420 scenes, so each kind has a well-defined shape; on the sentence side there are only 3 training sentences per kind, and the bag of words does not know context. When an unseen combination of words comes in, the text funnel wobbles.

Q4. Why did the first half of "걷다가 손 흔들기" (walking, then waving) come out wrong?

"걷" (walk) is in the dictionary but never appeared in a training sentence. The weights for that slot are still the small random values they started with, so it does not point anywhere meaningful on the map. Change it to "걸어가다가" (while walking) and walking past comes out.

Q5. Which carries a greater privacy risk, find or make? Why?

Find. It pulls out training scenes as they are, so if they were real people’s motion data, that person’s movements would come out unchanged (like generator C in 5-2). Make draws something new from the spot where several scenes gather.

Colab assignments

Basic

In Code 1’s CAPTIONS, change one of the throwing training sentences to something similar to "길에 캔을 던져 버린다" (throws a can away on the street) and retrain. Report how the test-sentence hits change.

Challenge

Instead of the bag of words, write a bag function that uses "overlapping two-character pieces" (e.g. "던져" → "던", "져", "던져"), and compare by test-sentence hits whether it handles unseen words ("걷다가", while walking) better.

Adjusting difficulty and going further

Beginners	The training playback in Section 3 alone is enough to get across "it learns by matching pairs"
Advanced	Have learners put Figure 1 of the CLIP and MotionCLIP papers next to Section 3 of this part and present the similarities and differences
Next episode	5-5 From "Dance!" to a Dance Video — dressing the stick figure’s dance in the Assistant’s appearance