5-5 From "Dance!" to a Dance Video — How a Prompt Becomes the Assistant’s Dance

1Mission

Final order — "Make the Assistant dance with a single word: Dance!"

Up to 5-4, the stick figure understood words and danced. But it is still only a skeleton. The last mission is to dress this skeleton in a human appearance and turn it into a video. The model will be the Assistant.

[SYSTEM] 07:30 · 5-4 completion report received. Final order: prompt → dance video. Attached: the Assistant’s reference image, one video (10 s), two cut sheets.

DoctorOne photo of the Assistant takes care of appearance, and the stick figure’s dance takes care of motion. With both as conditions, a video generation model draws the Assistant anew in every frame.

AssistantLooking at the video, I’m wearing an idol outfit? In the reference image it’s a lab coat… And the screen says "motion match score 76.5%". Is that good or bad?

Fruit Fly ResearcherGood question! First we need to check how that number was calculated. Let’s take the video apart frame by frame. And let’s measure the match score ourselves.

Lab RobotAnalysis of all 240 frames of the video is complete. The number box did not change even once from start to finish.

Full-body front view of the Assistant: pink hair with a navy stripe in the bangs, buns on both sides, a white lab coat over a pale purple hoodie, jeans and pale purple sneakers — The Assistant’s reference image (front view from the character sheet). Section 3 discusses why it differs from the outfit in the video.

2In one line

From "Dance!" to video — six steps in one line

Click a box to see that step explained. Steps 1–4 are calculations built in this series; steps 5–6 are shown by the given video.

Step	Input	Output	Part of this series
1–2 Text funnel	"Dance!"	One point on the shared map (8 values)	5-4 sections 2·3
3 Motion generation	A point on the map	Stick-figure motion (frames × 13 joints)	5-3 section 6, 5-4 section 5
4 Pose map	Joint coordinates	Colored-bar drawing on a black background (per frame)	The reverse direction of 5-1 section 3
5 Pose-guided generation	Reference image + pose map	A frame of the Assistant in that pose	Section 3 of this part
6 Video	Frames	Dance video	Section 4 of this part

3Drawing the Assistant

Dressing the pose map as the Assistant — pulling a frame out of noise

Most pose-guided video generation models are diffusion models. They start from random noise and strip a little of it away at each step, guided by two conditions: "look like the reference image, pose like the pose map." Move the slider to strip away the noise.

Steps removed

Appearance encoder	Extracts features of the face, hair and clothes from the reference image and keeps referring to them while drawing (ReferenceNet-style)
Pose guider	Folds the pose map with a small convolutional neural network and adds it on top of the noise. "An arm goes here" is fixed from the very start
Temporal layers	Lets neighboring frames refer to each other so clothing colors and hair do not flicker

How this picture differs from real generation

This screen mixes noise in proportion into a finished frame (the Assistant from the video) to show "the share of the original signal left at each step" (the √ᾱ values follow a real diffusion model’s schedule). Real models handle noise in a latent space where the image is folded small, and at each step they predict "this much here is noise" and subtract it [R1]. A representative family of architectures that moves a person using a reference image and poses is Animate Anyone [R2]. The video does not state which model or settings were used to make it.

The Assistant in the reference image wears a lab coat, but the Assistant in the video wears an idol outfit. In principle, pose-guided generation carries over the reference image’s appearance as is, so the video may have been made with a different reference image in the idol outfit (e.g. looking like the "Assistant solo, 12 cuts" below), or the outfit may have been changed with a text prompt. The video alone cannot tell us which.

12 cuts of the Assistant in an idol outfit dancing alone: waving, hands clasped at the chest, a full spin, a back view, a knee lift and more — Assistant solo, 12 cuts. Same outfit as the Assistant in the video.

4Video

Opening the video — 5 seconds of stick figure, 5 seconds of the Assistant

The video joins the two dances one after the other, not side by side (0–4.96 s stick figure, 4.96–10 s Assistant). Press "Side-by-side view" to load the same file in two screens and compare the same moments. Shift the offset of the right screen to find the moment when the two dances line up.

Original as is (10 s)

Cut sheet · stick figure (cut times are approximations dividing 5 s into 12; click to jump to that moment)

Cut sheet · Assistant

Speed

View

The sound is off at first. Use "Sound on" to hear the music. In side-by-side view, try moving the offset to around 0.9 s.

4.96 sWhere the scene changes (frame 119, Code 2)

−0.92 sOffset with the highest correlation of motion size: the Assistant moves first (Code 2, correlation 0.48)

MirroredThe stick figure points to the right side of the screen, the Assistant to the left (cut sheets 7–8 and 4–5)

Has musicAAC stereo. Besides the picture, there is sound too

5Match score

Examining the "motion match score 76.5%"

First we check what the number in the video is, then we measure a match score ourselves. Even for the same dance, the number changes a lot depending on how you measure.

What we checked (Code 2)	Result	Meaning
Does the number box change from frame to frame?	Max pixel difference from the first frame: 0.07	It stays at 76.5% for all 240 frames. It is not a value recalculated for each frame
Match-score boxes in the first and second scenes	Mean difference at the same time points: 0.16	The same overlay was played once over each scene. The swaying pink bars at the bottom are just decoration too
Calculation method stated	None	From the video alone you cannot tell what was compared with what, or with which score

Measure it yourself — same choreography, different timing (illustrative calculation)

On the left is a 5-second choreography (wave → point → arms out with a knee lift → both arms up); on the right is the same choreography followed late. The joint distance score aligns the pelvis, then averages exp(−distance²/2·0.1²) over the joints; the limb direction score is the average cosine similarity of the directions of 8 bones.

The Assistant’s expression, which changes with the score

Follower’s lag1.00 s

Results of Code 1

Case	Joint distance score	Limb direction score
Same dance, same time	100.0%	100.0%
Following 0.33 s late	90.8%	96.4%
Following 1.0 s late	77.6%	87.8%
Mirrored dance	82.5%	88.6%
1.0 s late + mirrored	78.7%	86.5%
The 1.0 s late dance shifted 15 frames earlier to align	100.0%	—
Aligned with dynamic time warping (DTW)	99.6%	—
Mirrored dance too: timing aligned only / mirroring undone as well	79.6% / 100.0%	—

How to read 76.5%

The same dance scores 77.6% if it is 1 second late. That is the example from Code 1. The two dances in the video are about 0.9 s apart and mirrored as well. 76.5% may be a number that measured these offsets as they are, but since the calculation method is not stated, we cannot check.
Different kinds of score give different numbers. The same 1-second lag gives 77.6% by joint distance and 87.8% by limb direction. Whenever you quote a match score, you must also say what was compared.
Aligning the timing changes things. If you find and correct the offset, or use dynamic time warping, the same dance gets close to 100%. "Did they follow the dance well?" and "Were they in the same pose at the same moment?" are different questions.

6Recap

The road the stick figure traveled — five missions on one page

A single stick figure in a CCTV alley became features, became material for generation, was folded and unfolded, was paired with words, and finally became the Assistant’s dance. Click a card to go to that part.

5-1 RecognitionCCTV littering

6 million on-screen numbers → 26 joint numbers → 4 features → verdict

60.7% → 99.7%

Wrist-speed rule → 4 features + learning

5-2 GenerationFrom recognizing to making

Flip the arrow and you learn a distribution, not an answer

44.2% → 0.0%

Bone-length error: coordinate dice → knob (latent variable) sampling

5-3 CompressionEncoder · decoder

Fold 1,196 numbers into 16 and unfold them again

99.8%

Motion held in 16 numbers

5-4 TextFrom words to motion

Contrastive learning puts words and motions on the same map

1.90 → 0.01

Pair-matching loss, 400 steps

5-5 FinalFrom "Dance!" to a dance video

Dress the pose map as the Assistant and make a video

76.5%

Match score shown in the video (calculation method not stated)

Sentences that tie the series together

Everything is numbers: CCTV frames, joints and sentences can only be computed once they are turned into numbers.
A good representation is half the battle: what you represent things with, such as 4 features (5-1), angles (5-2), 16 latent numbers (5-3) or a shared map (5-4), decides performance.
Recognizing and making use the same numbers in opposite directions: the stick figure that caught littering became the skeleton of a dance.

Questions that remain

Before you trust a number: 99.7% was a result on synthetic data, and 76.5% has no stated calculation method. A number should always come with "what, and how".
The human part: people are responsible for the final check of a littering verdict (5-1) and for consent to use real people’s motion and appearance (5-2·5-5).

Colab code — run this page yourself

Both code notebooks use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab via File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · How do you measure a motion match score?Linked to section 5 · numpy · CPU about 5 s

Builds a 5-second choreography and compares it, with two kinds of score, against versions followed late or mirrored left-right. Timing alignment and dynamic time warping (DTW) are included too. To use your own data, instead of dance_seq, put joint coordinates (frames, 13, 2) extracted from two videos by pose estimation into A and B.

Run result (run on 2026-10-05, numpy 2.4 · CPU). It uses no random numbers, so the output is the same whenever you run it.

Code 2 · Taking the video apart into numbers (OpenCV)Linked to sections 4·5 · numpy, OpenCV · CPU about 3 s · video file required

Measures the video’s resolution and length, the scene change, changes in the match-score number box, and the time offset between the two scenes. The video-analysis numbers in sections 4·5 come from here. Upload match_score_video.mp4, the same file included on this page, to Colab and use it.

Run result (run on 2026-10-05, numpy 2.4 · OpenCV 4.13 · CPU). With the same video, the output is the same whenever you run it.

AI prompts — using this code as a starting point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. When you press "Copy," the selected code is attached after the prompt. Just replace the parts in 〔〕 with your own situation.

Code to attach

The AI's answer is only a starting point. Always verify suggested papers, figures, and library functions against the original sources and official documentation. When using footage of real people, first check filming consent and personal data processing rules.

Glossary

The same dictionary as the yellow dotted underlines in the text. Click a section name to jump to where the term first appears.

Analogy collection — and where the analogies break

Analogies are a first step toward understanding, but they can also plant misconceptions. Analogies to human perception are marked by strength. Direct there is evidence that it actually does the same thing · Partial only some properties are similar · None no known counterpart

Concept	Everyday analogy	Human perception analogy	Where the analogy breaks
Pose-guided generation	Posing a mannequin, dressing it, and taking a photo	Partly A person seeing a motion and picturing a specific person doing it	A mannequin keeps the same clothes, but a generative model redraws the clothes and face every frame, so they can vary a little
Reference image	An actor’s profile photo	None	The model invents the back view and fingers that are not in the profile photo. The invented parts may differ from the real person
Pose map	Dance notation (a written record of choreography)	None	People read and interpret dance notation, but a pose map holds only joint positions. Facial expressions, fingers and rhythm are left out
Motion match score	A karaoke score	None	As with a karaoke score, when the calculation is hidden you cannot tell from the number alone whether it was good. What was compared with what, and how, comes first
Time alignment (DTW)	Re-editing two people’s off-beat dance videos so they match the beat	Partly People recognize it as "the same dance" even when the beat is a little off	DTW only stretches and squeezes time; it never changes the order. It cannot align a dance whose moves come in a different order

About the author and references

E-1 About the author · E-2 Academic references · E-3 Sources of figures and numbers

E-1 About the author

Ahn Sangsun (CEO, M-Robo Co., Ltd.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Co., Ltd., a spin-off venture of Maeil Business Newspaper, where he currently serves as CEO. M-Robo began as an in-house venture at a news company and is a media venture that has provided a robot reporter service (StockBot) to Kookmin Ilbo and carried out numerous projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. While serving as CEO, he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of "AI Crime and Self-Defense" and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a Ph.D. in Economics (Dongguk University, microeconomics and mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Lectures

Ahn Sangsun (2025). "Data Science for Business" lecture notes. Department of AI Convergence, Open Cyber University of Korea.
Ahn Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn Sangsun (2024). [K-Digital] Fintech Data Analysis Basics. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work because of a site redesign, please visit www.step.or.kr.

Papers and presentations

Ahn Sangsun (2025). A study on a household expenditure prediction model using the Korean Retirement and Income Study panel data: An AI model-based approach. Future Society, 16(2). Full text
Ahn Sangsun (2025). A study on a household expenditure prediction model using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn Sangsun (2025). A study on an algorithm for detecting suspected drug transactions based on SNS text: Focusing on pattern and context analysis. Future Society, 16(1). Full text
Ahn Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. Future Society, 15(3). Full text
Ahn Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation at the 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation at the 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
Ahn Sangsun (2024). Analysis and forecasting of virtual asset time-series data: Comparing time-series models and AI models, and suggesting improvements. Future Society, 15(1). Full text

E-2 Academic references

[R1] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33. arXiv:2006.11239 — section 3
[R2] Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., & Bo, L. (2024). Animate Anyone: Consistent and controllable image-to-video synthesis for character animation. CVPR 2024. arXiv:2311.17117 — sections 2·3
[R3] Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017). Realtime multi-person 2D pose estimation using part affinity fields. CVPR 2017. arXiv:1611.08050 — section 2 (the colored-bar format of pose maps)
[R4] Sakoe, H., & Chiba, S. (1978). Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1), 43–49. doi:10.1109/TASSP.1978.1163055 — section 5 (dynamic time warping)
[R5] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014). Microsoft COCO: Common objects in context. ECCV 2014. arXiv:1405.0312 — section 5 (source of OKS, the joint-distance-based score)

E-3 Where the pictures and numbers come from

The Assistant’s character sheet, the expression sheet, the two cut sheets, the Assistant solo 12 cuts and the dance video are materials made by the author
The video-analysis numbers in sections 4·5 were measured directly from the video with Code 2. The cut-sheet times are approximations dividing 5 s into 12
The match-score calculation in section 5 uses a choreography made for illustration; it does not recalculate the 76.5% in the video
The noise pictures in section 3 were made by mixing noise into a finished frame. They are not a recording of the actual generation process

Tips for educators

An operating guide for instructors, on-the-job trainers and tutors. Times are estimates, so adjust them to your actual class. This part also works as a wrap-up class for the series.

Audience	University students and working adults who have finished 5-1 to 5-4. Planners and practitioners who deal with generated videos and "AI scores" at work
Prerequisites	Pose estimation from 5-1, words → motion from 5-4, the basic idea of diffusion models (removing noise)
Learning goals	① Explain the six steps from prompt to video in order ② Say what the reference image and the pose map each determine ③ When looking at a number like a match score, ask "what was compared, and how?", and give an example of how a time offset changes the score
Estimated time	Reading 30–40 min · video demo 20 min · Colab practice 25 min (Code 1 12 min, Code 2 13 min)

Three ways to run it

Lecture, 50 min

Focus on the series recap

Intro 5 min (section 1) → section 2, six steps, 10 min → section 4, side-by-side video, 10 min → section 5, match-score experiment, 12 min → section 6, series map, 8 min → check questions 5 min.

On-the-job training, 90 min

Focus on verifying scores

Sections 4·5, 25 min → each team picks one "AI score in our work" and traces back what was compared and how, 25 min → measuring the video directly with Code 2, 20 min → discussion of labeling generated content and consent, based on the outfit difference in section 3, 20 min.

1:1 tutoring

Pace by level

Beginners: sections 2·4·6 only. Have intermediate learners compute score_dist by hand for just one frame and follow DTW’s table filling on a 3 × 3 grid.

Teaching points by section

Section	Question to pause and ask	Hands-on tip
2 In one line	"Which of these steps were computed directly in this series, and which come from the given video?"	Click the boxes one by one and read the last sentence of the explanation below
3 Drawing the Assistant	"The reference image shows a lab coat, so why is the video an idol outfit?"	Drag the step from 0 → 25 and compare with the solo 12 cuts below
4 Video	"At what second does the spin happen in each of the two dances?"	In side-by-side view, move the offset from 0 → 0.9 s
5 Match score	"It is the same dance, so why 77.6%? How do you make it 100%?"	At a lag of 1.0 s, press "Align timing"
6 Recap	"If you had to pick the single most important number from the five parts, which would it be?"	Click a card to go back to that part

Top 5 common misconceptions

Misconception	Correct explanation	Scene to show
Say "Dance!" and the AI understands dancing and makes the video	It is a flow of several steps handing numbers along. Words → a point on the map → joint coordinates → pose map → frames	Section 2
The match score shown in the video is a reliable measurement	This video’s 76.5% is the same value for all 240 frames, and its calculation method is not stated	Section 5, upper table
A low match score means the dance was not followed well	The same dance scores 77.6% if it is 1 second late. Align the timing and the left-right and it becomes 100%	Section 5, "Measure it yourself"
The reference image’s appearance comes out exactly as is	That is the principle, but parts not in the reference (such as the back view) are invented, and the outfit can differ, as in this video	Section 3, note below
With just a pose map anyone can be made to dance, so there is no problem	When making a video with a real person’s appearance, you need that person’s consent and a label marking it as generated	Section 6, "The human part"

Check questions (click to expand the answers)

Q1. How many frames does a 10-second video at 24 frames per second have? If the scene change is at frame 119, how many seconds is that?

24 × 10 = 240 frames. 119 ÷ 24 ≈ 4.96 s.

Q2. When a joint is 0.1 away, what is its score exp(−d²/2σ²)? (σ = 0.1)

exp(−0.01 / 0.02) = exp(−0.5) ≈ 0.607. When the distance is as large as σ, that joint scores about 61 points.

Q3. The same dance was followed 1 second late. What is the score when comparing frames as they are, and the score after shifting 15 frames earlier? (Code 1)

77.6% as is, 100.0% when shifted to align. The lag was being counted as "wrong".

Q4. How did we find out that the video’s 76.5% is not a value calculated for each frame?

The pixels of the number box stayed almost the same as the first frame for all 240 frames (max difference 0.07), and the match-score boxes of the two scenes were almost identical at the same time points (mean difference 0.16). The same overlay was played twice.

Q5. In pose-guided generation, what do the reference image and the pose map each determine? In the noise pictures of section 3, which steps are the job of which condition?

The reference image sets the appearance (face, hair, clothes); the pose map sets the pose. Early in denoising (big outlines) the pose map does most of the work; later on (face, accessories, patterns) the reference image does.

Colab assignments

Basic

In Code 1, change σ to 0.05, 0.1 and 0.2 and compare the joint distance scores for "Following 1.0 s late". Write two sentences on how σ affects the score and why σ should also be reported when you present a match score.

Challenge

Extract joint coordinates from both scenes of this video with MediaPipe (also record the frames where it fails on the Assistant scene), then compute the actual match score with score_dist and best_shift from Code 1. Report the scores before and after timing alignment and with the mirroring undone, compared with 76.5%.

Adjusting difficulty and going further

Beginners	The side-by-side view in section 4 and the single "Align timing" button in section 5 are enough to get the core of this part
Advanced	Have learners present the architecture diagram of the Animate Anyone paper matched to the table in section 3 (appearance encoder · pose guider · temporal layers)
Series review	Have learners explain in one line each what the five card numbers in section 6 measured (60.7% → 99.7%, 44.2% → 0.0%, 99.8%, 1.90 → 0.01, 76.5%)