MISSION RECEIVEDAM 02:13 · CAM-07
Someone on the CCTV screen threw a can and disappeared. A person sees at a glance that "they dumped it," but a computer has to find the person among millions of numbers and read their movement. HQ's orders came in two parts: recognize the dumping motion, then create that motion in reverse. A single stick figure starts out catching a litterer, becomes motion data, gets folded and unfolded, is paired with words, and in the end turns into a video of the Assistant dancing to the single word "Dance!" Together with the lab crew, we will solve the five missions one by one.
This is the record from 3 minutes after the alarm went off.
Click "Start mission" to go to that part. If you mark a finished mission as "Done," this device will remember it.
Turning people into stick figures to catch them
Orders At 2 a.m., someone threw a can in an alley and disappeared. From the 6 million numbers on the screen, spot the "dumping motion" first.
Gear Object detection · Pose estimation · 4 motion features · Rules vs. learning · Live control room
Flipping the arrows of the feature table
Orders Using the features that recognized littering, now create a "throwing" motion. Find out why averaging two motions makes the arm stretch.
Gear Discriminative vs. generative · Motion dataset · The trap of the average motion · The Evaluator's scoring
Folding and unfolding motion
Orders If you fold 1,196 numbers down to just a few and unfold them again, does the motion survive? Walk between walking and throwing on the latent space map.
Gear Autoencoder · Bottleneck size · Latent space interpolation · Finding abnormal motions by reconstruction error
Placing words and gestures on the same map
Orders Pair the words "waving a hand" with a hand-waving motion. Watch how tangled lines find their partners as the model learns.
Gear Text encoder · Contrastive learning · Word–motion retrieval · Text-conditioned generation
How a prompt becomes the Assistant's dance
Orders The single word "Dance!" becomes a stick figure's dance, and then a video of the Assistant dancing. Uncover what the 76.5% motion match score on the screen really means.
Gear Prompt → motion · Pose-guided video generation · Motion match score · Video player
Turn a CCTV screen into numbers, toggle detection boxes and joints on and off, play stick-figure scenes, experiment with rules by moving thresholds, and even run a live control room. See for yourself how the numbers actually change.
Every mission has code you can copy, download, or open directly as a notebook. It uses only libraries built into Colab, and the actual run results are included.
Each section has an explanation for beginners, tooltips on terms with dotted underlines, and check questions with answer explanations.
Includes prompts that use the code as a starting point for hackathon plans, competition proposals, and research experiment designs.
Analogies such as "laying a wire doll over a photo" and "a referee's video review," together with notes on where each analogy breaks.
Plans for lecture-style classes, on-the-job training, and 1:1 tutoring, questions for each section, and five common misconceptions.
◆ How a photo becomes numbers — 5-1's "the screen is numbers" is covered in more detail in 4-2 Cat vs. Human.
◆ Encoder–decoder and attention — You can preview the autoencoder and encoder–decoder that 5-3 and 5-4 build on in 3-7 Sequence Models, and the backbone of the text encoder in 3-8 Attention & Transformers.