Cats and dogs both have fur, eyes and ears. What does it take to separate two things whose low-level features are so similar? We rotate in 3D to see why a perceptron or SVM gets stuck with a single plane, and watch a hidden layer change the coordinates. Then we follow the whole process by which a CNN turns a photo into probabilities, along with how it learns, and try for ourselves a data augmentation method that pastes in a key patch and mixes the labels (Attentive CutMix).
57.7% → 99.2%Same points: linear SVM vs MLP with 3 hidden neurons (Code 1)
29,858Numbers a small CNN learns
7 × 7Grid of the last feature map (same as ResNet-50)
6/49Cat share of the label in Attentive CutMix
93.3 → 95.2%CutMix effect with 200 photos (Code 2)
2Code samples that run right away in Colab
1Problem
Cat vs Dog — Both Have Fur, Eyes and Ears
Both have fur, two eyes and upright ears. Low-level features alone can hardly tell them apart. What does a model need to stack up to separate the two?
An image made by the author. It shows the procedure of Attentive CutMix [R6].
In 4-2, cat vs human differed right from the low-level features: fur versus skin. Cat vs dog is different. Both have fur, so their edge maps look alike, and the difference lies in the shapes of parts such as muzzle length, nose size and face outline.
This episode looks at three things: why stacking layers separates what a single plane couldn’t (Section 2), how a CNN turns a photo into probabilities (Sections 3–4), and a data augmentation method that teaches the model to focus on the key parts, the Attentive CutMix in the figure (Sections 5–6).
Explained for Beginners
The figure shows a method that picks the patch of a cat photo the model responded to most strongly (near the eyes and ears), pastes it onto a dog photo, and mixes the label in proportion to the pasted area. Since 6 of the 49 cells are cat, the label becomes “cat 6/49, dog 43/49.” You’ll try it yourself in Section 6.
2Why Stack Layers
Classification Boundaries and Changing Coordinates — Why We Stack Layers
The perceptron and SVM from 4-1 draw a single plane. In raw pixel coordinates, a single plane can’t split the classes. The fix is to change the coordinates, and who decides how to do that is what separates a kernel SVM from a multilayer neural network.
1Model
A linear classifier draws a single plane in feature space (a line in 2D, a plane in 3D, a hyperplane in higher dimensions) and answers according to which side a point falls on. The single-layer perceptron and the linear SVM both have this shape, and both find weights w and a bias b. The only difference is the rule for choosing the plane. The perceptron stops once no points are misclassified, while the SVM picks the plane with the widest gap to the two groups (the margin). So finding weights is not what sets the two apart.
Linear SVM ≈ single-layer perceptronOne plane in the original coordinates
Kernel SVMChange coordinates with a human-chosen formula, then a plane
Multilayer perceptron · CNNLearn even how to change the coordinates, then a plane
Put simplyIt’s like separating black and white marbles mixed on a table using a single ruler. If the black marbles are clustered in the middle and the white marbles surround them, wherever you place the ruler, one group gets cut. Now bend the table so the middle sinks: the black marbles roll down, the white ones stay up, and a single horizontal board can separate them. A kernel SVM is a person deciding how to bend the table; a multilayer neural network learns how to bend it on its own.
2Formula
Linear classifier: ŷ = 1 if w · x + b > 0, otherwise 0
Linear SVM: the (w, b) that classifies every sample correctly while maximizing the margin 2 / ‖w‖The boundary has the same shape as the perceptron’s; only the rule for “which plane to choose” differs. In practice, a soft margin that allows a few mistakes is used.
Kernel SVM: w · φ(x) + b, φ is chosen by a person (e.g., φ(x₁, x₂) = (x₁, x₂, x₁² + x₂²))
Multilayer perceptron: h = σ(W₁x + b₁), ŷ = [ w₂ · h + b₂ > 0 ]The hidden layer h is the new coordinates, and W₁ and b₁ are set by training. The last line is a single-layer perceptron drawing a plane in the new coordinates h.
w, b
The orientation and position of the plane. Every method finds these
φ(x)
A function that changes the coordinates. In a kernel SVM, it is set by the kernel a person chooses
σ
A bent function such as tanh or ReLU (an activation function). Without it, stacking layers is the same as a single plane
h
The output of the hidden layer. New features created by training
Reading the Formula Piece by Piece
w · x + bThe perceptron’s weighted sum. Multiply each feature by its weight and add them up; if the result is greater than 0, answer with one class. Connecting the points where it equals 0 gives the boundary plane.
φ(x)Changing the coordinates. The goal is for a boundary that was curved in the original space to become a flat plane in the new space.
σ(W₁x + b₁)φ turned into a learnable formula. Changing W₁ changes how the coordinates are transformed, and training adjusts W₁ so that the final plane separates the classes well. The convolution layers of a CNN play this role too.
Try It with NumbersPoints (0.1, 0.2) and (0.7, 0.1): in the original coordinates one is in the middle and one is outside, and no single straight line can wrap around the middle Change to φ = (x₁, x₂, x₁² + x₂²) and the third coordinate becomes 0.05 and 0.50 → a single plane, “third coordinate = 0.3,” separates them
3Rotate in 3D · Splitting the Same Points Three Ways
Two groups of points: one in the middle and one surrounding it (made-up data for illustration). Switch methods with the buttons and drag the figure to rotate it. The hidden layer in ③ (3 neurons) was actually trained in your browser when this page opened.
Method
Accuracy with one plane: -
Axes: -
● Middle group ● Surrounding group Purple surface = boundary plane. After clicking the figure, you can also rotate it with the arrow keys.
4Compared on the Same Criteria
Method
Boundary shape
Who sets the coordinates (features)
On raw pixel photos
Introduced
Single-layer perceptron
A single plane
Not changed
Doesn’t work well
1958
Linear SVM
A single plane (maximum margin)
Not changed
Doesn’t work well (usually better than the perceptron)
1990s
Kernel SVM
A curved boundary in the original space
A person, via the kernel
Works on simple images (handwritten digits); limited on natural photos
1992~
Multilayer perceptron · CNN
A curved boundary in the original space
Learned from data
Works well (needs enough data)
1986 on; CNN leap in 2012
On handwritten digits (MNIST), a kernel SVM reached an error rate of around 1% using only pixels (the comparison in LeCun et al., 1998). The more complex the photos get, the harder it is for a person to pick the right kernel. Conversely, attaching a linear SVM to the feature vectors extracted by a trained CNN classifies well too (Razavian et al., 2014). This means what made the difference in performance was not the type of classifier but the features.
5Interpretation · Correcting Common Misconceptions
“The perceptron came along because SVMs didn’t work” gets the order backwards. The perceptron (1958) came first, and SVMs became mainstream in the 1990s–2000s, when neural networks with stacked layers were hard to train. In 2012, a deep CNN beat SVM-based methods by a wide margin at large-scale photo classification, and the tide turned again.
“The perceptron is accurate because it finds weights” is only half right. SVMs find weights too. A single-layer perceptron is a one-plane classifier just like a linear SVM, so it gets stuck in much the same way on raw pixel photos. What works well is the multilayer perceptron with hidden layers, and the CNN.
In short, the difference comes not from SVM versus perceptron, but from whether the coordinates (features) are set by a person or learned from data. A pixel rule picking meaningless background cells (4-4) happens because pixels are bad coordinates, and the filters in 4-2 and the CNN in this episode are devices that change those coordinates through training. Besides stacking layers, the CNN’s win also came from weight sharing in convolution, large amounts of labeled data and GPU computing.
✓Check Question
Q1. Which correctly describes the relationship between the single-layer perceptron and the linear SVM?
Q2. What is the most important difference between a kernel SVM and a multilayer perceptron?
Q3. What happens if you remove σ (the activation function) from the hidden layer?
Explained for Beginners
Each hidden neuron makes one new coordinate through “weighted sum → bent function (tanh, ReLU).” Training adjusts how these coordinates are made so that the final plane separates the classes well. In Code 1, on the same points, a linear SVM got 57.7% right, an SVM with the human-chosen x₁² + x₂² added got 98.8%, and a multilayer perceptron with 3 hidden neurons got 99.2%. The convolution layers of a CNN do this same “changing of coordinates,” tailored to photos.
3CNN Structure
A CNN at a Glance — Feature Maps That Shrink and Thicken
Repeat convolution → ReLU → pooling, and the photo gets smaller while the number of feature maps grows. At the end, flatten them and feed them into a perceptron layer to get a score for each class.
1Model
A convolutional neural network (CNN) is a model built by stacking the parts from 4-2 (convolution and pooling), and it is the photo-specific version of the multilayer neural network from Section 2 that “learns to change the coordinates, then draws a plane.” Repeat [convolution → ReLU → pooling] several times and the photo gets smaller and smaller, while the number of feature maps (channels) grows. Flatten the last feature map into a single row and feed it into a fully connected layer (many perceptrons), and you get one score per class; softmax then turns these scores into probabilities that add up to 1.
InputPixel matrix 56 × 56 × 3
MiddleFeature maps that shrink and thicken
OutputP(cat), P(dog)
Put simplyIt’s like a company’s reporting chain. Frontline staff (layer 1) report small facts like “there’s a line here,” team leads (layers 2–3) bundle those reports and pass up “there are eyes and ears here,” and executives (the fully connected layer) look only at the summary and decide “93% likely a cat.” The higher you go, the shorter the reports get and the more abstract their content becomes.
h = flatten(F(L)), z = W·h + bFeed h, the last feature map laid out in a single row, into a perceptron layer to get the class scores z.
pk = ezk / Σj ezj, L = −log pcorrect
ℓ
Layer number. F⁽ˡ⁾ is the feature map of layer ℓ
K⁽ˡ⁾
The bundle of filters in layer ℓ. With multiple channels, the filter is that much thicker (3 × 3 × number of channels)
h
The flattened features. For 7 × 7 × 64, that’s 3,136 numbers
z, p, L
Scores, probabilities, cross-entropy loss
Reading the Formula Piece by Piece
F⁽ˡ⁻¹⁾ ∗ K⁽ˡ⁾The previous layer’s feature map is the next layer’s input. Layer 2 filters look not at pixels but at the “edge maps” found by layer 1, and find corners and curves.
flattenLays a three-dimensional box of numbers out in a single row. The numbers stay the same; only their order changes.
ez / ΣezSoftmax. The larger the score, the larger the probability, and all the probabilities add up to 1.
−log pcorrectCross-entropy. If the probability given to the correct answer is 1, the penalty is 0; the smaller it is, the bigger the penalty. Training adjusts both the filters K and W so that this value goes down.
Try It with NumbersScores z = (cat 3.0, dog 0.5) → e³ = 20.09, e⁰·⁵ = 1.65, sum 21.74 p(cat) = 20.09 / 21.74 = 0.924, p(dog) = 1.65 / 21.74 = 0.076 If the correct answer is cat, L = −log 0.924 = 0.079 (a small penalty). If it had been dog, L = −log 0.076 = 2.58 (a big penalty)
3Diagram · A Small CNN at a Glance
Stage
What it does
Output size
Number of values
Numbers to learn (parameters)
Input
Pixel matrix
56 × 56 × 3
9,408
0
Conv1
16 filters of 3 × 3, edge padding
56 × 56 × 16
50,176
3·3·3·16 + 16 = 448
Pooling1
2 × 2 max pooling
28 × 28 × 16
12,544
0
Conv2 + pooling2
32 filters → half size
14 × 14 × 32
6,272
3·3·16·32 + 32 = 4,640
Conv3 + pooling3
64 filters → half size
7 × 7 × 64
3,136
3·3·32·64 + 64 = 18,496
Flatten
Into one row
3,136
3,136
0
Fully connected
2 scores
2
2
3,136·2 + 2 = 6,274
Softmax
Into probabilities
2
2
0 (total 29,858)
Keep in mind that the last feature map is 7 × 7. ResNet-50, which is widely used in practice, also ends with a 7 × 7 grid feature map (2,048 channels) when you feed it a 224 × 224 photo, and the Attentive CutMix in Section 6 uses exactly these 49 cells.
Explained for Beginners
Follow the numbers in the table. A 56 × 56 × 3 photo goes through convolution and pooling three times to become 7 × 7 × 64, and the 3,136 flattened numbers pass through a fully connected layer to become 2 scores. There are 29,858 numbers to learn in total. ResNet-50, widely used in practice, has about 25 million, but it shares the 7 × 7 grid for its last feature map [R4].
4Probabilities
From Photo to Probability — Softmax and Training
Softmax turns the final scores into probabilities, and the filters and weights are adjusted according to the gap from the correct answer (cross-entropy).
1Animated Diagram · From Photo to Probability
Pick a photo above. The layer 1 feature maps and the pooling results are computed on this page by applying filters to the actual photo. The responses of the later layers and the final probabilities are example values chosen to show the flow.
Photo
Input matrix
Layer 1: find edges
Pooling: shrink
Later layers: find parts
Scores → probabilities
2Softmax Calculator
Move the two scores produced by the fully connected layer. Assume the correct answer is cat.
3Interpretation · How Are the Filter Numbers Decided?
At first, a CNN’s filter numbers K and the fully connected layer’s W are all random. Even if you feed in a cat photo, the probabilities come out about half and half.
Once you give it the correct answer, it computes the loss L = −log pcorrect and, using the gradient descent from an earlier episode, adjusts every number a little in the direction that reduces the loss. This gradient is passed backward through the layers, from the output toward the input (backpropagation).
Repeat this process over tens of thousands of photos, and without anyone telling them to, the layer 1 filters take the shape of edge filters, and the filters in later layers come to respond to eyes or ears. Learning even “which features to look at” from data is the biggest difference from the old way, where people chose the features.
4Key Characteristics
Strengths
Learns everything from pixels to probabilities in one go (no feature design needed)
Shares filters, so it has far fewer parameters than a fully connected network of the same size
The deeper the layer, the more abstract the features it represents
Limitations
Needs many labeled photos for training
Can be confidently wrong on combinations it never learned (a chihuahua that looks like a muffin)
A high probability doesn’t mean the reason is right (it may get it right by looking only at the background)
Where it fits: Neocognitron (1980) → LeNet (1989, 1998) → AlexNet (2012) → VGG and GoogLeNet (2014) → ResNet (2016). Today’s vision transformers likewise start by cutting a photo into small patches and turning them into numbers.
✓Check Question
Q1. As data passes through a CNN, the feature maps generally…
Q2. If the scores z are (2.0, 2.0), what are the softmax probabilities?
Q3. What gets adjusted during training?
Explained for Beginners
Softmax turns scores into probabilities that add up to 1. Cross-entropy gives a bigger penalty the lower the probability given to the correct answer. Training means adjusting every filter number and weight a little so that this penalty goes down, and the gradient is passed backward through the layers from the output toward the input (backpropagation) [R3].
5Cat vs Dog
Cat vs Dog — The Shapes of Parts Decide
The low-level features (fur edges) are similar. Part features seen from layer 3 on, such as muzzle length, nose size and face outline, tell the two apart.
②Cat vs Dog Medium
This time, the low-level features are similar. Both have fur, so the edge strength maps are dense across the whole photo, and both have two eyes and upright ears. Layer 1–2 features alone can hardly tell them apart.
The difference lies in the shapes of parts. A dog’s muzzle sticks far forward, its nose is big, and its mouth may hang open with the tongue showing. A cat’s face is round, with a small nose and mouth. Features from layer 3 on that look at the “layout of eyes, nose and muzzle” tell the two apart.
That’s why techniques were developed to teach the model to focus on these key parts rather than the overall feel of the photo (fur color, background). Section 6 covers the Attentive CutMix from the Section 1 image.
Explained for Beginners
In the pair comparison, switch to the “Edge strength” view and you’ll see both photos have dense edge maps because of the fur. Also lower the resolution to 7 and see what disappears first. To make the model look at the parts of the face rather than easy clues like fur color or the background (a lawn), how you present the training data becomes important. That is the data augmentation in Section 6.
6Data Augmentation
Data Augmentation and Attentive CutMix — Paste a Patch, Mix the Label Too
When photos are few or the model leans on the wrong clues, change or mix the training photos to teach it “what to look at.”
1Types of Data Augmentation · Showing the Same Photo in Different Ways
Method
What it does
Label?
What it teaches
Flip · crop · rotate
Flip left–right, or crop part and enlarge it
Unchanged
Same object even if position or direction changes
Change brightness · color
Makes it look like a photo under different lighting
In Colab Code 2, when the same CNN was trained on only 20 handwritten digits per class (200 in total), test accuracy was 93.3% as is and 95.2% with CutMix (average over 3 seeds). It’s an example of mixing and pasting helping when photos are few. With different data or models, the size of the effect will differ.
②+Try Attentive CutMix · Pasting a Cat’s Key Patches onto a Dog
The method has four steps. (1) Feed a cat photo into an already trained CNN to get the last 7 × 7 feature map. (2) Pick the k cells out of 49 with the strongest response. (3) Paste the matching patches of the cat photo onto the same positions in a random dog photo. (4) Mix the label by area ratio too. With k = 6, it’s cat 6/49, dog 43/49.
Cat · weak-response cells darkenedRandom dog photoMixed photo
Label of the mixed photo
Note: The paper picks patches using the actual responses of a ResNet-50 trained on ImageNet. Here, instead of a trained model, we used “the average edge strength within a cell” as a stand-in for the response. So cells that simply have many edges, such as forehead stripes or background edges, may be picked besides the eyes and nose. Which cells a truly trained model picks may differ from this.
Why this helps
A cat from patches aloneIn the mixed photo, the only cat content is six eye and ear patches. To get “12% cat” right from these, the model has to look at part features, not fur color or background.
Mix the label tooInstead of insisting on “100% dog,” the model is made to answer 88%. This has a regularizing effect that curbs the model’s habit of being overconfident (overfitting).
The difference from random patchesThe original CutMix cuts and pastes a single rectangle from a random position. Only background might get pasted, creating a wrong label like “30% cat.” Attentive CutMix reduces this problem by picking the places with the strongest response.
Explained for Beginners
Data augmentation is a way to improve performance by changing only the training photos, without changing the model structure. It has a regularizing effect that reduces overfitting, that is, fitting too closely to the training photos alone. Remember that the hands-on exercise above used “edge strength within a cell” as a stand-in for the response instead of a trained ResNet-50.
A
Colab Code — Run This Page Yourself
Both code samples use only libraries that come preinstalled in Colab and built-in data. Click “Download notebook (.ipynb)” and open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.
Code 1 · Why Stack Layers — Watching a Hidden Layer Change the CoordinatesLinked to Sections 2 and 4 · numpy, scikit-learn, matplotlib · under 1 second on CPU
Compares a perceptron, a linear SVM, a human-chosen transform, an RBF kernel SVM and a multilayer perceptron with 3 hidden neurons (backpropagation written directly in numpy) on the two groups, middle and surrounding, and computes softmax and cross-entropy.
Output (run on 2026-10-04, numpy 2.4 · scikit-learn 1.8 · CPU). The random seed is fixed, so it gives the same result whenever you run it.
Code 2 · A Small CNN and CutMix — Does Mixing and Pasting Help When Photos Are Few?Linked to Sections 3 and 6 · numpy, tensorflow (Keras), scikit-learn · about 1 minute on CPU
Uses only 20 images per class of 8 × 8 handwritten digits for training, trains the same CNN as is and with CutMix, and compares test accuracy over 3 seeds. To switch to cat and dog photos, pick classes 3 (cat) and 5 (dog) from keras.datasets.cifar10 and change the Input shape to (32, 32, 3).
Output (run on 2026-10-04, TensorFlow 2.21 · Keras 3 · CPU). The numbers may change slightly depending on the seed, versions and hardware.
B
AI Prompts to Try — Use This Code as a Starting Point
These are prompts to paste into an AI assistant such as ChatGPT, Qwen, or Claude. Click “Copy” and the code you selected is attached after the prompt. Replace the text inside 〔 〕 with your own situation.
Code to attach
An AI’s answer is only a starting point. Always check any papers, figures, or library functions it suggests against the originals and the official documentation. If you used AI help for a competition entry or a paper, disclose that you did and to what extent, following the rules of the organizers or the journal.
C
Glossary
This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.
D
Analogies — and Where the Analogy Breaks Down
Analogies are a first step toward understanding, but they can also plant misunderstandings. Analogies to the human eye and brain are marked by strength. Direct There is evidence that it actually does the same thing · Partial Only some properties are similar · None No known counterpart
Concept
Everyday analogy
Human eye/brain analogy
Where the analogy breaks down
Hidden layer
Smoothing out crumpled paper and splitting it with a ruler
None
A person smooths out the paper, but a hidden layer learns “how to smooth it out.” And in reality, this happens in hundreds of dimensions, so you can’t see it
CNN layers
A reporting chain: frontline staff → team lead → executive
Partly In the visual pathway, too, as you go from the retina → primary visual cortex → higher areas, neurons respond to wider regions and more complex shapes
The brain has many top-down connections and lots of attention. A CNN computes in one direction only
Softmax
Turning vote counts into vote shares
None
A vote share is an actual election result, but a softmax probability is only the model’s “degree of confidence” and may not equal the chance of being right
Backpropagation
When the result is wrong, assigning blame backward, starting from the last step of the process
None
How much blame each part gets is computed exactly with derivatives. No person decides who was at fault
CutMix
A collage cut and pasted from two photos, with the answer split in proportion to the pasted area
None
If the pasted patch is only background, you get a wrong answer like “30% cat.” Attentive CutMix reduces this
E
About the Author and References
E-1 About the Author · E-2 Academic References · E-3 Figure and Data Sources
E-1 About the Author
Sangsun Ahn (CEO, M-Robo Inc.)
After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., an external venture of Maeil Business Newspaper, where he currently serves as CEO. M-Robo Inc. is a media venture that began as an in-house venture of a news company. It provided a robot reporter service (StockBot) to the Kukmin Ilbo and has carried out numerous projects for organizations including the Korea Internet & Security Agency, the Korea Press Foundation, and the Korea Radio Promotion Association. It now runs its own AI news service, Money-Robo. While serving as CEO, he won the Asia Open Data Hackathon and received the Minister of the Interior and Safety Award twice for contributions to data entrepreneurship, and he is the inventor and registrant of five patents in artificial intelligence and blockchain. He is the author of the book “AI Crime and Self-Defense”, and he has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a Ph.D. in economics (Dongguk University, mathematical microeconomics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile
Courses
Ahn, Sangsun (2025). “Data Science for Business” (course materials). Department of AI Convergence, Open Cyber University of Korea.
Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn, Sangsun (2024). [K-Digital] Fintech Data Analysis Basics. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation at the 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation at the 10th Panel Study of Workers’ Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers’ Compensation & Welfare Service.
Ahn, Sangsun (2024). Analysis and forecasting of virtual asset time-series data: Comparing time-series and AI models and suggesting improvements. “Future Society”, 15(1). Full text
E-2 Academic References
[R1] Rosenblatt, F. (1958). The perceptron. Psychological Review, 65(6), 386–408. — Section 2
[R2] Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. COLT 1992. — Section 2 (kernel SVM)
[R3] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. — Sections 2·4
[R4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016. — Sections 3·6
[R5] Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). CutMix: Regularization strategy to train strong classifiers with localizable features. ICCV 2019. — Section 6, Code 2
[R6] Walawalkar, D., Shen, Z., Liu, Z., & Savvides, M. (2020). Attentive CutMix: An enhanced data augmentation approach for deep learning based image classification. ICASSP 2020. — Sections 1·6
[R7] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. ICLR 2018. — Section 6
[R8] Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-the-shelf: An astounding baseline for recognition. CVPR Workshops 2014. — Section 2
[R9] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS 2012. — Section 2
E-3 Figure and Data Sources
The image in Section 1 was made by the author and illustrates the procedure from [R6]
The points in Section 2 are made-up data for illustration; the hidden layer in ③ is actually trained in your browser when the page opens
In the Section 4 animation, the layer-1 feature maps and pooling are real computations; the later-layer responses and probabilities are example values
The 7 × 7 response map in Section 6 uses per-cell edge strength as a stand-in for a trained model
F
TIP for Educators
A guide for instructors, on-the-job training (OJT) leads, and tutors. Times are estimates, so adjust them to your actual class.
Audience
College students and working adults who have finished 4-1 and 4-2. Practitioners who need to explain “why deep learning has to be deep”
Prerequisites
The perceptron and SVM from 4-1, and convolution from 4-2. What the exponential function e^z means
Learning Goals
① Give an example of data that stumps a linear classifier, and explain how a hidden layer changes the coordinates ② Compute the output size and parameter count of each layer in a CNN ③ Compute softmax and cross-entropy, and explain how CutMix mixes the labels
Estimated Time
Reading 45–55 min · Hands-on demo 25 min · Colab practice 30 min (Code 1: 10 min; Code 2: 1 min to run + 19 min to interpret)
Three Lesson Plans
50-min lecture
Concept-focused
Intro 5 min (Section 1) → Section 2 3D figure 15 min → Section 3 size table 10 min → Section 4 softmax calculator 10 min → Section 6 CutMix 5 min → Check Questions 5 min.
90-min on-the-job training
Work-focused
Discuss Section 2 recast as a “can’t be split by a plane” case from your own data, 20 min → Use the Section 3 table to compute the parameter count for your input size, 15 min → Pick the method that fits your photos from the Section 6 augmentation table, 20 min → Adapt Code 2 to your few-photos situation and run it, 35 min.
1:1 tutoring
Paced by level
Beginners: Sections 1, 2 and 4 only. Intermediate learners: have them recompute the parameter counts in the Section 3 table by hand, and match the three backpropagation lines in Code 1 to the formulas line by line.
Teaching Points by Section
Section
Questions to pause and ask
Interaction tips
1 Problem
“What do cats and dogs have in common?”
First ask where the 6/49 in the image comes from
2 Why stack layers
“Wherever you place the plane, why does one side get cut?”
Press ① → ② → ③ in order and rotate the figure
3 CNN structure
“How many learnable numbers does a pooling layer have?”
Point up and down the size and parameter columns of the table
4 Probability
“If the scores are equal, what are the probabilities?”
Set the two scores equal in the softmax calculator
5 Cat vs dog
“What’s wrong with telling them apart by fur color alone?”
Look at the edge strength and resolution 7 in the pair comparison
6 Data augmentation
“If the cat patch is 10 cells, what’s the label?”
Move the k slider from 1 → 15 and watch the label bars
Top 5 Common Misconceptions
Misconception
Correct explanation
Scene to show
Just stacking layers gives you a curved boundary
Without a bent function (activation), stacked layers are the same as a single plane
Section 2 Check Question Q3
SVMs can’t do nonlinear classification
A kernel SVM makes curved boundaries. But a person picks the kernel
② in Section 2 and the RBF result in Code 1
Softmax probability = chance of being right
It’s only the model’s confidence and can differ from the actual accuracy (the calibration problem)
Spread the scores far apart in the Section 4 calculator
CNNs do well because they have so many learnable numbers
They reuse the same filter, so they have far fewer than a fully connected network
The 29,858 in the Section 3 table
Data augmentation just makes more photos
It’s regularization that teaches the model what to look at. Some methods even mix the labels
The Section 6 augmentation table and Code 2
Check Questions (click to expand the explanations)
Q1. When the points are tangled into a center and a ring around it, why does a linear SVM get stuck?
Its boundary is a single plane (a straight line in 2D), so however you orient it, it cuts the ring group in two. In Code 1 it scored 57.7%.
Q2. How many parameters does a convolution layer have that processes a 16-channel feature map with 32 filters of size 3 × 3?
3 × 3 × 16 × 32 + 32 (biases) = 4,640.
Q3. For scores z = (2.0, 2.0), what are the softmax probabilities, and what is the cross-entropy when the correct answer is the first class?
Probabilities (0.5, 0.5). Loss −log 0.5 ≈ 0.693.
Q4. On a 7 × 7 grid, a cat patch covering 10 cells was pasted. What are the label proportions?
Cat 10/49 ≈ 0.204, dog 39/49 ≈ 0.796.
Q5. In Code 2, CutMix raised the test accuracy. Why was the training accuracy 100% for both?
With only 200 training photos, both models can get every training photo right. The difference shows up on test photos they have never seen (the degree of overfitting).
Colab Assignments
Basic
In Code 1, change the number of hidden neurons to 1, 2, 3 and 8, tabulate the accuracy, and write down from how many neurons the center and ring can be separated, and why.
Challenge
In Code 2, set PER_CLASS to 10, 20, 50 and 100, and report with 3 seeds how the effect of CutMix (the difference in test accuracy) changes with the number of photos.
Adjusting Difficulty and Learning More
Beginners
You can skip the table in Section 3 and the formulas in Section 4. Make sure to work through the 3D figure in Section 2 and the CutMix hands-on in Section 6
Advanced
Have them add both Mixup and CutMix to Code 2, compare them, and present how label-mixing augmentation changes the model’s confidence (the softmax maximum)
Learn more
3-3 Images and CNNs · 4-4 Chihuahua vs Muffin (high-level features and the limits of models)
