Deep Learning A to Z · 3-3

Images and CNNs — From a Grid of Numbers to Probabilities

To a computer, a photo is a grid packed with numbers. Both hand-crafted features and a neural network that lays the pixels out in a single line (MLP) are easily thrown off by this grid. We follow what the convolutional neural network (CNN), which slides one small filter across the grid, solved and what it still has not, computing each number ourselves.

Written by Sangsun Ahn (CEO, M-Robo Inc.) · About the author

1980Neocognitron published [R2]
1998LeNet and handwritten digit recognition [R3]
15.3%Top-5 error of the winning CNN at ImageNet 2012 (runner-up 26.2%) [R4]
9Weights in one 3×3 filter
2,371 : 527Parameter counts of this page's MLP and CNN
2Code examples you can run right away in Colab
1Number grid

A photo is a grid of numbers

If you zoom far into a picture on a screen, you see small squares (pixels). One pixel is one number for brightness — or, in color, three numbers for red, green and blue.

Picture
View

Explained for beginners

A number grid is usually stored as integers from 0 (black) to 255 (white). When feeding it to a neural network, it is often divided by 255 to get values from 0 to 1. The calculations on this page also use 0–1. One phone photo (e.g. 4000×3000 color) holds 36 million numbers. The fact that the number grid is this large is the starting point for the following sections.

2Classic ML

Classic machine learning — hand-crafted features only withstand the changes you expected

We need to tell three shapes (+, ×, ○) apart. A person chose two clues: "Is the center filled?" and "Are the horizontal/vertical strokes darker than the diagonal strokes?" The machine only learns the weights for these two numbers.

Test condition

So what did people do?

The features in this example were made simple on purpose. Real researchers spent years designing far more elaborate features, such as counting edge directions region by region. Even so, objects in photos vary endlessly in position, size, lighting and angle, and it was hard for people to foresee every change and build it into features. The tide turned at the 2012 ImageNet competition, which classifies photos into 1,000 categories: a CNN that learns even the features by itself reached a top-5 error of 15.3%, far ahead of the runner-up (26.2%) [R4].

3MLP limits

Four reasons an MLP gets stuck on photos

To feed a photo to the MLP (multilayer perceptron) from 3-2, the grid must be unrolled into one line. Let's look one by one at the problems that appear the moment it is unrolled.

Photo size (for ②)
4CNN's answer

CNN's four answers — slide a small filter

Local connections · weight sharing · pooling · stacking layersFukushima 1980 → LeCun 1998

MLP's problemCNN's answerHow far it solves it
① Loss of neighbor relationsLocal connections: one neuron looks only at a small neighborhood patch such as 3×3Looking at neighboring pixels together is built into the structure
② Weight explosionWeight sharing: the same filter is used at every positionNumber of filters × 9 is enough, regardless of photo size
③ Position changesWeight sharing + pooling: the same filter finds the pattern wherever it is, and pooling absorbs small shiftsRelatively less shaken. Not perfect: accuracy drops when the picture is shifted a lot (Section 8) [R6]
④ Pixel order ignoredStacking layers: group nearby pixels to build up wider and wider shapesIn return, it relies on the assumption that "nearby pixels are related." Shuffling the pixels lowers performance (Section 8)

Link to the fruit fly and the brain

In the cat visual cortex, Hubel and Wiesel distinguished "simple cells," which respond to a bar of a particular orientation in a narrow region, from "complex cells," which respond to the same orientation even if the position differs slightly [R1]. Fukushima's Neocognitron imitated these two cell types by alternately stacking "feature-finding layers" and "position-blurring layers" [R2]. Today's CNN convolution and pooling layers are their descendants. However, the Neocognitron learned by a method other than backpropagation; it was LeCun and colleagues who made a backpropagation-trained CNN succeed at handwriting recognition [R3].

5Convolution

Computing a convolution — multiply cell by cell, then add everything

Output = Σ (input patch × filter)6×6 input · 3×3 filter → 4×4 output

Filter

Explained for beginners

Place the filter window over one spot of the input. Multiply the 9 overlapping cells pairwise and add them all; that sum becomes one output cell. Move the window one cell at a time and repeat. On a 6×6 input, a 3×3 window fits in 4 positions across and 4 down, so the output is 4×4 (without padding, i.e. filling the border with zeros).

The vertical edge filter has +1 in the left column and −1 in the right column. So it gives a positive value where "the left is bright and the right is dark," a negative value in the opposite case, and close to 0 where it is uniform. A neural network learns such filter numbers by backpropagation instead of having people set them. Note also that the "convolution" of deep learning, unlike mathematical convolution, does not flip the filter (cross-correlation). Since the numbers are learned, the result is the same [R8].

6ReLU · pooling

ReLU and pooling — erase negatives, keep one of four

Pass the 4×4 output from Section 5 through ReLU (negative → 0), then keep only one number for each 2×2 window (maximum or average). We also see what changes when the picture is shifted by one cell.

Pooling
Filter
Input

What pooling does and its limits

Pooling cuts the numbers to a quarter, lightening the next layer's computation, and absorbs small shifts that happen inside a 2×2 window. A shift that crosses a window boundary, or a shift of 2 or more cells, changes the result directly. That is why "relatively less shaken" is more accurate than "CNNs are robust to position changes." In fact, research shows that even well-known large CNNs can change their answer when a picture is shifted by just a few pixels [R6].

7To probabilities

Flatten → probabilities — how feature maps become an answer

Turn the last feature maps into a single line of numbers, then pass them through a fully connected layer and softmax to produce "probability of +, probability of ×, probability of ○." We use the CNN from Section 8, trained in advance, as is.

Input
How to make one line

Explained for beginners

Classic CNNs such as LeNet joined the last maps together as they were (flattening) and fed them to a fully connected layer [R3]. Because this passes along even "which cell of the map the feature was in," the network becomes sensitive to position again. Global pooling, which keeps only one number per map (the maximum or the average), erases position information, so it is less shaken by shifts and has fewer parameters [R5]. The CNN on this page uses global max pooling.

Softmax turns the 3 scores into 3 probabilities that add up to 1. The larger the score, the larger the probability. A high probability is not always right: on appearances not seen during training, the model can give a high probability even to a wrong answer (3-1 Section 4).

8Dashboard

MLP vs CNN — a head-to-head on the same data

Both train on 120 pictures of 3 kinds of 12×12 shapes (40 each). All training pictures are centered (±0.5 cell) and mixed with noise, stray strokes, rotation and stroke-width changes. Each test uses 150 pictures never seen before.

Experiment

Experimental conditions (precisely)

  1. Models: MLP = 144 pixels → 16 hidden (ReLU) → 3. CNN = 6 3×3 convolutions (ReLU) → 2×2 max pooling → 8 3×3 convolutions (ReLU) → global max pooling → 3. Both are trained on the same training data with stochastic gradient descent (learning rate 0.02, one picture at a time) for 30 epochs. Random seed fixed.
  2. Shift position: only centered pictures are used for training; only the test pictures are shifted 1–3 cells in one direction (up, down, left or right). No training on shifted pictures (data augmentation) was done.
  3. Shuffle pixels: one rule for reordering the 144 cells is fixed and applied identically to both training and test pictures, and both models are retrained from scratch. This is not an experiment that feeds shuffled pictures to an already-trained model.
  4. Range of the numbers: this dashboard shows the result of a single seed. Colab Code 2 runs the same experiment in Keras averaged over 3 seeds; the numbers differ slightly but the trend is the same (Section A).

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · Computing convolution, ReLU and pooling by hand with NumPyLinked to Sections 5·6 · numpy, matplotlib · under 1 s on CPU

Computes convolution → ReLU → max and average pooling with the same 6×6 digit "7" and the same 4 filters as Sections 5·6. The output numbers match the page's numbers cell for cell. To use your own data, change only IMG at the part marked [To use your own data] in the code.

Run output (run on 2026-10-03, numpy 2.4 · matplotlib 3.10 · CPU). The calculation has no randomness, so it is the same whenever you run it.
Code 2 · MLP vs CNN in Keras — shifting position and shuffling pixelsLinked to Sections 3·8 · tensorflow (Keras), numpy, matplotlib · about 1–2 min on CPU

Repeats the experiment in a production library, with the same model structures and the same shape rules as Section 8. The optimizer is changed to Adam (40 epochs), and the run is repeated over 3 seeds and averaged. For pixel shuffling, the models are retrained from scratch on data shuffled with the same rule.

Run output (run on 2026-10-03, TensorFlow 2.21 · Keras 3.15 · CPU). The numbers differ from the browser dashboard (single seed, SGD), but the trend is the same. On a GPU or other versions the numbers may differ slightly.

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔 〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

ConceptEveryday analogyFruit fly / brain analogyWhere the analogy breaks down
PixelOne mosaic tilePartial One ommatidium of the fruit fly's compound eyeOmmatidia are arranged in a hexagonal pattern and are sensitive to changes in brightness. A photo's pixels are static values on a square grid
FilterA magnifying glass that reveals only a particular patternPartial Orientation-selective cells in the primary visual cortex [R1]A neuron's receptive field differs slightly from place to place. A CNN filter is exactly the same at every position
Weight sharingStamping the same stamp all over a sheet of paperPartial Neurons of the same type in the fruit fly visual system are repeated across the whole visual fieldEach neuron in the brain has its own connection strengths. They do not "share" numbers
PoolingMaking a reduced copy of a mapPartial Complex cells that respond even when the position differs slightly [R1]Whether complex cells compute exactly a maximum has not been confirmed
Stacking layersDrawing a picture as dots → lines → shapesPartial The visual pathway V1 → V2 → V4 → inferior temporal cortex (3-1 Section 5)The brain has many feedback connections; a CNN flows in one direction only
FlattenLaying the stones of a Go board out in a single lineNoneOnce laid out in a line, "who is whose neighbor" disappears (Section 3 ①)
SoftmaxTurning vote counts into vote sharesNoneA high vote share does not make an answer correct. On a picture it has never seen, the model can be confidently wrong
Shuffle pixelsMixing up puzzle pieces by a fixed ruleNonePeople can hardly recognize a shuffled picture, but the MLP learns it just as well. What "seeing like a human" means differs from model to model

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

  • Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
  • Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
  • Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
  • Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
  • Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
  • Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
  • Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
  • Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

  1. [R1] Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160(1), 106–154. doi:10.1113/jphysiol.1962.sp006837 — Section 4 (simple cells, complex cells)
  2. [R2] Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4), 193–202. doi:10.1007/BF00344251 — Hero, Section 4
  3. [R3] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. doi:10.1109/5.726791 — Hero, Sections 4·7
  4. [R4] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. doi:10.1145/3065386 (originally presented: NIPS 2012) — Hero, Section 2 (ILSVRC-2012 top-5 error 15.3%, runner-up 26.2%)
  5. [R5] Lin, M., Chen, Q., & Yan, S. (2014). Network in network. ICLR 2014. arXiv:1312.4400 — Section 7 (global average pooling)
  6. [R6] Azulay, A., & Weiss, Y. (2019). Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184), 1–25. jmlr.org — Sections 4·6·8 (CNNs are not fully invariant even to small shifts)
  7. [R7] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. doi:10.1038/nature14539 — Sections 4·5
  8. [R8] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. deeplearningbook.org — Sections 5·6 (Chapter 9, Convolutional Networks: cross-correlation, pooling)

E-3 Figure sources

The following materials served as references for structure and content.

  • CNN structure infographic (including convolution and pooling examples) — Sections 5·6
  • The color fruit fly in Section 1 — series-wide pixel art

Tips for Educators

A guide for instructors, on-the-job (OJT) trainers, and tutors. Times are estimates; adjust them to your actual class.

AudienceUniversity students and working professionals who have finished 3-2. Practitioners who handle image data (inspection images, medical images, document scans)
PrerequisitesThe hidden layer and backpropagation concepts from 3-2. Multiplication and addition alone are enough to follow the calculations in Sections 5·6
Learning objectives① Explain that a photo is a grid of numbers and give four reasons an MLP gets stuck ② Compute convolution, ReLU and pooling by hand ③ Use experimental results to explain that a CNN is "relatively less shaken by position changes, but not perfectly"
Estimated timeReading 40–50 min · animation demos 25 min · Colab practice 30 min (Code 1: 10 min, Code 2: 20 min)

Three lesson plans

Lecture, 50 min

Concept-focused

Intro 5 min (the numbers of the Section 1 color fruit fly) → Section 3, the four reasons, 10 min → Sections 5·6 hand calculation 15 min → Section 8 dashboard 15 min → Check questions 5 min.

Workplace OJT, 90 min

Focused on applying to work

Section 3 ② weight explosion, calculated with your company's image size, 10 min → Section 8 20 min (what if the on-site camera position changes?) → Code 2 hands-on 30 min → discussion "What augmentation does our data need?" 20 min → prompt cards 10 min.

1:1 tutoring

Pace by level

Beginners: Sections 1·3·5·8 only. Intermediate learners: have them compute one output cell of Section 5 by hand and compare it with the conv2d function in Code 1, then count the parameters of flattening vs. global pooling themselves in Section 7.

Teaching points by section

SectionQuestions to pause and askHands-on tips
1 Number grid"What numbers is the red of the fruit fly's eye in the three channels?"On the color fruit fly, press the Red, Green and Blue views in turn
2 Classic ML"Why do hand-crafted features collapse on shifted pictures?"Press Clean picture → Shifted 2 cells in order and compare the accuracy bars
3 MLP limits"How many positions apart in the line is the pixel directly below?"In ② Weight explosion, press 1000×1000 so learners are surprised by the size of the number
4 CNN's answer"Why does sharing a filter make the number of weights independent of photo size?"In Stacking layers, point out how the range one neuron sees widens 3 → 4 → 8
5 Convolution"Why is the output at this spot negative?"Pause and read the formula together using "Step one cell"
6 ReLU · pooling"After a one-cell shift, which changes less: before or after pooling?"Alternate between the original picture and the picture shifted by 1 cell
7 To probabilities"Which is less sensitive to position: flattening or global pooling?"Choose "+ shifted 2 cells," then switch between the two methods
8 Dashboard"Who wins on centered pictures, and who wins on shifted pictures?"Look at Shift position first, then switch to Shuffle pixels and see the result flip

Top 5 common misconceptions

MisconceptionCorrect explanationScene to show
A CNN recognizes things equally well even when their position changesIt is only relatively less shaken. Accuracy drops when the picture is shifted a lot, and even large CNNs can change their answer after a shift of a few pixelsSection 8 Shift position, Section 6
A CNN is always more accurate than an MLPWhen pictures are always in the same place, as on this page, the MLP can even score higher. A CNN's strengths are parameter efficiency and robustness to shiftsSection 8 accuracy at 0 cells
Filters are set by peopleThe filters in Section 5 are for explanation. A real CNN learns its filter numbers by backpropagationThe learned maps in Section 7
Pooling adds informationPooling reduces the numbers and discards information. In exchange, it absorbs small shifts and cuts computationSection 6, 4×4 → 2×2
A CNN is the same as the brain's visual cortexIt was inspired by simple and complex cells, but its learning method and connection structure differThe "Partial" labels in common Section D

Check questions (click to expand the explanation)

Q1. If a 224×224 color photo is fed to an MLP with 1,000 hidden neurons, how many weights does the first layer have? (excluding biases)

224 × 224 × 3 = 150,528 inputs × 1,000 = 150,528,000 (about 150 million). A convolution layer with 64 3×3 color filters has 3 × 3 × 3 × 64 = 1,728 (1,792 including biases).

Q2. If a 3×3 filter is applied to a 6×6 input with no padding and a stride of 1, what is the output size?

(6 − 3 + 1) × (6 − 3 + 1) = 4×4.

Q3. Show by calculation why the output is −3 when the vertical edge filter from Section 5 is placed at the top-left position.

The second row of the overlapping patch is [0, 3, 3], and each row of the filter is [1, 0, −1]. 0×1 + 3×0 + 3×(−1) = −3. The first and third rows are all 0, so the total is still −3.

Q4. What are the max pooling value and the average pooling value of the 2×2 window [6, 3, 0, 3]?

Max 6, average (6 + 3 + 0 + 3) / 4 = 3.

Q5. If the pixels are shuffled by the same rule and the models are trained from scratch, which drops more, the MLP or the CNN, and why?

The CNN. The CNN builds the assumption that "nearby pixels are related" into its structure (3×3 filters, pooling), but after shuffling, nearby cells become unrelated pixels. The MLP never used the positional relations between pixels in the first place, so shuffling leaves it almost unchanged. The key condition is that the models are trained from scratch on shuffled data, not that shuffled pictures are fed to an already-trained model.

Colab assignments

Basic

Add one diagonal edge filter to FILTERS in Code 1, find which cell of the 6×6 "7" responds most strongly, and explain why in two sentences.

Challenge

Mix pictures shifted by 1–2 cells into the training data of Code 2 (data augmentation). Summarize in a table how both models' accuracy at 0–3 cells changes, and compare "robustness gained from structure" with "robustness gained from data."

Adjusting difficulty and further learning

BeginnersSections 2 and 7 can be skipped without breaking the flow. For Section 8, read only the result bars together
AdvancedHave learners turn the receptive-field calculation in Section 4 Stacking layers (3 → 4 → 8) into a general formula. Discuss the limits in Section 6 together with the tables in the [R6] paper
Further learningSangsun Ahn, “Data Science for Business” lecture notes (data science in general) · STEP [K-Digital] AI Natural Language Processing (linked to 3-5 and 3-8) · STEP [K-Digital] Fundamentals of Fintech Data Analysis (linked to 3-6)
Materials for instructorsGrading criteria for the check questions and sample answers for the assignments are on the Answer key for educators page (3-3)

Well done!

You've read "3-3 Images and CNNs" all the way through. Everyone at the lab will sum it up with a word each.

the Professor
the ProfessorWe followed a photo all the way from a grid of numbers to probabilities.
  • A photo is a grid of numbers
  • One small filter finds the same pattern anywhere (weight sharing)
  • When the position shifts, the MLP is badly shaken while the CNN is relatively less shaken (with limits)
the Assistant
the AssistantEven when the 7 is shifted, the CNN still recognizes it.
the Fly Researcher
the Fly ResearcherMy compound eye is hundreds of little filters, too.