Deep Learning A to Z · 3-2

Building Blocks of Neural Networks — From the Perceptron to Backpropagation

A neural network is a combination of a few building blocks: neurons that weigh their inputs, activation functions that bend the signal, hidden layers that stack up, loss functions that measure how wrong the output is, and backpropagation, which hands out blame backward to fix the weights. You will move each block yourself, and at the end you will follow the full workflow of building a real model from these blocks, evaluating it, and deploying it.

Written by Sangsun Ahn (CEO, M-Robo Inc.) · About the author

1958Perceptron introduced [R1]
1986Backpropagation paper [R3]
12activation functions to plot yourself
0.25Maximum gradient of the sigmoid = the seed of vanishing gradients
3optimizers racing side by side
2Code examples you can run right away in Colab
1Perceptron

The perceptron — weigh the inputs and split with a single straight line

y = step(w₁x₁ + w₂x₂ + b)Rosenblatt, 1958

In the fruit fly

When a fruit smell and a vinegar smell arrive together, a neuron receives the two signals at different strengths and adds them up. If the sum crosses the firing threshold, it sends an "approach" signal. Synaptic strength is the weight, and the firing threshold is the bias.

In a neural network

Multiply each input by its weight, add them up, and add the bias to get z. If z is 0 or more, output 1; otherwise output 0. Seen on a plane, a single straight line w₁x₁ + w₂x₂ + b = 0 separates "food" from "not food." Each time it meets a misclassified point, the perceptron moves the line a little toward that point. That is the perceptron's learning rule.

Problem

Explained for beginners

The perceptron's learning rule is one line. Pick one misclassified point and apply w ← w + η·(target − prediction)·x. If the target was 1 but it said 0, pull the line toward that point; in the opposite case, push it away. For a linearly separable problem, this rule alone will eventually get every point right.

A problem like XOR, "food only when exactly one of the two is present," cannot be split by any straight line. After Minsky and Papert proved this limitation mathematically in 1969, neural network research stalled for a while. The solution is to add one more layer: the hidden layer. But first, let's look at functions that bend smoothly instead of stepping.

2Activation functions

Activation functions — how should the signal bend?

An activation function takes the weighted sum z and decides the neuron's output a. It needs a bend for stacking layers to mean anything, and the shape of its gradient (derivative) governs how fast learning goes.

Function f(z)Gradient f′(z)Current input z
Input z

Vanishing gradients — why ReLU became the standard

Backpropagation multiplies by the activation function's gradient each time it goes back through a layer. The sigmoid's gradient is at most 0.25. Going back through 10 layers shrinks the signal to at most 0.25¹⁰ ≈ 0.00000095 times its size. Almost no "fix this" signal reaches the early layers. This is the vanishing gradient problem.

ReLU's gradient is exactly 1 in the positive range, so the signal does not shrink no matter how many layers it goes back through, and the computation is just "compare with 0," so it is fast. On the other hand, its gradient is 0 in the negative range, so a neuron that has switched off may never come back. Variants such as Leaky ReLU, ELU, and Swish try to fill this gap. No function is always the best; you choose according to the problem and the model.

3Hidden layers

Hidden layers — stacking layers only means something with an activation function

Input 2 → hidden → hidden → output 1Same data, same structure, only the activation function differs

Neurons per hidden layer

linear ∘ linear = linear

If you stack layers without an activation function, W₂(W₁x) = (W₂W₁)x, which is exactly the same as a single weight matrix. For example, if W₁ = [[1, 2], [0, 1]] and W₂ = [[2, 0], [1, 1]], then W₂W₁ = [[2, 4], [1, 3]]. Passing through the two layers or through this one matrix gives the same result. Even with 100 layers, you can only draw one straight line.

When an activation function bends the signal at every layer, straight-line pieces overlap and curve, forming a complex boundary such as a spiral. If you reduce the number of hidden neurons to 4, you can also see that there are not enough pieces to bend, so the boundary cannot wrap all the way around the spiral.

4Representation learning

Representation learning — features become more abstract layer by layer

Dots (pixels) → directions → junctions → shapes. Each layer uses the previous layer's features as material to build features one step larger.

Input figure

One neuron in numbers

Weigh the 4 features from the previous layer (0.8, 0.1, 0.4, 0.0) with the weights (0.5, 0.3, 0.2, 0.1) and add a bias of 0.1.

z = 0.8×0.5 + 0.1×0.3 + 0.4×0.2 + 0.0×0.1 + 0.1
= 0.40 + 0.03 + 0.08 + 0.00 + 0.10 = 0.61
a = ReLU(0.61) = max(0, 0.61) = 0.61

Hand-designed features vs. self-learned features

The direction and junction detectors in the figure above are rules a person set in advance for the sake of explanation. Older machine learning really did have people design features this way. Deep learning learns the weights of these detectors from the data by itself. That is why it also finds features people did not think of.

What features actually form at each layer for images is covered in 3-3 (Images and CNNs).

5Loss functions

Loss functions (cost functions) — how wrong, in a single number

Learning means reducing the loss. What you choose as the loss changes what the model "fears."

Mean squared error (MSE)

(target − prediction)². The default for regression problems, where you predict a number. Used to predict continuous values such as house prices, household spending, or temperature. The backpropagation dashboard below uses this loss.

Cross-entropy

−log(probability given to the correct answer). The default for classification problems. Being confidently wrong makes the penalty grow without limit. It pairs with a sigmoid or softmax at the output.

Target
Model's predicted probability p

Why cross-entropy for classification?

With a sigmoid at the output, the MSE gradient becomes 2(p − t)·p(1 − p). If the target is 1 and the model is confidently wrong with p = 0.01, p(1 − p) is close to 0, so the gradient is almost 0 too. It is badly wrong, yet the "fix this" signal is weak. With cross-entropy, the sigmoid's derivative cancels out neatly and the gradient is simply p − t, that is, −0.99. The more wrong it is, the harder it corrects.

6Backpropagation

Gradient descent and backpropagation — guess, measure, split the blame backward, and fix

Input 2 → hidden 3 → output 1Target 1.00 · loss = (target − output)² · learning rate 0.1

01 Forward pass

First, make a guess

The input is computed along the connections all the way to the output. The first output is 0.41.

02 Loss

Measure how wrong it is

(1.00 − 0.41)² ≈ 0.35. This is the MSE from section 5 above.

03 Backpropagation

Split up the blame

Going backward from the output, compute each weight's share of the blame (its gradient).

04 Update

Fix a little at a time

New weight = old weight − learning rate × gradient. This is gradient descent.

01 Forward pass→ 02 Loss→ 03 Backpropagation→ 04 Update↺ again

Click a weight cell to highlight that connection in the figure, and "Follow the numbers" below shows that weight's calculation.

Speed
Learning rate η
Connections to fix

Follow the numbers — one round for a single weight

Backpropagation is "passing on blame by multiplication"

The error amount at the output, −2 × (target − output), is determined first. The blame for a hidden→output weight is this multiplied by the signal that came in through that weight (the hidden value). An input→hidden weight goes back one more step: multiply by the output weight v to send the blame to the hidden neuron, multiply by h(1 − h) for how sensitive the hidden neuron was, and multiply by the incoming input x. This "multiply and send backward" is the chain rule. The vanishing gradients from section 2 also arise from exactly this multiplication.

Clicking "Output side only (fruit fly style)" freezes the input→hidden weights, and a teaching signal coming down from above (likened to dopamine in the fruit fly) fixes only the output-side connections. This is a heavily simplified analogy of the learning structure in the fruit fly's mushroom body, and whether the dopamine signal is computed like a gradient is still under research.

7Optimizers

Optimizers — three ways of walking down the valley

If you draw the loss as terrain, learning means walking down to find the lowest point. Even looking at the same gradient, each optimizer moves its steps differently.

SGD: one step as big as the gradientMomentum: adds the inertia it has built upAdam: adjusts the step size per direction
Learning rate η
Mini-batch noise
SGD: w ← w − η·g
Momentum: v ← 0.9·v − η·g,  w ← w + v
Adam: m ← 0.9·m + 0.1·g,  s ← 0.999·s + 0.001·g²,  w ← w − η·m̂ / (√ŝ + ε)  (m̂ and ŝ are bias-corrected for the initial values)

Explained for beginners

This valley is long horizontally and steep vertically. SGD moves only as much as the gradient, so in the steep vertical direction it bounces up and down with big steps (zigzag), and with a small learning rate it crawls along the gentle horizontal direction (check this with "0.1 Small"). Momentum remembers the speed it has built up, so it accelerates in the consistent direction (horizontal) while the zigzags cancel each other out. Adam measures the gradient size separately for each direction and adjusts the step size, so it descends horizontally and vertically at a similar pace.

"Mini-batch noise" is the jitter that comes from computing the gradient on only part of the data, as in real training. Set the learning rate to "Large" and SGD bounces over the valley walls. No optimizer always wins. Adam is widely used because it is a safe default for most problems, but there is also research showing that well-tuned SGD generalized better to unseen data on several deep learning tasks [R12].

8Overfitting

Overfitting and regularization — memorizing vs. understanding

14 training points · 80 validation pointsInput 1 → hidden 48 (tanh) → output 1, Adam

Dropout
Weight decay (L2)
Early stopping

Three remedies

  1. Early stopping: Use the model from the point where validation loss was lowest. It is the simplest and is used almost always.
  2. Dropout: During each training step, randomly switch off some hidden neurons. This stops the network from memorizing points by relying on a few specific neurons. At test time, all neurons are switched on.
  3. Weight decay (L2): Add a penalty to the loss that grows as the weights grow. This keeps the curve from becoming too wiggly.

The surest remedy is to collect more data. With little data and a big model, memorizing is easy. The numbers in this figure come from an illustrative experiment computed directly on this page, and the random seed is fixed so that it gives the same result every time it runs.

9Full workflow

The full model-building workflow — from data to deployment

All the blocks we have seen so far fit into a single step: training. In real work, what comes before and after is longer. The example is a "find the defects" model for a factory inspection station.

① Collect→ ② Split→ ③ Preprocess→ ④ Train→ ⑤ Evaluate→ ⑥ Save · deploy→ ⑦ Monitor

⑤ Evaluate — accuracy alone will fool you

Of 200 inspected products, 20 are actually defective. The model outputs a "probability of being defective" for each product. Try changing the cutoff above which a product is judged defective (the threshold).

Defect threshold

Common mistakes at each stage

  1. Split: Use the test data only once, at the very end. Choose settings with the validation data, and measure final performance with the test data. For data with a time order, do not shuffle randomly; train on the past and evaluate on the future (more in 3-6).
  2. Preprocess: Compute scaling references (standardization) such as the mean and standard deviation from the training data only. Computing them on all the data lets test information leak in (data leakage).
  3. Evaluate: If only 10% are defective, just saying "all normal" gives 90% accuracy. Look at the side you must not miss (recall) together with the cost of false alarms (precision).
  4. Deploy · monitor: When the factory lighting changes or new parts come in, the data changes and performance drops. Keep measuring performance after deployment, and retrain when needed.

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · A small neural network built by hand in NumPyLinked to sections 3·5·6·8 · numpy, matplotlib · about 10 s on CPU

The four steps forward pass → loss → backpropagation → update, written out as-is without libraries. It trains the same spiral data with "no activation function" and with "tanh hidden layers" and compares them. To use your own data, change only X and y in the part of the code marked [To use your own data].

Output (run on 2026-10-03, numpy 2.4 · CPU). The random seed is fixed, so you get the same result.
Code 2 · The same neural network in Keras + preventing overfittingLinked to sections 2·3·7·8·9 · tensorflow (Keras) · about 1–2 min on CPU

This is how it is done in practice. It compares three models: no activation function / ReLU / ReLU + dropout + early stopping, and saves the last model to a file (step ⑥ Save in section 9).

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔 〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

ConceptEveryday analogyFruit fly / brain analogyWhere the analogy breaks down
WeightThe different trust you give each opinionPartial Synaptic strengthReal synapses change strength with time and body state, and their signals are not continuous numbers but electrical pulses (spikes)
Bias / thresholdThe baseline for making a decisionPartial A neuron's firing thresholdA neuron's threshold is not fixed; it keeps changing with fatigue and adaptation
Activation functionA volume knob that has a limit no matter how far you turn itPartial Firing-rate curve as a function of stimulus strengthA neural network outputs a single number, but a neuron emits pulses over time
Hidden layerA middle manager who gathers and organizes reportsPartial Kenyon cells of the mushroom bodyMost input-side connections in the mushroom body do not change through learning. A hidden layer in deep learning learns all of its connections
Loss functionA test scoring sheetPartial Outcomes such as sugar (reward) and electric shock (punishment)The fruit fly does not compute "how wrong it was" as a single number
BackpropagationTracing responsibility backward after an accidentNone Dopamine signals are not computed backward through layersWhether the brain does something similar to backpropagation by other means is still under research
Learning rateHow much you correct after one piece of feedbackPartial The amount a synapse changes from one experienceHow much the brain changes depends on the situation, attention, and repetition
OptimizerHow a hiker walks down a foggy mountainNoneA real loss landscape has millions of dimensions, so you cannot see it at a glance like in the figure
OverfittingA student who memorized only the answers to past exam questionsNonePeople memorize too, but the way people generalize differs from neural networks
DropoutA team that keeps working even when a few members are absentNone Some interpret it as resembling occasional failures of synaptic transmission, but the evidence is weakAt test time all neurons are switched on. Team members are not always absent

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

  • Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
  • Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
  • Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
  • Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
  • Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
  • Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
  • Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
  • Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

  1. [R1] Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408. doi:10.1037/h0042519 — Section 1
  2. [R2] Minsky, M. L., & Papert, S. (1969). Perceptrons. MIT Press. — Section 1 (XOR limitation)
  3. [R3] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. doi:10.1038/323533a0 — Sections 4·6
  4. [R4] Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. Proceedings of ICML 2010, 807–814. — Section 2 (ReLU)
  5. [R5] Maas, A. L., Hannun, A. Y., & Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic models. ICML 2013 Workshop on Deep Learning for Audio, Speech and Language Processing. — Section 2 (Leaky ReLU)
  6. [R6] He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. Proceedings of ICCV 2015, 1026–1034. doi:10.1109/ICCV.2015.123 — Section 2 (PReLU)
  7. [R7] Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2016). Fast and accurate deep network learning by exponential linear units (ELUs). ICLR 2016. arXiv:1511.07289 — Section 2 (ELU)
  8. [R8] Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017). Self-normalizing neural networks. Advances in Neural Information Processing Systems 30. arXiv:1706.02515 — Section 2 (SELU)
  9. [R9] Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Searching for activation functions. arXiv:1710.05941 — Section 2 (Swish)
  10. [R10] Misra, D. (2020). Mish: A self regularized non-monotonic activation function. BMVC 2020. arXiv:1908.08681 — Section 2 (Mish)
  11. [R11] Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. ICLR 2015. arXiv:1412.6980 — Section 7
  12. [R12] Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., & Recht, B. (2017). The marginal value of adaptive gradient methods in machine learning. Advances in Neural Information Processing Systems 30. arXiv:1705.08292 — Section 7 (generalization comparison of Adam and SGD)
  13. [R13] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56), 1929–1958. jmlr.org — Section 8
  14. [R14] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. doi:10.1038/nature14539 — Sections 3·4
  15. [R15] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. deeplearningbook.org — Sections 5·7·8·9 (loss functions, optimization, regularization, early stopping)

E-3 Figure sources

The following materials served as references for structure and content.

  • DataCraft Hub, "Activation Functions — Adding Nonlinearity to Neural Networks" (Series 01 · 05) infographic — Sections 2·3
  • DataCraft Hub, "Representation Learning — How Neural Networks Learn Useful Representations" (Series 01 · 06) infographic — Section 4
  • mltut, "Activation Functions" infographic — Section 2. Hard Sigmoid follows the definition in that figure (0.2x + 0.5)
  • mltut, "A Quick Guide to Neural Networks" infographic — Sections 1·3
  • "How AI Learns? From Data to Intelligence" infographic (creator not credited) — Section 9
  • "How a neural network actually learns" educational animation (21 s, creator not credited) — starting values in Section 6

Tips for Educators

A guide for instructors, on-the-job (OJT) trainers, and tutors. Times are estimates; adjust them to your actual class.

AudienceUniversity students and working professionals learning neural networks for the first time. With basic Python, go as far as the Colab practice; without it, as far as the animations in the main text
PrerequisitesMultiplication and addition, and reading function graphs. For derivatives, "gradient = how much it changes when you change it a little" is enough
Learning objectives① Explain what the perceptron, activation functions, and hidden layers each do ② Follow forward pass → loss → backpropagation → update in numbers ③ Recognize overfitting and respond with validation data, early stopping, and dropout
Estimated timeReading 40–50 min · animation demos 25 min · Colab practice 30 min (Code 1: 10 min, Code 2: 20 min)

Three lesson plans

Lecture, 50 min

Concept-focused

Intro 5 min (demo the XOR failure in section 1) → sections 2·3 demos 15 min → sections 5·6 dashboard 15 min → section 8 overfitting 10 min → check questions 5 min.

Workplace OJT, 90 min

Focused on applying to work

Section 6 dashboard 15 min → sections 8·9 25 min (discuss the accuracy trap with work examples) → Code 2 practice 30 min → "apply it to our team's data" with the prompt cards 20 min.

1:1 tutoring

Pace by level

Beginners do only sections 1·2·3·6. Intermediate learners calculate the "Follow the numbers" in section 6 once by hand, then compare it with the backpropagation part of Code 1.

Teaching points by section

SectionQuestions to pause and askHands-on tips
1 Perceptron"With XOR, why do misclassified points remain no matter how you move the line?"Train on the easy problem first to show success, then switch to XOR
2 Activation functions"Where is the sigmoid's gradient at its maximum, and what is that value?"Stop on the sigmoid, then move the slider to 0 to confirm 0.25
3 Hidden layers"We stacked two layers, so why is the left side still a straight line?"Reduce the number of neurons to 4 to show the right side failing too
4 Representation learning"Why is □ judged by its 4 corners? What would a person look at?"Alternate between T and L and point out that only the junction layer changes
5 Loss functions"When the model is confidently wrong, which loss pushes harder to fix it?"Set p to 0.01 and compare the two gradient values
6 Backpropagation"Which has the larger gradient: a weight close to the output or one far from it?"Run one round slowly with "Next step," then switch between weight cells and read the formulas
7 Optimizers"Why does SGD zigzag?"Change the learning rate to "Large" to show SGD bouncing out
8 Overfitting"If the training loss keeps going down, is it a good model?"Turn off all regularization and run to the end, then turn each one on in turn and compare
9 Full workflow"In defect inspection, which costs more: a miss or a false alarm?"Move the threshold to 0.2 and 0.8 and watch precision and recall move in opposite directions

Top 5 common misconceptions

MisconceptionCorrect explanationScene to show
Stacking more layers always makes it smarterWithout activation functions, any number of layers equals one straight line. Even with them, it overfits when data is scarceSection 3 left figure, section 8
Neural networks are an exact copy of the brainThey borrowed only the skeleton: layers, connection strengths, and thresholds. Backpropagation has not been confirmed in the brainThe "None" marks in common section D
Low training loss means a good modelPerformance on unseen data (validation and test) is the standardSection 8 right graph
High accuracy means a good modelOn imbalanced data, even "all normal" gets high accuracy. Look at precision and recall togetherSection 9, threshold 0.95
Adam is always the bestIt is only a safe default. Depending on the problem, SGD + momentum can be betterSection 7, learning rate "Small"

Check questions (click to expand the explanation)

Q1. If you stack 10 hidden layers in a neural network without activation functions, what kind of boundary can it draw?

Only a single straight line (a plane in higher dimensions). Combining linear transformations many times still gives a single linear transformation (W₂W₁x = Wx). The left figure in section 3 is an example.

Q2. Why does using ReLU instead of the sigmoid reduce vanishing gradients?

The sigmoid's gradient is at most 0.25, so it shrinks quickly as it is multiplied at each layer. ReLU's gradient is 1 in the positive range, so it does not shrink going backward. However, it is 0 in the negative range, so dead neurons can appear.

Q3. When the target is 1 and the prediction is 0.41, what are the MSE loss and ∂L/∂y?

L = (1 − 0.41)² = 0.3481 ≈ 0.35. ∂L/∂y = −2 × (1 − 0.41) = −1.18. These match the first-round values in the section 6 dashboard.

Q4. Training loss keeps decreasing, but validation loss starts rising at some point. What is happening, and what should you do?

It is overfitting. Use the model from the point where validation loss was lowest (early stopping), turn on dropout or weight decay, or add more data.

Q5. In an inspection with a 10% defect rate, what are the accuracy and recall of a model that answers "all normal"?

Accuracy 90%, recall 0%. It found no defects at all. That is why you must not look at accuracy alone on imbalanced data.

Colab assignments

Basic

In Code 1, change tanh to ReLU (two lines: the activation function and its derivative). Change the number of hidden neurons to 4, 16, and 64, put the validation accuracy in a table, and write three sentences on the best setting and why.

Challenge

Replace the data in Code 2 with your own work data (2 or more features, labels 0/1). Split it into training/validation/test at 6:2:2 and report accuracy, precision, and recall on the test data. Judge, with reasons, whether a threshold of 0.5 fits your work.

Adjusting difficulty and further learning

BeginnersSections 4·7 can be skipped without breaking the flow. In section 6, go through just one round together with the "Next step" button
AdvancedHave learners calculate the "Follow the numbers" in section 6 by hand, then match it one by one with the d1·d2·d3 lines in Code 1. Discuss what would happen if learning rate decay (a schedule) were added to section 7
Further learningSangsun Ahn, “Data Science for Business” lecture notes (data science in general) · STEP [K-Digital] AI Natural Language Processing (linked to 3-5 and 3-8) · STEP [K-Digital] Fundamentals of Fintech Data Analysis (linked to 3-6)
Materials for instructorsGrading criteria for the check questions and sample answers for the assignments are on the Answer key for educators page (3-2)

Well done!

You've read 「3-2 Building Blocks of Neural Networks」 to the end. Everyone at the lab has a few words to sum it up.

the Professor
ProfessorWe assembled the building blocks of a neural network one by one.
  • A neuron is a weighted sum wrapped in an activation function
  • Backpropagation goes back through the error and fixes the weights little by little
  • Check for overfitting with validation data
Assistant
AssistantThe formulas on the blackboard don't scare me anymore.
Fly Researcher
Fly ResearcherOne step at a time along the gradient. Noted.