Deep Learning A to Z · 3-4

Speech — How Air Vibrations Become a Matrix of Numbers

Sound is a vibration of the air. Measure this vibration at regular intervals (sampling), write it down on a fixed set of levels (quantization), cut it into short pieces (windows), split out which pitches of vibration are mixed in (Fourier transform), and regroup them on a scale that, like the human ear, looks more finely at low sounds (mel) — and you get a matrix of numbers that a neural network can read. Every sound is synthesized directly on this page.

Written by Sangsun Ahn (CEO, M-Robo Inc.) · About the author

1949Shannon's paper that set out the sampling theorem [R1]
8,000 → 1,960numbers in 1 second of sound (8 kHz), once it becomes a mel matrix
6.02 dBgain in signal-to-noise ratio for each extra quantization bit
25 ms · 10 mswindow length and window step on this page
20 binsnumber of mel filters (this page)
2Code examples you can run right away in Colab
1Vibration

Sound is a vibration of the air

When a speaker cone moves back and forth, the air is squeezed and loosened again and again, and this spreads outward. How many times it vibrates per second is the pitch (frequency); how strongly it vibrates is the loudness (amplitude).

Sound
Volume
2Sampling

Sampling — how many times per second do we measure?

measurements ≥ 2 × highest frequencyShannon, 1949

Measurements per second

Explained for beginners

To capture a vibration properly, you must measure at least twice during each cycle. Measuring fs times per second can capture only up to fs/2 Hz (the Nyquist limit); faster vibrations show up as a misleadingly slow vibration (aliasing) [R1]. The human ear hears up to about 20,000 Hz, so music CDs measure 44,100 times per second. The telephone only needs to carry speech, so it uses 8,000. The mel spectrogram on this page also uses 8,000 (8 kHz). That is why it cannot capture sounds above 4,000 Hz. Real recording equipment filters out sounds above the limit before sampling.

3Quantization

Quantization — on a ruler with how many marks do we write the measured values?

A measured value is a real number whose decimals go on forever. To store it, you must round it to the nearest of a fixed set of levels. With b bits there are 2b levels. The difference created by rounding is quantization noise.

Bits
4Framing · window

Framing and windows — cut short, soften the edges

Speech changes from moment to moment. So we cut it into 25 ms pieces, analyze each piece separately, and repeat while moving 10 ms at a time. The multiplication that gently shrinks both ends of each piece is the "window."

Window shape

Why shrink the ends?

A plain cut makes the values drop off abruptly at both ends of the piece. The Fourier transform calculates as if the piece repeats endlessly. So the sudden jump at the cut leaks out into frequencies that were not there (leakage). The Hann window multiplies by a curve that rises smoothly from 0 to 1 and back (0.5 − 0.5·cos), bringing both ends to 0. The price is a slightly wider peak. 25 ms·10 ms are values commonly used in speech analysis, and they are changed depending on the task.

5Fourier transform

Fourier transform — splitting mixed vibrations by pitch

Any sound can be expressed as a sum of sine waves of different pitches. The Fourier transform takes a waveform and returns "how much of each Hz component it contains." It is like a prism splitting white light into a rainbow.

Sound
6Mel spectrogram

Mel spectrogram — a matrix of numbers grouped on the human ear's scale

mel = 2595 · log10(1 + Hz / 700)"Do-Re-Mi-Fa" 1 s → 98 frames × 20 bins

Frame to view

Explained for beginners

The human ear easily tells apart a small difference between low sounds (100 Hz and 200 Hz), but has trouble with the same difference between high sounds (3,000 Hz and 3,100 Hz). The mel scale divides low frequencies finely and high frequencies coarsely. Grouping the Fourier transform result (129 bins) with 20 triangular filters on this scale and taking the log (decibels) gives the mel spectrogram. The formula above is one widely used definition. Features based on the mel scale (MFCC) performed better than other features in speech recognition and were long used as a de facto standard [R2].

7How models read it

How models read it — like a picture, or in order

A mel spectrogram is a "frequency × time" matrix of numbers. Neural networks read it in two ways. The detailed architectures were covered in earlier parts.

Reading mode

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · How sound becomes a matrix of numbers — step by step with NumPyLinked to Sections 2–6 · numpy, matplotlib · CPU 1 s

Computes sampling, quantization, windows, the Fourier transform and the mel spectrogram without libraries. The numbers on the page (aliasing 3 Hz, leakage 7.95% → 0.05%, mel bin values) come out exactly. To use your own data, change SIGNAL and FS.

Run output (run on 2026-10-03, numpy 2.4 · matplotlib 3.10 · CPU). The calculation has no randomness, so it is the same whenever you run it.
Code 2 · Making the same mel spectrogram with librosaLinked to Section 6 · numpy, librosa, matplotlib · CPU a few seconds

Converts the same sound into a mel spectrogram with librosa, the library used in practice, and compares it with the hand-made result. To use a recording, follow the [To use your own data] line in the code (prepare your own file).

Run results (run on 2026-10-03, librosa 1.0 · numpy 2.4 · CPU). librosa cuts frames at the FFT length (256), so it has one frame fewer (97). The other numbers are almost the same (correlation 0.998).

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔 〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

ConceptEveryday analogyFruit fly / brain analogyWhere the analogy breaks down
SamplingPhotographing a runner once every secondPartial A fruit fly's compound eye also follows changes in brightness only at a fixed rateA photo captures an instant, but a sample is only a value at a fixed interval. Motion faster than the interval looks wrong (aliasing)
QuantizationMeasuring height with a ruler marked in 1 cmNoneA ruler's marks are even, but in real audio the marks are set by the number of bits
WindowBlurring the edges of a photo (vignetting)NoneA window is multiplied in not to look nice, but to reduce the fake frequencies created by the cut ends
Fourier transformA prism splitting light into a rainbowDirect The human cochlea responds to different pitches at different positions, splitting sound by pitchThe cochlea splits continuously over time and responds nonlinearly to changes in loudness. The Fourier transform computes one cut piece at a time
Mel scalePiano keys: the same spacing every time the pitch doublesPartial Pitch intervals as people perceive themThere are several mel formulas, and people hear differently. The formula on this page is just one widely used version
Mel spectrogramSheet music: time (horizontal) × pitch (vertical)NoneSheet music writes each note, but a spectrogram writes all the energy, including harmonics and noise, into each cell

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

  • Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
  • Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
  • Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
  • Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
  • Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
  • Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
  • Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
  • Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
  • Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

  1. [R1] Shannon, C. E. (1949). Communication in the presence of noise. Proceedings of the IRE, 37(1), 10–21. doi:10.1109/JRPROC.1949.232969 — Hero, Section 2 (sampling theorem)
  2. [R2] Davis, S. B., & Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4), 357–366. — Section 6 (mel-frequency cepstrum gave the best recognition performance among the features compared)

E-3 Figure sources

All sounds were synthesized on the page (no recordings).

  • The mel scale formula 2595·log₁₀(1 + f/700) and the 25 ms·10 ms window are conventional values widely used in speech processing. Some libraries use other definitions (such as Slaney's), so Code 2 is set to the same definition

Tips for Educators

A guide for instructors, on-the-job (OJT) trainers, and tutors. Times are estimates; adjust them to your actual class.

AudienceUniversity students and practitioners who want to work with speech and sound data (call center recordings, equipment noise, voice commands)
PrerequisitesSine curves and logarithms. If you know "a photo is a grid of numbers" from 3-3, Sections 6 and 7 are easy
Learning objectives① Explain with numbers what sampling and quantization lose ② Explain in order how sound becomes a matrix through windows, the Fourier transform and mel filters ③ Distinguish the two ways CNNs and sequence models read the mel matrix
Estimated timeReading 35–45 min · animation and sound demos 20 min · Colab practice 25 min (Code 1 15 min, Code 2 10 min)

Three lesson plans

Lecture, 50 min

Concept-focused

Intro 5 min (Section 1, listening) → Section 2 aliasing 10 min → Section 3 comparing sounds by bit depth 10 min → Sections 4·5 15 min → Section 6 matrix of numbers 5 min → Check questions 5 min.

Workplace OJT, 90 min

Focused on applying to work

Compare Sections 2·3 with your equipment's recording specs (sampling rate, bits) 15 min → Sections 4–6 25 min → put your own recording into Code 2 30 min → discussion "Is our data CNN-type or sequence-type?" 20 min.

1:1 tutoring

Pace by level

Beginners: Sections 1·2·5·6 only. Intermediate learners: have them compute one cell of mel_filterbank in Code 1 by hand and compare it with the filter triangle figure on the page.

Teaching points by section

SectionQuestions to pause and askHands-on tips
1 Vibration"How many times faster does 440 Hz vibrate than 220 Hz? How will it sound?"Play the two tones in turn (if there is no sound, check the device volume)
2 Sampling"Why does it look like 3 Hz when we measure 8 times per second?"Press 50 → 12 → 8 times in that order
3 Quantization"What sounds different between a 3-bit sound and a 16-bit sound?"Play it at 3 bits and at 16 bits
4 Framing · window"Why does a plain cut create frequencies that were not there?"Switch between Rectangular ↔ Hann and watch the skirt of the graph below
5 Fourier transform"It is the same 'Do', so why do a piano and a violin sound different?"Play the pure tone → instrument tone in order and compare the number of peaks
6 Mel spectrogram"How many numbers did 1 second of sound become?"Press the Do → Fa frames and watch the brightest cell move upward
7 How models read it"Which approach suits doorbell sound classification, and which suits dictation?"Switch between the two modes

Top 5 common misconceptions

MisconceptionCorrect explanationScene to show
Measuring more often is always betterTwice the highest frequency you want to capture is enough. More only increases storage12 times in Section 2 (5 Hz is captured)
Fewer bits make the sound quieterIt is the noise, not the volume, that grows. Each bit makes it about 6 dB cleanerSection 3
The Fourier transform analyzes the whole sound at onceIt is done separately for each short piece (frame). That is why we can see changes over timeSections 4·6
A spectrogram is a picture, so it can be treated just like a photoThe horizontal (time) and vertical (frequency) axes mean different things. Shifting vertically makes it a different soundSection 7
A mel spectrogram keeps the sound as it isIt is a reduced summary that discards phase and fine frequency information (8,000 → 1,960)Section 6 numeric summary

Check questions (click to expand the explanation)

Q1. What is the highest frequency that can be captured at a sampling rate of 8,000 Hz?

8,000 ÷ 2 = 4,000 Hz (the Nyquist limit).

Q2. If a 5 Hz vibration is measured 8 times per second, what frequency does it appear to be?

|5 − 8| = 3 Hz. It exceeds the limit (4 Hz), so aliasing occurs.

Q3. What is the theoretical SNR difference between 8 bits and 16 bits?

6.02 × (16 − 8) ≈ 48 dB. The theoretical values are 49.92 dB and 98.08 dB.

Q4. If 1 second of 8 kHz sound is cut with a 25 ms window (200 samples) at a 10 ms step (80 samples), how many frames are there?

(8,000 − 200) ÷ 80 + 1 = 98 (drop the decimals, then add 1).

Q5. How many numbers are in a matrix of 98 frames × 20 mel bins, and what fraction of the original sound's numbers is that?

1,960. 8,000 ÷ 1,960 ≈ 4.1, so about one quarter.

Colab assignments

Basic

In Code 1, change NMEL from 20 to 40 and FRAME from 200 to 400. Write three sentences on how the size and picture of the mel matrix change, connecting this to the trade-off between time resolution and frequency resolution.

Challenge

Run Code 2 with a file you recorded yourself of your voice or of equipment sounds (prepare your own file; mind personal information). Record the same sound twice, compare how similar the mel matrices are using the correlation coefficient, and write what causes the differences.

Adjusting difficulty and further learning

BeginnersSections 3·4 can be skipped without breaking the flow. Be sure to do the listening in Sections 1·5
AdvancedHave them investigate how window length, step and number of mel bins affect recognition performance. Check the papers they find in the original
Further learningSangsun Ahn, “Data Science for Business” lecture notes (data science in general) · STEP [K-Digital] AI Natural Language Processing (linked to 3-5 and 3-8) · STEP [K-Digital] Fundamentals of Fintech Data Analysis (linked to 3-6)
Materials for instructorsGrading criteria for the check questions and sample answers for the assignments are on the Answer key for educators page (3-4)

Well done!

You have read "3-4 Speech" to the end. Everyone in the lab will sum it up for you in a word each.

the Professor
the ProfessorYou went through the whole process of air vibrations becoming a matrix of numbers.
  • Sound can be captured only up to half the number of measurements per second
  • Cut with windows and Fourier-transformed, sound splits by pitch
  • The 8,000 numbers of 1 second of sound become a mel matrix of 1,960
the Assistant
the AssistantThe 3-bit sound hurts my ears!
the Fly Researcher
the Fly ResearcherPlease turn my wingbeat sound into a mel matrix too.