3-5 Text — How Characters Become Number Vectors

1Unicode

Unicode — a number for every character, and bytes

Hangul syllable code = 0xAC00 + (initial × 21 + medial) × 28 + finalUnicode Standard §3.12

Example sentence

2Tokenization

Tokenization — neither characters nor words, but "frequent pieces"

Splitting by character gives too many tokens; splitting by word cannot handle words never seen before. BPE finds the middle ground by repeatedly merging the two pieces that most often appear next to each other in the corpus.

Number of merges

Sentence

3Token IDs

Token IDs — which slot in the vocabulary?

After 60 BPE merges, collecting the tokens that appear in the corpus gives a vocabulary. Each token gets the number of its slot in the vocabulary. A token that is not in the vocabulary gets the "unknown" ID.

Sentence

4Embedding

Embedding — turning IDs into "coordinates that carry meaning"

IDs 54 and 55 are neighboring numbers, but their meanings have nothing to do with each other. So each ID gets a vector with several dimensions. The goal is for words used in similar ways to get nearby coordinates. Here we built an 8-dimensional vector for each of the 55 words from "how often two words appear together within ±2 positions in the same sentence".

Word

Explained for beginners

The idea that "words that appear with similar words have similar meanings" is old. A representative early method is latent semantic analysis (LSA), which reduces a word–document occurrence table with singular value decomposition to find hidden semantic structure [R2]. This page takes log(1 + count) of the table of how often words appear together, and extracts 8 dimensions with eigendecomposition. Today's language models learn embeddings as part of training to predict the next token. The position information of Section 5 is also added to these vectors.

5Position info

Position information — a bag of tokens does not know order

If you put tokens into a bag and count them without order (a bag of tokens), the order disappears. In Korean, particles mark each word's role, so the meaning often stays the same even when the word order changes — but not always.

Comparison

6Model processing

How the model processes it — the probability of the next token

Text → tokens → IDs → vectors (+ position) → Transformer (3-8) → probability of the next token. The simplest language model looks only at the previous word and counts the next word. We compute it ourselves with this corpus and see what a Transformer does in addition.

Previous word

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · How characters become numbers — Unicode, tokens, IDs, embeddingsLinked to Sections 1–6 · numpy, matplotlib · CPU 1 s

Computes Unicode and UTF-8, the Hangul jamo formula, BPE training and application, token IDs, embeddings built from co-occurrence counts, and previous-word probabilities without any NLP library. To use your own data, change CORPUS.

Output (run on 2026-10-03, Python 3.12 · numpy 2.4 · CPU). There is no randomness in the computation, so it gives the same result whenever you run it.

Code 2 · The same flow with scikit-learn — token IDs, sentence vectors, similar sentencesLinked to Sections 2, 3 and 4 · numpy, scikit-learn · CPU 1 s

Uses a production library to turn sentences into a "sentences × tokens" matrix, compares word-level and character-piece-level splitting, and finds similar sentences with TF-IDF and latent semantic analysis.

Output (run on 2026-10-03, scikit-learn 1.8 · numpy 2.4 · CPU). TruncatedSVD is fixed with seed 0.

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

Concept	Everyday analogy	Fruit fly / brain analogy	Where the analogy breaks down
Unicode	An ID number given to every character in the world	None	The number only tells characters apart; it says nothing about how similar two characters are
BPE	Making an abbreviation for words often used together ("초파리" as one chunk)	None	People make abbreviations by looking at meaning, but BPE merges by counts alone. Pieces unrelated to meaning also appear
Token ID	A library book's call number	None	Call numbers group books by subject, but token IDs are just Korean alphabetical order or order of appearance
Embedding	Coordinates on a map: nearby neighborhoods are similar	Partial The fruit fly's mushroom body represents an odor as an activity pattern (a vector) across many Kenyon cells	The mushroom body spreads an odor widely to make it easy to tell apart (sparse), while an embedding packs it densely into a few hundred dimensions. They are also built differently
Bag of tokens	Cutting a sentence into word cards and putting them in an envelope	None	Taking the cards out of the envelope does not tell you the original order. In Korean, the particle cards give some clues
Next-token probability	Guessing the next word in a word-chain game	Partial There is research showing that the brain also predicts upcoming speech sounds	The brain's predictions are tied to meaning, situation and bodily experience. The model computes only from statistics of text pieces

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

[R1] Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the ACL, 1715–1725. doi:10.18653/v1/P16-1162 — Section 2 (BPE tokenization)
[R2] Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407. doi:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 — Section 4, Code 2 (latent semantic analysis)
[R3] The Unicode Consortium. The Unicode Standard, §3.12 Conjoining Jamo Behavior (formulas for composing and decomposing Hangul syllables). unicode.org — Section 1

E-3 Corpus and figure references

The corpus is a 36-sentence example
The numbers come from the same computation as Code 1

Tips for Educators

A guide for instructors, on-the-job (OJT) trainers, and tutors. Times are estimates; adjust them to your actual class.

Audience	University students and practitioners who want to work with text data such as customer inquiries, reviews and documents. Anyone curious about how language models take in text
Prerequisites	The idea of vectors and cosine similarity. Knowing position information from 3-8 makes Sections 5 and 6 easier
Learning objectives	① Explain character → Unicode → bytes ② Explain how BPE builds a token vocabulary and how it handles unseen words ③ Explain why IDs, embeddings and position information are each needed
Estimated time	Reading 35–45 min · Demo 20 min · Colab practice 25 min (Code 1 15 min, Code 2 10 min)

Three lesson plans

Lecture, 50 min

Concept-focused

Intro 5 min (Section 1, 4-byte emoji) → Section 2 BPE 15 min → Sections 3 and 4 15 min → Section 5 dog bites man 5 min → Section 6 5 min → Check questions 5 min.

Workplace OJT, 90 min

Focused on applying to work

Build a corpus from 30 sentences of your team's documents 15 min → change CORPUS in Code 1 and recompute BPE and embeddings 30 min → find similar inquiries with Code 2 25 min → discussion on "unknown" tokens and handling personal data 20 min.

1:1 tutoring

Pace by level

Beginners: Sections 1, 2 and 4 only. Intermediate learners: have them do three BPE merges by hand (count word frequencies → find the most frequent pair), then compare with the rule list in Section 2.

Teaching points by section

Section	Questions to pause and ask	Hands-on tips
1 Unicode	"Why do texts with the same number of characters take different amounts of storage?"	Order: Korean sentence → "Hello 초파리" → emoji sentence
2 Tokenization	"Why was the Korean ending '한다' merged first?"	0 → 60 merges, then the sentence with an unseen word
3 Token IDs	"Does it mean anything that ID 54 is bigger than ID 5?"	Point out the "unknown" token in "AI가 꽃을 본다" ("AI sees a flower")
4 Embedding	"We never taught it 'insect', so why did the insects cluster together?"	Compare the kinds of neighbors in the order 초파리는 → 신경망은
5 Position info	"Korean word order is free, so why is position information needed?"	Press the three comparisons in turn
6 Model processing	"What is the weakness of a model that looks only at the previous word?"	Compare the probabilities for "초파리는" and "꿀벌은"

Top 5 common misconceptions

Misconception	Correct explanation	Scene to show
Token = word	A token is a piece determined by the corpus. It may be a character, a word, or something in between	Section 2
One Hangul character is 1 byte	In UTF-8, a Hangul syllable is 3 bytes	Section 1
Close token IDs mean close meanings	An ID is just a slot position in the vocabulary. Meaning is carried by the embedding	Sections 3 and 4
Close embeddings mean synonyms	Words that are used together also become close. What "similar" means depends on how the embedding is built	Section 4 "신경망은"
The model understands text like a person	It computes the probability of the next token with IDs and vectors. Whether it understands is a separate question	Section 6

Check questions (click to expand the explanation)

Q1. The Unicode code point of "가" is 0xAC00. What is the code point of "각", whose initial is ㄱ (0), medial ㅏ (0) and final ㄱ (1)?

0xAC00 + (0 × 21 + 0) × 28 + 1 = 0xAC01.

Q2. How many bytes does "초파리 🙂" (5 characters including the space) take in UTF-8?

3 Hangul characters × 3 + 1 space + 4 for the emoji = 14 bytes.

Q3. In BPE, what does "the pair that most often appears together" count?

For every word in the corpus, it counts each pair of adjacent pieces, weighted by how many times the word appears, and adds them up. The pair with the largest count is merged into one.

Q4. How is the unseen word "초파리들이" ("the fruit flies" + subject particle) handled at the word level and with BPE?

At the word level it is not in the vocabulary, so the whole word becomes "unknown". With BPE it is split into known pieces, like ["초파리", "들", "이"].

Q5. What are the bags of tokens for "dog bites man" and "man bites dog"? What is needed?

Both are the same: {dog:1, bites:1, man:1}. To tell the order apart, position information is needed.

Colab assignments

Basic

Add "나비는 꽃을 좋아한다" ("the butterfly likes flowers") and "나비는 날개가 있다" ("the butterfly has wings") to CORPUS in Code 1 and run it again. Check the nearest words to "나비는" (butterfly) and write down why, connecting it to the explanation in Section 4.

Challenge

Change the number of merges in bpe_train in Code 1 to 10, 30, 60 and 100, and make a table of vocabulary size and average number of tokens per sentence. Explain the trade-off between vocabulary size and token count, and give your reasons for which value you would choose for business documents.

Adjusting difficulty and further learning

Beginners	The jamo formula in Section 1 and Section 3 can be skipped without breaking the flow
Advanced	Have them research and compare tokenization methods other than BPE (unigram language model, morphological-analyzer-based). Check the papers they find in the original
Further learning	Sangsun Ahn, lecture notes for “Data Science for Business” (data science in general) · STEP [K-Digital] AI Natural Language Processing (directly linked to this part) · STEP [K-Digital] Fintech Data Analysis Basics (linked to 3-6)
Materials for instructors	Grading criteria for the check questions and sample answers for the assignments are on the answer key for educators (3-5)