3-6 Time Series & Stock Prices — Beat the Baseline First

1Windows & Targets

Windows, normalization, targets — turning a row of numbers into a problem

A single row of prices is not enough to learn from. You have to turn it into a problem: "look at the last L days and predict the next day." Each time you slide the window by one step, you get one more (input, answer) pair.

Values shown

Window length

Why returns instead of prices, and how far normalization goes

Prices sit at different levels in different periods (about $120 in 2015, about $90 in 2016), so the test period brings ranges never seen during training. That is why we convert them to daily rates of change (log return = log(today/yesterday) × 100%). The reference for normalization (subtracting the mean and dividing by the standard deviation) is computed only on the training period. If you use the mean of the whole period, information from the test period leaks into training (the leakage in Section 4). You also have to decide first what the target is: "next day's return (a number)", "up or down (direction)", or "size of the move".

2Hopes vs. Reality

Why deep learning raises hopes — and what the data says first

A neural network finds nonlinear relationships among many signals on its own. That is why people have high hopes for it on problems like stock prices, where many factors are tangled together. But there is something to ask the data first: "Is yesterday's return related to today's return?"

Autocorrelation of what

3Research Cases

Cases where it helped and cases where it didn't

Two large studies reach different conclusions. You need to look at both: under which conditions things improved and under which they did not.

Case

4Traps

Four traps — leakage, overfitting, structural change, transaction costs

Most "amazing results" in stock prediction come from one of these four. We look at each one with numbers from real data.

Trap

Transaction cost (per switch)

5Walk-Forward

Walk-forward validation — learn only from the past, step forward one step at a time

Evaluate exactly as in real use. Train only on the past up to that day, predict the next few days, and once those days have passed, add them in and train again. The test period must always come after the training period.

How to split

6Comparison

Baseline → linear → LSTM — same test, same yardstick

The last 200 trading days (2016-05-04 ~ 2017-02-16) were predicted with walk-forward. The strategy is "hold if you expect a rise, cash if you expect a fall", and 0.1% is subtracted every time you switch between holding and cash.

View

Colab Code — Run This Page Yourself

Both code examples use only libraries preinstalled in Colab. After "Download notebook (.ipynb)", open it in Colab with File → Upload notebook, or copy the code and paste it into a new notebook cell.

Code 1 · Evaluating the baseline and a linear model with walk-forwardLinked to Sections 1·4·5·6 · numpy, matplotlib · CPU 1 s · internet required (data download)

Downloads a public CSV, evaluates two baselines and a ridge linear model with walk-forward, and compares the strategy after transaction costs with buy-and-hold. It also shows the leakage and overfitting traps in numbers. To use your own data, change PRICE.

Output (run on 2026-10-03, numpy 2.4 · CPU). With the same data, it gives the same result whenever you run it (the overfitting experiment fixes seed 0).

Code 2 · Evaluating a Keras LSTM with the same walk-forwardLinked to Section 6 · tensorflow (Keras), numpy · CPU about 2–4 min

An LSTM that predicts the next day from the last 20 days of returns is retrained every 50 days. The results of two seeds are reported separately. The table in Section 6 is built directly from this output.

Output (run on 2026-10-03, TensorFlow 2.21 · Keras 3.15 · CPU). Numbers may differ depending on seed and version.

AI Prompts — Using This Code as a Starting Point

These are prompts to paste into AI assistants such as ChatGPT, Qwen, or Claude. Pressing "Copy" attaches the chosen code after the prompt. Replace the parts in 〔〕 with your own situation.

Code to attach

An AI's answer is only a starting point. Always check any papers, figures, or library functions it suggests against the original sources and official documentation. If you used AI help for a competition entry or paper, disclose the fact and extent of its use according to the rules of the organizers and the journal.

Glossary

This is the same dictionary as the yellow dotted underlines in the text. The same terms are used throughout the series. Click a section name to jump to where the term first appears.

Analogies — and Where They Break Down

Analogies are a first step to understanding, but they can also seed misunderstanding. Fruit fly and brain analogies are marked by strength. Direct evidence that they actually do the same thing · Partial only some properties are similar · None no known counterpart

Concept	Everyday analogy	Fruit fly / brain analogy	Where the analogy breaks down
Window	Predicting tomorrow's weather from the past week's weather	Partial A fruit fly also decides its direction from changes in smell over the last few seconds	Weather follows physical laws, but with stock prices, people's predictions themselves change the price
Baseline	A weather forecaster who says "tomorrow will be the same as today"	None	It looks simple but is hard to beat. Every model has to beat this first
Data leakage	The score of a student who saw the answer sheet before the exam	None	Usually you peek without realizing it. It sneaks in through normalization and feature calculations
Overfitting (selection bias)	Tossing 100 coins and calling the ones that landed heads most often "lucky coins"	None	The "luck" vanishes on the next toss. But with real strategies, this is much harder to spot
Structural change	Predicting winter sales from summer clothing sales records	Partial When the environment changes, a fruit fly's learned memories are updated too	Seasons can be predicted, but you never know when the character of the market will change
Walk-forward	A mock exam where each week you predict the next week using only the records so far	None	Unlike a mock exam, the market does not wait for you, whether your prediction is right or wrong

About the Author and References

E-1 About the author · E-2 Academic references · E-3 Figure sources

E-1 About the author

Sangsun Ahn (CEO, M-Robo Inc.)

After working at the Korea Electrical Safety Corporation and the Maeil Business Newspaper, he founded M-Robo Inc., a spin-off venture of Maeil Business, where he currently serves as CEO. M-Robo Inc. began as an in-house venture of a media company; it provided a robot journalist (StockBot) service to Kukmin Ilbo and has carried out many projects for the Korea Internet & Security Agency, the Korea Press Foundation, the Korea Communications Agency, and others. It now runs its own AI news service, Money-Robo. As CEO he won the Asia Open Data Hackathon and twice received the Minister of the Interior and Safety Award for contributions to data entrepreneurship, and he is the inventor and registrant of 5 patents in artificial intelligence and blockchain. He is the author of “AI Crime and Self-Defense”, and has developed e-learning content on fintech, statistics, and artificial intelligence for the Online Lifelong Education Institute of Korea University of Technology and Education. He holds a PhD in economics (Dongguk University, micro-mathematical economics). He teaches data and AI courses at several universities, including Kookmin University and Hoseo University. Full profile

Courses

Ahn, Sangsun (2025). “Data Science for Business” lecture notes. Department of AI Convergence, Open Cyber University of Korea.
Ahn, Sangsun (2025). [K-Digital] AI Natural Language Processing. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.
Ahn, Sangsun (2024). [K-Digital] Fundamentals of Fintech Data Analysis. STEP (Smart Training Education Platform), Online Lifelong Education Institute, Korea University of Technology and Education.

STEP course links: link.step.or.kr/1qyVoJ · link.step.or.kr/M9na9N · link.step.or.kr/brqEyc
If the links do not work due to site reorganization, please visit www.step.or.kr.

Papers and presentations

Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI model-based approach. “Future Society”, 16(2). Full text
Ahn, Sangsun (2025). A study on household expenditure prediction models using the Korean Retirement and Income Study panel data: An AI approach. Presentation at the KReIS Panel Conference (2025.05.13), National Pension Research Institute, National Pension Service. Slides
Ahn, Sangsun (2025). A study on an algorithm for detecting suspected drug transactions from SNS text: Focusing on pattern and context analysis. “Future Society”, 16(1). Full text
Ahn, Sangsun (2024). Statistics-based countermeasures against adversarial data for securing anomaly detection models in finance: Empirical analysis and simulation results. “Future Society”, 15(3). Full text
Ahn, Sangsun (2024). An AI-based solution for detecting SNS posts based on virtual character traits. Poster presentation, 2024 Summer Newmedia Party (2024.06.19), Seoul Media Institute of Technology.
Ahn, Sangsun (2024). Machine learning models using industrial accident panel data. Poster presentation, 10th Panel Study of Workers' Compensation Insurance Conference (2024.05.17), Labor Welfare Research Institute, Korea Workers' Compensation & Welfare Service.
Ahn, Sangsun (2024). Analysis and prediction of virtual asset time series data: Comparing time series models and AI models, and suggesting improvements. “Future Society”, 15(1). Full text

E-2 Academic references

[R1] Gu, S., Kelly, B., & Xiu, D. (2020). Empirical asset pricing via machine learning. The Review of Financial Studies, 33(5), 2223–2273. doi:10.1093/rfs/hhaa009 — Section 3 (trees and neural networks performed best; out-of-sample Sharpe ratio of the neural-network long-short portfolio 1.35)
[R2] Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1), 54–74. — Section 3 (all 6 pure machine-learning methods performed poorly; the winner was a statistics–machine learning hybrid)

E-3 Data and figure references

Stock data: Plotly example dataset finance-charts-apple.csv (GitHub plotly/datasets). The dividend-adjusted close (AAPL.Adjusted) was used

Tips for Educators

A guide to running the session for instructors, OJT trainers and tutors. Times are estimates, so adjust them to your actual class. So that this part is not read as an investment recommendation, state clearly at the start and end of class: "This is an educational example, not investment advice."

Audience	University students and practitioners who want to forecast time series such as demand, sales, sensor data or prices; fintech beginners
Prerequisites	Mean and standard deviation, overfitting and validation from 3-2, LSTM from 3-7
Learning objectives	① Turn a time series into a learning problem using windows, normalization and targets ② Explain leakage, overfitting, structural change and transaction costs with numbers ③ Compare baseline and models by the same yardstick using walk-forward
Estimated time	Reading 40–50 min · demo 20 min · Colab practice 30 min (Code 1 10 min, Code 2 run 4 min + interpretation 15 min)

Three lesson plans

Lecture, 50 min

Concept-focused

Introduction 5 min (the hero's "LSTM is worse than the baseline") → Sections 1·2 15 min → the four traps in Section 4 15 min → Sections 5·6 10 min → check questions 5 min.

Workplace OJT, 90 min

Focused on applying to work

Set a baseline for your team's forecasting target (demand, sales, etc.) 15 min → turn Section 4 into a leakage checklist for your data 25 min → put your time series into Code 1 35 min → discussion on how to report results 15 min.

1:1 tutoring

Pace by level

Beginners: Sections 1·4·6 only. Have intermediate learners trace the walk-forward loop in Code 1 by hand, and check for themselves how the numbers change when the standardization reference is switched to the whole period.

Teaching points by section

Section	Questions to pause and ask	Hands-on tips
1 Windows & Targets	"What goes wrong if you feed in prices as they are?"	Click Price → Daily return, in that order
2 Hopes vs. Reality	"Which is easier to predict: direction or size of the move?"	Click Returns → Absolute returns, in that order, and compare the number of bars outside the band
3 Research Cases	"Why do the two studies reach different conclusions?"	Compare the data size and problem type rows side by side
4 Traps	"If you test 100 and pick one, can you trust its score?"	Look at Overfitting → Transaction costs (0 → 0.3%), in that order
5 Walk-Forward	"Why is a random shuffled split not allowed?"	Switch between the two modes
6 Comparison	"The LSTM was worse than the baseline. What could we try changing?"	Look at RMSE → strategy return, in that order

Top 5 common misconceptions

Misconception	Correct explanation	Scene to show
LSTMs specialize in time series, so they beat the baseline	In problems with few clues, the baseline is hard to beat. In this experiment, the LSTM was worse than the baseline	Section 6
A 55% direction hit rate makes money	If you lose big when you're wrong, you lose overall. You also need to account for transaction costs and compare with buy-and-hold	Baseline row in Section 6 (55.0%)
If it did well in the backtest, it will do well in real trading	When you pick after many tries, chance creeps in. The character of the market also changes	Section 4: overfitting, structural change
Normalizing once on the whole dataset is fine	The reference must be computed only on the training period. Otherwise future information leaks	Section 1, Section 4 leakage
Neural networks are useless for stock prediction	There are studies where they helped with large data, many signals and cross-sectional problems. Conditions matter	Section 3

Check questions (click to expand the explanation)

Q1. With 505 returns and a 5-day window, how many (input, answer) pairs are there?

505 − 5 = 500.

Q2. If you retrain every 20 days over 200 test days, how many times do you train? How many pairs are in the first training run?

10 times. The first training run has 500 − 200 = 300 pairs, and the last has 480 pairs.

Q3. If a strategy traded 40 times and the transaction cost is 0.1% per trade, how much return was lost to costs?

40 × 0.1 = 4%p (approximate, based on log returns).

Q4. Picking the best of 100 random rules on the training period gave an excess of +43.7%p on the training period and −22.4%p on the test period. What does this show?

A result picked from many tries is fitted to chance, so it does not repeat on a new period (overfitting, selection bias). The test period should be used only once, at the end.

Q5. The baseline "tomorrow = today" had a smaller prediction error than the LSTM. How should this be written in a report?

Report the result as it is: "On this data (1 stock, 506 days, signal = past returns) and with this setup (20-day window, retraining every 50 days, 40 epochs), the LSTM did not beat the baseline (RMSE 1.462%·1.550% vs. 1.209%)." Then state which conditions could be changed and the limits of generalization.

Colab assignments

Basic

In ridge_fit in Code 1, change the standardization reference (mu, sd) to the whole period (all of X) and run it again. Record how the RMSE and strategy return change, and write in two sentences why this is leakage.

Challenge

Change the target of Code 2 from "next day's return" to "absolute value of next day's return (size of the move)". Use "mean absolute value over the past 20 days" as the baseline. Linking to the autocorrelation in Section 2, report with 3 or more seeds whether the LSTM beats the baseline at predicting volatility.

Adjusting difficulty and further learning

Beginners	Sections 2·3 can be skipped without breaking the flow. Be sure to cover the overfitting in Section 4 and the table in Section 6
Advanced	Have learners research how to correct the performance of a backtest that was tried many times (multiple testing). Check the papers they find in the original
Further learning	Sangsun Ahn, “Data Science for Business” lecture notes (data science in general) · STEP [K-Digital] Fintech Data Analysis Basics (directly connected to this part) · STEP [K-Digital] AI Natural Language Processing (connected to 3-5·3-8)
Materials for instructors	Grading criteria for the check questions and sample answers for the assignments are on the Answer key for educators page (3-6)