Why does AI fail at predicting markets? Overfitting Issue
As a treasury specialist, there’s been one question that has constantly been on my mind during these past two years of my career: Why does AI fail at predicting markets? After all, AI has way more processing power and speed than any human. It should be able to analyze financial market parameters much faster and more efficiently than we ever could — or so I thought. This question kept nagging at me, so I decided to dig deeper. What I found was that the issue is far more complex than I initially assumed. There are several reasons why AI struggles with market predictions, but in today’s article, I’ll focus on just one: the problem of overfitting. While researching machine learning and overfitting, I came across the book Learning from Data — A Short Course (Abu-Mostafa et al., 2012), which helped me understand the topic much better. In my future articles, I plan to share more insights from this book as well.
Now, if you’re ready, let’s dive in!
Technical Definition of Overfitting
Overfitting is when an AI model learns patterns in the training data too well and therefore models noise or random relationships in the dataset instead of real signals. But before we understand the concept of overfitting, let’s first learn some basic concepts:
• Dataset: An organized collection of information that we want to analyze, process, or model. In the context of machine learning, a dataset usually consists of a large number of samples or observations, and each sample is represented by a set of features.
• Noise: Random or unsystematic variability or irregularity in the data. In data science terms, noise refers to unpredictable fluctuations that are not caused by the underlying true relationships.
• Target function: In the context of machine learning, a target function is a mathematical function that represents the ideal relationship that we are trying to estimate. This is sometimes called a “true function” or “underlying process.”
• Model function: A model function is a mathematical expression that we create to approximate the target function. This is sometimes called a “hypothesis function.” This is the function that the machine learning algorithm learns from data and uses to make predictions.
Now, to better understand the concept of overfitting, we can examine the graph.1. Let’s say we have a data set consisting of 5 data (points A, B, C, D and E). Let the target function of this data set be the green curve (f(x)). The target function is the function that we expect to teach with the data set in our model and to produce by the model. As can be understood from graph.1, our target function fits the data set with very little noise (point D). On the other hand, our model produces the g(x) function that crosses(fits) through all points in the data set.
The g(x) function produced by the model is fitting with all the data in the data set, but it has nothing to do with the target function. In this case, when the g(x) function is tested with the data set, it has “overfitted” the data. Mathematically speaking: For a model f and data (x, y), if the model f shows a low error rate in the training data (Etrain) but a high error rate in the test data (Etest), the model has overfitted:
In this case, the model “memorizes” past data but cannot generalize to new, unseen data.
At one point, testing the functions produced by the model with the data set we have puts us in a tautological situation. Although there are many solutions to the overfitting problem, the situation in the financial world is much more complicated than it seems.
Features of Overfitting in Financial Data Sets
The problem of overfitting is even more pronounced in the financial world than in other fields, and we can separate the situation under 3 main headings: High dimensionality and low sample count, low signal-to-noise ratio, and constant change in market conditions. Now let’s briefly examine these headings:
1. High Dimensionality and Low Sample Count
The number of variables affecting financial markets is very high: price movements, volumes, indicators, macroeconomic data, news, social media sentiment, and many more. However, especially when working with daily or longer time frames, the number of data points is limited. This “high dimensionality, low sample count” situation creates a perfect environment for overfitting in statistics.
This phenomenon, which Richard Bellman calls the “curse of dimensionality”, works as follows: As the number of dimensions increases, the number of samples required to maintain the same data density increases exponentially. For example, if you want to make a reasonable generalization for a 10-attribute model, you need proportionally fewer data points than you would for a 100-attribute model.
2. Low Signal-to-Noise Ratio
In financial markets, real signals are often very weak. Most of the daily variation in a stock’s price is due to random fluctuations rather than changes in the underlying value. This low signal-to-noise ratio leads models to learn random patterns rather than true relationships.
For example, about 97% of the variance in the daily variation of the S&P 500 index can be considered noise, with only 3% being predictable signals. Therefore, a model that claims to be 98% accurate is almost certainly overfitting(Lopez de Prado, 2018).
3. Continuous Change in Market Conditions
Financial markets are not static; their underlying dynamics change over time. Factors such as pandemics, wars, technological innovations, or regulatory changes can completely change market behavior. Therefore, a model that worked perfectly on 2010–2015 data may fail completely in the 2020–2025 period. This constant change makes it less likely that the relationships that models learn from their training data will hold true in the future. In statistical terms, this is called “distribution shift,” and it makes the overfitting problem even worse.
Practical Examples of Overfitting
To understand how overfitting occurs in financial markets, let’s examine some typical examples:
Backtest Overfitting
Investment strategies are often tested on historical data (backtest). A strategist or AI system adjusts the parameters to give the best performance on historical data. However, with enough parameters and enough trials, it is possible to create a strategy that looks great on any random data set.
As an example, consider a system that uses any combination of 5 technical indicators on a financial time series and optimizes the parameters of these indicators. If we try just 5 different parameter values per indicator, there are a total of (5⁵) = 3,125 different strategy combinations. It is almost certain that at least a few of these trials will perform exceptionally well on historical data, purely by chance.
P-hacking and Multiple Hypothesis Testing
A common problem in financial research and AI models is “p-hacking” or multiple hypothesis testing. If a researcher or algorithm tests enough variables and enough relationships, it is a mathematical certainty that some of them will appear statistically significant.
For example, if you run 20 different correlation tests between two random time series (using different time lags or different transformations), the probability that at least one test will be “significant” by chance at a 95% confidence level is 64%. When hundreds or thousands of tests are run, this probability approaches 100%.
In the financial world, the sheer number of tests performed on time series (for example, different stocks, different time frames, different indicators) creates the perfect environment for discovering spurious patterns.
Strategies to Combat Overfitting
Overfitting in financial markets cannot be completely eliminated, but it can be reduced with effective techniques. Some of these techniques include:
- Regularization: reduces overfitting by controlling the complexity of the model.
- Forward cross-validation: shifts the time series forward, making the training and test points different.
Ensemble methods: attempt to reduce overfitting by combining multiple models. - Bayesian approaches: instead of estimating points by incorporating domain knowledge into the model, they add a domain probability distribution to the model.
- Economic constraints: instead of a purely statistical approach, they try to reduce overfitting by incorporating economic theory and financial knowledge into the model.
A New Perspective: Risk Management Instead of Prediction
Given the challenges AI faces in predicting financial markets, perhaps our focus should be different. Instead of making absolute predictions, a more realistic approach might be to focus on better understanding and managing uncertainty.In the 21st century, where new developments and milestones are achieved every day, understanding how AI works is crucial to using it effectively.
Thanks for your reading and patient :)
Semih
References
Abu-Mostafa, Y. S., Magdon-Ismail, M., & Lin, H. T. (2012). Learning from data (Vol. 4, p. 4). New York: AMLBook.
Bellman, R. 1952. “On the Theory of Dynamic Programming”. Proceedings of the National Academy of Sciences, 38(8), 716–719
Lopez de Prado, M. (2018). Advances in financial machine learning (chapter 1). Advances in Financial Machine Learning, Wiley, 1st Edition (2018).
