An unscrupulous investment advisor with a 100,000-person mailing list sends a newsletter to half of the list predicting that the market will rise and a version to the other half of the list predicting that it will fall. At the end of every quarter he discards the names of the people to whom he sent the wrong prediction and repeats the process with the remainder. After two years he signs up the 1,562 recipients who are amazed at his track record of predicting the market eight quarters in a row.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
The "Marketing trick" you should know....
- Thread starter Tb2018
- Start date
backtest overfitting
By backtest overfitting, we mean the usage of historical market data to develop an investment model, strategy or fund, where many variations are tried on the same fixed dataset. Backtest overfitting, a form of selection bias under multiple testing, has long plagued the field of finance and is now thought to be the leading reason why investments that look great when designed often disappoint when actually fielded to investors. Models, strategies and funds suffering from this type of statistical overfitting typically target the random patterns present in the limited in-sample test-set on which they are based, and thus often perform erratically when presented with new, truly out-of-sample data.
The potential for backtest overfitting in the financial field has grown enormously in recent years with the increased utilization of computer programs to search a space of millions or even billions of parameter variations for a given model, strategy or fund, and then to select only the “optimal” choice for publication or market implementation. The sobering consequence is that a significant portion of the models, strategies and funds employed in the investment world, including many of those marketed to individual investors, may be merely statistical mirages.
By backtest overfitting, we mean the usage of historical market data to develop an investment model, strategy or fund, where many variations are tried on the same fixed dataset. Backtest overfitting, a form of selection bias under multiple testing, has long plagued the field of finance and is now thought to be the leading reason why investments that look great when designed often disappoint when actually fielded to investors. Models, strategies and funds suffering from this type of statistical overfitting typically target the random patterns present in the limited in-sample test-set on which they are based, and thus often perform erratically when presented with new, truly out-of-sample data.
The potential for backtest overfitting in the financial field has grown enormously in recent years with the increased utilization of computer programs to search a space of millions or even billions of parameter variations for a given model, strategy or fund, and then to select only the “optimal” choice for publication or market implementation. The sobering consequence is that a significant portion of the models, strategies and funds employed in the investment world, including many of those marketed to individual investors, may be merely statistical mirages.
Statistical errors in finance
The field of finance is also coming to grips with the fact that the field is rife with the misuse of probability and statistics. Indeed, such errors are now thought to be a leading reason why investment strategies and funds that look great on paper often fall flat when actually fielded.A leading reason for such failures is backtest overfitting, namely the deplorable practice, conscious or not, of using historical market data to develop an investment model, fund or strategy, where too many variations are tried, relative to the amount of data available. Models, funds and strategies suffering from this type of statistical overfitting typically target the random patterns present in the limited in-sample test-set on which they are based, and thus often perform erratically when presented with new, truly out-of-sample data. The sobering consequence is that a significant portion of the models, funds and strategies employed in the investment world, including many of those marketed to individual investors, may be merely statistical mirages.
Other areas of finance that are rife with statistical errors include:
- Technical analysis. Although widespread in the field of finance, “technical analysis” is every bit as pseudoscientific as astrology. Does anyone really believe that low-tech analysis of “trends,” “waves,” “breakout patterns,” “triangle patterns,” “shoulders” and “Fibonacci ratios” (none of which withstand rigorous statistical scrutiny) can possibly compete with the mathematically and statistically sophisticated, big-data-crunching computer programs, operated by successful hedge funds and other large organizations, that troll financial markets for every conceivable trading opportunity? Think again. The bottom line is that technical analysis does not work in the market.
- Day trading. Another unpleasant truth is that day trading, namely the widespread practice of frequent buying and selling of securities by amateur investors through the trading day, does not work either. Study after study has shown that the large majority of day traders lose money, many with spectacular losses; only a tiny fraction regularly earn profits. For example, a 2017 U.C. Berkeley-Peking University study found that even the most experienced day-traders lose money, and nearly 75% of day-trading activity is by traders with a history of losses.
- Market forecasters. The statistical record of market forecasters is, in a word, dismal. According to Hickey’s analysis of market forecasts since 2000, for instance, the average gap between the median forecast and the actual S&P 500 index was 4.31 percentage points, or an error of 44%. In 2008 the median forecast was for a rise of 11.1% The actual performance? A fall of 38.5%, i.e., a whopping error of 49.6 percentage points. Similarly, Nir Kaissar lamented that the forecasts have been least useful when they mattered most. Jeff Sommer, a financial writer for the New York Times, recently summarized the dismal record of 2020 stock market forecasters as follows: “[A]s far as predicting the future goes, Wall Street’s record is remarkable for its ineptitude.” A recent study of 68 market forecasters by the present author and colleagues found accuracy results no better than chance.
Good info here. It may take some time to digest what you said and more than that to draw practical conclusions. Which is "very simple but not easy" - like options trading...
I can add another trick that can be used by strategy/indicator/whatever sellers to provide good track record. It is based on the same principle as mailing list trick you mention.
If one wants to sell subscription to, let's say: 100% accurate TA indicator (or 92.76% for 'engineer types'). He can put two trades in opposite directions. One will be a winner and the other a looser. Then he discard losers and post broker's statements for winning trades ->- pocket subscription fees.
It is all based on human psychology.
I cant decide what is worse: a cold blooded shenanigan trying to skim populus or somebody who truly and deeply believes he discovered the 100% indicator and talks it with full enthusiasm and excitement.
I wrote enough about BT and don't think there is need to expand more on that.
I'm still surprises about lack of response to those claims (or challenging them).
I'd like to stop on two quotes:
I refer to options related material but can assume it affects other fields as well.
The main reason (imo, imo) lays deeper. It is lack of understanding of mathematical probabilities and market probabilities. I observe this fallout even when listening to some great market traders. They do have results, they know how to trade but when describing what they are doing often they use statistics/probabilities numbers - but they do it only post factum - in real trading they do not use those as we think - even if they don't realize that consciously (otherwise they won't have results they have).
Good stuff Tb2018.
I can add another trick that can be used by strategy/indicator/whatever sellers to provide good track record. It is based on the same principle as mailing list trick you mention.
If one wants to sell subscription to, let's say: 100% accurate TA indicator (or 92.76% for 'engineer types'). He can put two trades in opposite directions. One will be a winner and the other a looser. Then he discard losers and post broker's statements for winning trades ->- pocket subscription fees.
It is all based on human psychology.
I cant decide what is worse: a cold blooded shenanigan trying to skim populus or somebody who truly and deeply believes he discovered the 100% indicator and talks it with full enthusiasm and excitement.
I wrote enough about BT and don't think there is need to expand more on that.
I'm still surprises about lack of response to those claims (or challenging them).
I'd like to stop on two quotes:
This is bull's eye hit.the field is rife with the misuse of probability and statistics
I refer to options related material but can assume it affects other fields as well.
This is one of the reasons but it refers back to BT issue.A leading reason for such failures is backtest overfitting,
The main reason (imo, imo) lays deeper. It is lack of understanding of mathematical probabilities and market probabilities. I observe this fallout even when listening to some great market traders. They do have results, they know how to trade but when describing what they are doing often they use statistics/probabilities numbers - but they do it only post factum - in real trading they do not use those as we think - even if they don't realize that consciously (otherwise they won't have results they have).
Good stuff Tb2018.
I often think of this topic (with desire for self improvement)
There is also a more discomforting observation to trusting false information.
I keep recalling Jack Nicholson's rant in "a few good men". " You can't handle the truth!" We so much want things to be simple, clean, clear, and nice, we may cease pursuit if these are not primary attributes--
We often fail to think about what we are doing adequately.
I recall a large engineering firm I was with for about 20 years, hiring this Russian math PHD guy (a very sharp guy) to report on a specific reliability related study. In the beginning of his paper, he included his assumptions for the study. (the assumptions were false) Our management ignored the assumptions and made decisions based on his results. The project was later terminated with heavy layoffs. Duh! -- The PHD guy was correct, the management just failed to understand! "IF x then y!" Does NOT infer x is true!
All too many times these things are self-inflicted! -- It (fallacy) can really sneak up on us, even when we THINK we are being mindful. -- I have bit my tail more times than I care to disclose!
There is also a more discomforting observation to trusting false information.
I keep recalling Jack Nicholson's rant in "a few good men". " You can't handle the truth!" We so much want things to be simple, clean, clear, and nice, we may cease pursuit if these are not primary attributes--
We often fail to think about what we are doing adequately.
I recall a large engineering firm I was with for about 20 years, hiring this Russian math PHD guy (a very sharp guy) to report on a specific reliability related study. In the beginning of his paper, he included his assumptions for the study. (the assumptions were false) Our management ignored the assumptions and made decisions based on his results. The project was later terminated with heavy layoffs. Duh! -- The PHD guy was correct, the management just failed to understand! "IF x then y!" Does NOT infer x is true!
All too many times these things are self-inflicted! -- It (fallacy) can really sneak up on us, even when we THINK we are being mindful. -- I have bit my tail more times than I care to disclose!
Last edited: