Completely agree. Weekly back test of trades during the greatest bull market and vol crush not seen before is meaningless. Also skill is a less of a factor during weekly trade setup.
Sheridan Mentoring $4k on 20K monthly
- Thread starter HowardS
- Start date
Completely agree. Weekly back test of trades during the greatest bull market and vol crush not seen before is meaningless. Also skill is a less of a factor during weekly trade setup.
G
Gabor Maly
Guest
Wov, strong statements there. Yes you need to make sure that your backtests cover a wide range but you also want to make sure you understand what market regime you are trading against....it is the bull market for now and who knows maybe for the next 5 years it stays like that.
Strong words? may be.
Any backtest should address 3 S's.
Scenario Analysis, Stress test and Sensitivity analysis.
When the period tested does not even a pick one of the most important (bearish) scenario, test is a suspect. Forget the extreme stress test, when there were no period of even technical bear market in the testing period. 40/50 fixed point wings when index was in the range of 1100 to 2550 is not anybody's definition of sensitivity analysis.
Any backtest should be taken with grain of salt because it is basically one path in the history. Atleast it gives range of scenario's when tested properly.
It was puzzling to see DGH's comment regarding "Skilled trader". Any backtest which requires skill of trader to be successful, then that backtest is doomed.
If you want a confirmation bias one can look at any short-vol ETF and extrapolate.
Any backtest should address 3 S's.
Scenario Analysis, Stress test and Sensitivity analysis.
When the period tested does not even a pick one of the most important (bearish) scenario, test is a suspect. Forget the extreme stress test, when there were no period of even technical bear market in the testing period. 40/50 fixed point wings when index was in the range of 1100 to 2550 is not anybody's definition of sensitivity analysis.
Any backtest should be taken with grain of salt because it is basically one path in the history. Atleast it gives range of scenario's when tested properly.
It was puzzling to see DGH's comment regarding "Skilled trader". Any backtest which requires skill of trader to be successful, then that backtest is doomed.
If you want a confirmation bias one can look at any short-vol ETF and extrapolate.
Hi Srini. My comments regarding "a skilled trader" did not relate to back testing in any way. This thread is very generic in the sense that it started as a discussion of making money with a weekly trading strategy, not as a back testing thread. I'm not a big fan of back testing, at least not in the manner in which many people use the results. There is always the possibility of unintentional curve fitting. There are several websites which sell back testing results or back testing methodology which, as you might expect, always make the product look good. None of them have decent predictive value in my view. Part of the problem, which became very apparent when I was working with the development of Quanty Carlo, is that missing or incorrect data will either nullify the test or, even worse, give a false result. In the Quanty Carlo developmental phase we would often get an individual result labeled "Aborted", usually because of a missing strike or a bad print. This anomaly skewed the interpretation of the results when looking at the usual performance parameter statistics. The other problem, of course, is the oft-stated maxim: what worked then might not work now. I do think, however, that there is some value in applying a strategy to the "worst" area of a price history to see how it held up and what the draw down was. That was my basic approach in the development of the RTT and the Weirdor. I knew that those strategies would make money in "normal" environments, but I used the appropriate price history and volatility data during "bad" environments to ensure that I had a workable strategy which included adjustments that I would make during those times. Just my two cents worth.
Are there any automated backtesters available that make cost sense for the retail trader? The last one I was looking into was Quanty Carlo and then it disappeared. I've done extensive backtesting with OV but it's exhausting because it will take me 3-5 months to go though the whole RUT data set. SPX is more frustrating... I don't think it has clean data since maybe 2009 or later? It also takes so long to load those matrices...
One thing I've found is that most people who talk about "backtesting" really don't know a whole lot about trading system development methodology. I would certainly consider myself a student of the game--not an expert--but I have read a handful of books on the subject and I have a statistical background as well. I'm sure there are people who backtest who know more than I do but where are they? Do they get scooped up by the institutions and work in quant departments? Do they become institutional themselves and are therefore hidden from [retail trader] view? This is one aspect of the industry that has been mysterious to me.
And I strongly believe that simply going institutional does not guarantee success. We've all probably seen the estimated trader success rate: fewer than 10-20% [full-timers?] succeed over a number of years. I consider this generic number to hold for any entrepreneurial pursuit. I've done a little digging with regard to hedge funds and found 3-5 years as the average lifetime or that 80% of them fail so "institutional" is not an exemption from failure.
UPDATE FROM TOM: CMLViz Trademachine is a really good backtesting platform.
Last edited by a moderator:
This. To see the truth of this statement, look no further than the restaurant industry in your city/town.
Posted by Ice101781
K
KevinK
Guest
Strong words indeed. To say that it is meaningless I think is dismissive and not conducive to fruitful discussion. The landscape of options trading and the market in general has changed so much in the last 5 years, not to mention the last 10,15 and 20 years that talking about back testing strategies through the 90s, 80s or earlier is a moot point. The depth of options expiration, algos, Dodd-Frank, plethora of vol based strategies, options traders etc. have all changed the market environment considerably.
I would disagree that skill is less of a factor during weekly trade setup. That depends largely on what strategy is being employed. I would say it requires even more skill (which is usually judgment when it comes to trading) as there is less room for error. The key advantage of short term trades in this environment is the limited impact sudden shifts in vol skew have on your position. Anyone trading longer term near ATM BWBs this year will have experienced draw downs in the big rallies we've seen over the past few weeks and in feb/march. The shorter term trades have fared much better in this environment. The damage being done to the longer term trades is primarily volatility skew shifts and not price/delta risk, at least from my experience. Shorter term trades offset that as the higher theta decay provides a buffer.
If we suddenly morphed into a bear market, I wouldn't be trading weekly strategies. There would be little need as the volatility landscape would be quite different. Could a 1987 type even happen at any moment? Theoretically yes, but in that case short or long term, nearly every market neutral trade will lose, which is why you should only employ a tiny fraction of your allotted options capital for 15DTE and under trades and be comfortable with the prospect of losing your capital (regardless of DTE). This is all part of crafting a sound trade plan.
Interesting discussions.... I do want to pick up on a point that Srini made, which I think is extremely important. To me, the primary objective of back testing isn't to find out whether a strategy will offer x% or y% profits, but rather to study in detail how the strategy will perform in various scenarios and to research how specific actions/adjustments will perform. In his words, the 3S is a good characterization. Seldom do I even care about the P&L of the strategy in back tests scenarios. These P&L might give us a sense of achievement, but they are not very useful and sometimes downright misleading.
Instead, what I focus on is how does the position react in a massive sell off, a long rally or a zig-zagging market etc.. What if IV explodes, or grinds down ? What if IV skew steepens or flattens? How does DTE affect the reactions? How about using different type of spreads with slightly different greeks - which one provides better results in which scenarios? Is 20 pt spread better or 40 pts and in which scenario ? What if we go more OTM, how does that affect the performance ? etc.. etc... you get what I mean. I deliberately find markets periods with such scenarios to perform tests and do research. And most important of all, not only do I want to know how different strategies react, I want to find out WHY - that is finding out by decomposing the complex position into its atomic parts and study the pricing behaviours.
Therefore, I do think those type of back testing that is conducted like speed dating, ie stepping through dates quickly, briefly stopping just to look for adjustment greeks, are in fact quite useless and build a false sense of security in the effectiveness of strategy and one's perception of trading skills. The reason is future will likely not repeat history exactly and what advance traders need instead is the deep understanding of how options behave in different scenarios that market might throw at them and build up a tool set to react.
OK - I make sure I don't miscommunicate - for a beginner trader, back testing this way just to learn how to trade a given plan, I think that's fair. The above comment is meant for advance traders.
Kevin's post is worth saving and coming back to as you progress. The fastest way to learn the trades is ironically doing the slowest backtesting as Kevin mentions. Once you understand the mechanics, stepping day by day through a trade and really consider what is happening in the market and how your trade is reacting will tremendously improve your trading. I am speaking from personal experience, I used to backtest a trade mechanically adjusting as I went without a lot of thought of what was really going on in my position/the market. When I would put it on live, naturally the trade would perform differently and I was left scrambling to figure out what I should do with the position.
Paul Demers
New member
I agree with Trader G. I find it interesting that this thread evolved into a discussion about backtesting. I guess my previous post is responsible for that and I am sorry that it hijacked the thread. I want to clarify the results that I showed. I created those results with an automated backtester that had no adjustments. I had done a manual backtest a few years ago with an adjustment plan but I could not find those results.
So this time it is different.
I have heard these similar arguments before. In 2007 real estate was no loose business, because it performed beautifully in 2000-2002 bear market & past 30 years and people are moving towards ownership society etc. In 2000, arguments like earnings , revenues does not apply to new companies. It has to be valued based on eyeballs on their website.
Same algo's can reverse quickly. Plethora of vol strategies have same component (short vol) and all those strategies and traders is basically betting on same crowded trade. One does not need to back test to see the continuation of status quo.
Generally shorter the time frame, higher the risk (This can't be disputed) and along with lower premium support. May be skill is required before you enter the trade to take the trade or not. You can take extreme example, entering the trade just before earnings. All the skill goes before entering the trade, not much can be done afterwards.
That discussion was about weekly backtest. How and why should anyone incorporate skill of trader in their testing?
SriniR