Moving Averages, the mother of all technical indicators Forex Trading Robot usage forces changes upon Forex Brokers
Apr 16

Understanding Trading System Backtesting Limitations

The last couple of months I have been heavily involved in developing and backtesting an automatic trading system for an institutional investor.  To protect the innocent, I’m not going to disclose what derivative instrument this new automatic trading system is targetting.  What I do want to share are some of the finer points of trading system backtesting.  Read on…!

Let’s first define what backtesting really is:  it’s using historical price data and push that data series through your trading system.  In the end you hope that the trades your system takes with this historical data would also have been taken in real-time, i.e. you hope that your backtesting results closely reflect what real trading would have produced.  Only - depending on your trading system basics - the don’t!

Let’s try and figure out what causes the discrepancies, shall we?

Historical data for backtesting your trading system

First of all we have to determine what kind of historical data is used to drive the trading system backtesting suite.

Open-High-Low-Close of candleThis illustration shows the normal candle presentation that is most often used to display market data.  Each candle can represent a specific time interval, a specific number of contracts handled or even a specific number of ticks (i.e. number of transactions without considering volume of these transaction too).

What kind of market data aggregation is used to define a single candle is left to the trader - or quite often his/her choice is limited by the trading software that he uses. Let’s for now assume that each of these candles contains one hour worth of market data.   Also, let’s assume that your system opened a position at the open of the rightmost candle, that the profit target is somewhere below the candle’s high and that the stop loss is somewhere above the candle’s low.

Then answer this question: how can the software determine which gets triggered first:  the profit target or the stop loss?  The answer is:  the software can’t !  This of course is detrimental to accurate backtesting.  Good backtesting software will play it on the safe side - this means it will prevent that the investor is lulled into a false sense of security.  This means the software will assume that the stop loss gets executed first.  It will close the position at a loss, without ever knowing whether perhaps in real life the profit target would have been hit first.

Conclusion:  for backtesting purposes candles don’t cut it, however they aggregate market data.  The only solid recourse is to use individual ticks to backtest your trading systems, as only then the exact order of events are known to your backtesting software.

Of course this poses a problem, as most retail trading platforms don’t offer genuine tick data going back any further than a couple of weeks up to perhaps one or two months - hardly enough to base a solid trading system backtesting program on.  Some trading platforms try to cop out and do the following:  they use the smallest data aggregation available (most often 1 minute candles) and then this software generates a synthetic tick data series that replaces the market data candles.  This software even tries to qualify this process with what it calls a ‘tick data modelling quality’, which is never above 90%  (it would be nice to learn how this metric is determined though…).  This method too cannot decide beforehand, how a one minute candle became to be, ie. in what order price action caused that candle to be shaped the way it is.  In that sense, backtesting with this software (even with 90% tick modelling quality - whatever that may mean) will NOT give proper backtesting results.

Use genuine/real tick data only to backtest any of your automatic trading systems, period!

This article only scratches the surface of backtesting automatic trading systems.  Other topics will be addressed in future contributions.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • bodytext
  • Bumpzee
  • Facebook
  • Furl
  • Mixx
  • NewsVine
  • Reddit
  • StumbleUpon
  • YahooMyWeb
  • Google
  • Blogosphere News
  • Propeller
  • Technorati
  • TwitThis
  • BlinkList
  • blogmarks
  • BlogMemes
  • E-mail this story to a friend!
  • feedmelinks
  • Live
  • MisterWong
  • Slashdot

Leave a Reply

You must be logged in to post a comment.