Out-of-sample Testing for Trading System Development

In the past couple of weeks I’ve received several emails asking about my back-testing approach, especially as it relates to out-of-sample testing.  If you’re not familiar with out-of-sample testing, let me first give a quick overview of what it’s all about. 

When a person is developing a trading system we all put on our best pair of rose-colored goggles and go to work.  After all, who wants to be a pessimist when our goal is to find a trading system that will make us lots of money.  As a result, it’s possible for a developer to over-optimize a system to the point where it’s really designed around the test data.  Optimizing is the process of refining your original system to provide the very best results.  And with our rosy outlook, it’s easy to get carried away with this.

For example, if my biggest losing trade happened to take place on a Wednesday, I could simply come up with a new rule (the fancy name is filter) that says “no trading on Wednesdays”.  The problem is that some rules may not make sense.  In fact, maybe (probably) that bad trade on a Wednesday was just a coincidence.  Some filters have real value, while some simply filter out coincidental outcomes.  The ones with real value are likely to be effective in the future, while the coincidences aren’t.

How then, does a developer avoid falling into the trap of over-optimization, or as Ernie Chan calls it, “data snooping”?  One answer is something called out-of-sample testing.  The way this works is that data is segmented into two or more parts, perhaps by date, or some other criteria.  The system is then developed and optimized on the first part (the in-sample).  Once the system is pretty much complete, the final strategy is then tested on the remaining data, which is the out-of-sample data.  If results are similar then you’ve probably been relatively honest with the data.  If they aren’t, you may have a curve-fit, “data-snooped” condition.

In short, out-of-sample data is any part of the data that isn’t part of the original testing, system development, and optimization.  That might include an extra data set, as described above, or it could include paper trading that happens in the future.  It can also include real time, real money trading that happens long after system development. 

The truth is we NEVER know for sure if a system will work in the future, but by doing adequate out-of-sample testing we can reduce the risk of a system that was fit to some historical data set.  In the next segment I’ll talk about several types of out-of-sample testing that I’ve done on the MultiStage Trading System.

Good Trading….