Out-of-sample Testing for the MultiStage Trading System

In the last post I talked about out-of-sample back-testing during trading system development.  As a follow-up I wanted to describe the different out-of-sample (OOS) tests I’ve used while developing the MultiStage Trading System.

I described basic OOS tests in my last post, which might consist of simply dividing test data into different date ranges.  In order to think about applying this to MSv8 (and other versions) it’s important to remember that this trading system has evolved over several years.  I’m currently on version 9, although v.8 is the current published version.

When I first started prototyping this idea I used a subset of the data I had, both by time, and by symbols.  Specifically, I started testing on the NAS100.  This symbol set seemed to reflect the volatility of the market better than the SP500, and it’s small size made it very quick to do optimizations. As far as time segmentation goes, much of the early development was done in 2007, and for a long time I used up to 2002 as my in-sample data. 

The early tests were straightforward out-of-sample tests.  I tested using data from 2003 to 2007, and used all of the symbols in the data base.  At that time I had only about 6000, but it’s since expanded to about 9000.  There was a little debugging to be done after this, so you have to be very careful not to change the way you are filtering data to remain bias-free.

Keeping in mind that OOS testing is really any testing that isn’t done on the original data set, that wasn’t the end of my OOS tests.  I wanted to trade real money, of course, but real money is, well… real money.  I’ve been developing trading systems for many years, and for many years prior to 2007, but I’m not about to part with my hard-earned money without some very cautious observation.  Consequently, my first use of the the MultiStage Trading System (version 1) was simply paper trading.  I traded in a sample account for a couple of months.  Then I started with VERY small position sizes.  In some cases just a few hundred dollars. 

Both paper trading, and these small “test” positions could both be viewed as further OOS testing.  But there’s more: This system has evolved over time.  Somewhere around mid-2008 v5 and then v6 were born.  In each case, I took the new versions and compared them with the old data.  Then I tested them on the OOS data, which was new since the publishing of the prior system.  Also, in each case I went back to the paper-trading model, and then the small position sizes before jumping in with both feet.  In 2009 v7 was developed, and in 2011 I began using v8.  In each case I followed this pattern.

At this point, there has been several forward-looking years of real-time, real-money trading activity, which pretty much puts to rest any concern of data-snooping or curve fitting.  The main point to see is that there are several different safeguards you can take during system development to be sure you are being objective with the data.

Good Trading…