How to Build and Trading System: Part 10 – Out of sample testing

The next post in this series is describing the next step in the back-testing process. One of the great concerns with back-testing in general is that there might be conditions that existed in the past that they looked good, but they aren’t necessarily hear to say. A bigger issue is that during the “optimization” process we might fine tune the system parameters to fit conditions that may not continue.

Out-of-sample testing is designed to flatten this risk to some degree. The concept is simple: During back-testing I won’t use all of my data, but I will keep some back to be tested after all optimization and final changes have been made. The data that is left untested during initial development is called “out-of-sample”, and it can be any cut on the data. It can be a date range (as I am showing in this example), or another sliver of the data. It can be half of the companies, different markets, etc.

As mentioned, in this case our out-of-sample data was a date range. Specifically, it was market data from 1/1/2005 to 12/31/2010. This will compare to our initial tests between 1/1/1999 and 12/21/2004. I ran the exact system we had evolved to in our last post against these new dates.

As you can see, the returns are quite different from out last results. Both raw returns and risk adjusted returns are much lower than the previous data. The system is the same, but the CAGR is nearly one-fifth of the prior returns. Think less about the actual numbers, and more about the huge difference in returns.

The equity curve looks okay, but there are many flat spots. If you started this approach in early 2006, would you still be persisting after 18 month during the run up in late 2007? That would be great, but many of us might have given up by then. Imagine if I had run the system against only the 2006 period. What would your impression be then.

In some ways, I‘ve done a disservice in developing a system that looks quite profitable. It’s appealing, I know, but it’s important to please heed the warning that I don’t recommend using this system. That are clearly down-swings you can see, and there is further downside you can’t see; just don’t do it.

One of the items, for example, that you can’t see is the transaction costs. In my case, I pay one cent per share for a buy/sell trade (half cent per side). It also doesn't show slippage. The point is, take it for what it is... just an educational series explaining how I go about building a trading system. Don't use it!

Good Trading…