Quantitative Trading: Part 6 - Backtesting

This is a huge topic. Backtesting is the idea of testing your very specific set of rules across real market data in the past. In other words, you simulate the trades as though you were going back in a time machine and really trading it. The problem is that you're not really going back in time. Because you are merely looking back, it's fraught with potential problems.

Impossible Trades:

This is a problem that is easily avoided, yet it's very real. It's possible in your backtesting rules, especially if you are new to writing code, to use data that would not have been available, or that you could not have possibly known at the time. Try the following trading system:

Buy = High > Ref(High,-1); // buy if the high is higher than the high of one day ago

BuyPrice = High;

Sell = High > Low; //sell same day

SellPrice = Low;

This system will rake in the money. In fact it never has a losing trade. It's amazing. The only problem is it's impossible. This might seem obvious, but you couldn't have known what the high of the day was going to be before the day is over. Therefore, the "BuyPrice" could not have been equal to the high of the day. Yet, that's what I wrote. This is a super obvious example, and most are not this simple, but it can happen. Be careful.

What to do: The best way to deal with this is to develop the system, then look at the actual trades on the chart. Trace them through and see if everything else makes sense. Once you have optimized your system and tested against out-of-sample data, the next step is to paper trade in real time. If you do some simulated trades in real time you will quickly discover that you don't have the data to make the trade.

Survivorship Bias:

Investopedia explains Survivorship Bias... "For example, a mutual fund company's selection of funds today will include only those that have been successful in the past. Many losing funds are closed and merged into other funds to hide poor performance. This is an important issue to take into account when analyzing past performance."

In other words, this is the practice of NOT including data from failed companies in your dataset. I commented in a past post that I have tested this and found that it hasn't been significant in shorter term holding periods (intraday to a few days). If you run a system with trend trades lasting a long time, this could be a problem.

What to do: The best way to handle this is to get a complete dataset, including the delisted stocks. If your data doesn't include Washington Mutual or Lehman Brothers, it doesn't have delisted stocks.

Curve Fitting:

In my opinion, this is one of the most significant areas of danger. It is very possible, almost likely, that system developer who are starting out will "over-optimize". In other words, they will develop a system and continue to modify and tweak it until it has satisfactory results. The problem is that the system is fit to the specific set of data being tested. It likely won't work in the future under a fresh data set. If your system looks like, "buy stock on the first Tuesday of the month, that have broken into a 10 day high, begin with the letter 'a', have three days of falling volume, a positive MACD crossover, RSI(10) below 50... "... well, that might be a problem. Generally speaking, the best systems are pretty simple.

What to do: Perhaps the best way to deal with curve fitting is to make sure and test against multiple data sets. If you test against a couple of years of data, and then optimize, and finally test against "out-of-sample" data, you will be pretty safe. "Out-of-sample" data is simply data that wasn't part of your initial testing data set. Save some data for out-of-sample testing. It's important. It will be the best way to avoid curve fitting. Aside from that, keep things very simple, and make your rules logical.

Slippage:

It's common in backtesting to forget slippage. Slippage is the price you have to pay for entering and exiting trades. It's also called "transaction costs". It comes in a couple of forms. First, you have the difference in the Bid and Ask prices... the "spread". When I test I assume the midpoint for stocks, and usually for options. But this isn't always what you get.

Second, there are commissions. It's often easier to test and compare results when commissions aren't included. In fact, when I started testing I always left them out. Later I started including them because it got really simple with Amibroker, and it's just more accurate.

What to do: A safe assumption might be to add something in for the bid-ask spread. For highly liquid issues this is a minor amount - usually less than a penny; but if you are testing options or futures or something else with a large spread you should include something reasonable. For commissions, my advice is include them with your testing.

It's not emotional:

One of the challenges with backtesting is that it's not real. That can never be fixed. In fact, it's actually the whole point. I'd much rather test and tweak when I'm not using my hard earned money. But when the real money gets involved somehow there is an emotional mind shift. It's REAL MONEY! To make matters worse, you have time to think about it. When I backtest I can test thousands of trades in a few minutes. But when I start trading for real I make a trade and then have 3 or 4 days to think about it. All kinds of interesting ideas creep in.

What to do: This purely a trading psychology challenge. Traders need to develop a steely discipline around following their rules. The moment you start looking at a trade and decide to help your system repeat after me: "I will follow the rules." Do whatever it takes to trick yourself into following the system.

I found that it was very helpful for me to only trade after hours. When prices are not moving and blinking and everything is calm, I can easily just enter my trades according to the rules. I've never been a big intraday trader, but I'm sure I would be challenged. Over time things do get easier. Most traders will develop a rhythm wherein you just enter trades every day. It's extremely important to commit to never trading anything that isn't tested. If your idea is so good, make sure that it works against test data.

Liquidity:

I have a system that will turn $100,000 into literally billions of dollars in 10 years. Everything has been accounted for. Commissions are included, it works equally well against out-of-sample data, it's simple, and it doesn't work. The problem is that it trades stocks that are not liquid enough to support the trade sizes. Unless you specifically write liquidity requirements into your code, it may not be included in your testing system.

What to do: This is simple to solve by simply writing liquidity requirements into your code. The way I handle it is to multiply volume and price, and do that for a 90 day average. That tells me how many dollars are trading in that stock on average for 90 days. Generally I never want to trade more than 1% or 2% of that liquidity. One other thing to keep in mind is that if your system works your account will grow. Be sure your system will scale up to support larger account sizes and still not violate your liquidity rules.

Wow! That's a long post and I'm tired of typing.

Good trading...