Your browser (Internet Explorer 6) is out of date. It has known security flaws and may not display all features of this and other websites. Learn how to update your browser.

System Lab: Trading System “B” Backtest

In a prior post I discussed the performance of a system I’ve been trading live for over two years (and whose algobot is running as we speak).  For the purposes of this discussion, I’ll refer to this system as “System A”.

As mentioned in the post, System A performs well but has a known weakness: It is very susceptible to Black Swans (outlier market shocks) and can generate some uncomfortable drawdowns when they hit.  The system usually recovers from these drawdowns reasonably fast, but they’re certainly no fun to sit through when you’re trading with real $$$.

These Black Swans seem to be hitting the market with increasing frequency, so after surviving the most recent crash (August 2011), I started experimenting with a variation of System A (we’ll call it “System B”) that temporarily moves to the sidelines the moment it senses a crash is imminent.

The backtesting results of System B are encouraging.  Here is the equity curve from an 8-year trading simulation run (click to zoom):

In this particular run, the system generated 14,637 trades (about 10 trades per day) and had an annualized return of 40% with a -6% maximum peak-to-trough drawdown.  This translates into a Calmar Ratio of 7:1, which isn’t bad considering the events of the past four years.  Note that these returns and drawdowns are at 1:1 leverage so – depending on your risk tolerance – you could ratchet up them to a much higher absolute level.

The Buy and Hold return during the same 8yr period was +19% with a -51% max drawdown (vs +40%/-6%), so on both an absolute and risk adjusted basis System B’s returns were quite respectable.

I’m particularly happy with the performance during the 2008 meltdown, which sent the S&P 500 down almost -60%.   System B chugged right along with only a -5% drawdown, which it recovered from in less than a month.  It also cleared +35% for calendar 2008.

Optimization Details

The system was optimized using GA and Monte Carlo sampling techniques on the most recent 4 years of data, and the above equity curve was generated by running the final winning chromosome over the entire 8-year data set.

Thus, while the above results are at least partially in-sample, they were not curve fit to the particular path of the 8 year dataset.  Also, the trades / parameters ratio is also quite high (over 3000:1), which helps one have more confidence that the results are not spurious.

Plan of Attack

The next step for me is to code the actual trading bot that will connect to my brokerage and run the strategy in real time.  Once the bot is written, I’ll test it for a few days using my broker’s simulator mode to make sure there’s no bugs, then go live with a small account and scale in my equity over time as the system (hopefully) performs as expected.  I’ll be posting blog updates on my progress as things proceed.  I’m cautiously optimistic;  Fingers crossed!



  • Have you tried to take into account the difference in mean reversion-tendencies across different symbols? For example, I’ve noticed that for EOD data some stocks are more “trending” than other which would make it hard to optimize a mean reversion strategy on them. I can’t see a reason why all stock would be mean reverting intraday…


    February 12, 2012

  • The system is adaptive in the sense that it accounts for the individual behavior of each stock (but not exactly in the way you’re describing). I’d agree that just as stocks never trend all the time, neither will they RTM consistently either. The trick of course is to know when they are about to change their behavior; I view that as more or less an intractable problem, along the lines of accurately picking tops or bottoms.



    February 12, 2012

  • Mkey, here’s a cool idea: since the topology (don’t know if that the right word, in layman’s terms it should be landscape ) of any n-dimensional parameter space changes rapidly as you go OOS, improved performance in prediction could be achieved by aggregating the prediction from several solutions around the global maximum and perhaps also around each local as well. We’re essentially hedging our bet that the performance IS correlates in a meaningful way with the OOS period by saying that the correlation will probably so small that it’s best not to put all eggs in a basket.

    I think this is similar to what Quantitative Investment Management does, create a lot of good strategies and trade based on the overall score. I have tested this with some encouraging results and would like to hear you opinion.


    March 6, 2012

  • Hi Emil,

    Thanks for another thoughtful post. I came across a paper a few years ago that described something similar; The researchers were using NNs to generate trading signals, but instead of using a single large NN, they had a network topology such that they aggregated signals from multiple NNs of smaller dimensions. I think they called them “consensus networks” and I can’t remember if they trained the smaller NNs individually or collectively (it seems to me if you trained them simultaneously your risk of curve fitting increases greatly, as we’re talking huge degrees of freedom), but it was certainly an interesting idea that could be applied to non-NN implementations.

    Its certainly no secret that diversifying across (uncorrelated) systems as well as instruments is good for your soul (and Calmar ratio), and is one of the few free lunches in trading. The trick of course is finding those systems ;-)



    March 13, 2012