Download (direct link):
Now that this study of the optimization results on the two moving average crossover system has shattered the myth of a single optimal parameter set, we can proceed to examine some basic principles of optimization. Despite the fact that there is no single optimal parameter set, there is nothing wrong with conducting an optimization study and then using it to identify various robust and distinct parameter sets. In analyzing such results, we seek to accomplish two goals: the elimination of suboptimal and nonsensical parameter sets (e.g., setting our shorter-term moving average to 2 days), and use of these results to identify various robust and distinct parameter sets.
The phenomenon known as profit spikes often makes the elimination of suboptimal parameter sets more difficult. Profit spikes are profitable parameter sets surrounded by those exhibiting consistently inferior performance. Pardo provides a sound mechanism to weed out such anomalies by averaging of profit spike parameter sets with those of neighboring parameter sets. This averaging technique helps both in eliminating aberrant performers that display a high probability of yielding suboptimal results going forward and in identifying more robust “hilltops” in performance. Such performance hilltops are superior performers that are adjacent to other similarly robust parameter sets.14
SYSTEM DEVELOPMENT PROCESS
The out-of-sample or walk-forward study is probably one of the most important aspects of the system development process. This procedure of setting aside a statistically significant (and most up-to-date15) portion of the data series to ensure that the system is behaving as forecasted is crucial to system developers because it enables us to test the system prior to committing actual funds.
System Development and Analysis
The most essential aspect of the out-of-sample data window is its integrity. Data integrity is defined here as the inability of our in-sample results to bleed through into our out-of-sample data. Although data integrity of the out-of-sample window might appear to be a given prerequisite, it never hurts to restate the obvious, especially since failure to adhere to this rule will necessarily compromise the value of all out-of-sample testing. Other, less critical characteristics of the out-of-ample window are that it generally should contain somewhere between 10 to 20 percent of the data displayed within the in-sample window.16
Although out-of-sample results never look exactly like those of our insample performance, there should be a strong positive correlation between the two data sets. If walk-forward performance yields results that are drastically different from those of in-sample data (e.g., postoptimization drawdowns exceeding 15 percent of in-sample), we probably should abandon the trading system. This seemingly drastic response to excessive drawdowns is prudent due to the nature of the optimization process.
Remember that the optimization process is one in which underperforming parameter sets are rejected in favor of robust ones. This process of filtering out poor performers can lead to an underestimation of the true risk entailed in employment of a particular trading system. There are only two ways to discover poor real-time performance of a trading system: the out-of-sample study and a real-time trading account. Consequently, the 15 percent rule seems a prudent alternative to the real-time failure of the trading system.17
Failure of the out-of-sample study is most commonly due either to data curve fitting (e.g., in-sample study was conducted on too few markets or too small of a data sample and as a result did not capture all types of market environments) or to parameter curve fitting.18 Other possibilities could be an unprecedented shift in market dynamics. Such a shift is exemplified by the unprecedented increase in volatility exhibited by the Nymex natural gas contract (see Figure 7.3).
For example, if the in-sample study included Nymex natural gas data from 1990 to 1999 and the out-of-sample study included the same contract’s data from the year 2000, there is a high probability that trading systems attempting to profit by fading unsustainable levels of volatility would have succeeded during our in-sample study and failed miserably in the out-ofsample study.
Traders and system developers alike must be ever mindful of paradigm shifts in market dynamics. Because markets are rarely stagnant, what worked in the past may not be robust enough to survive dramatic shifts in the dynamics of market behavior as exemplified by our study of natural gas in 2000. Despite our diligence in backtesting and forward (out-of-sample) testing of a wide variety of asset classes and market
MECHANICAL TRADING SYSTEMS
^91 11992 |1993 11994 |1995 11996 |1997 |1998 |1999 |2000 |2001 |2002 |2003
FIGURE 7.3 Rolling front-month Nymex natural gas futures.
Note: All trade summaries include $100 round-turn trade deductions for slippage and commissions. ©2004 CQG, Inc. All rights reserved worldwide.