Download (direct link):
Limited Utility of Optimization Studies
As of this writing, data vendors cannot optimize entire portfolios of assets. Consequently, the best parameter set for each asset is not likely to be same as the best parameter set for a diversified portfolio. Furthermore, even if data vendors offering backtesting and optimization studies could determine the best parameter set for a diversified portfolio on any particular system, this parameter set probably would not retain its status as top performer if it were simultaneously traded in conjunction with another low- and/or negatively correlated system (e.g., trend-following and mean reversion systems).
Optimization: Avoiding the Pitfalls
The most obvious pitfall in system development in general, and in the optimization process specifically, is known as curve fitting. Curve fitting can be broken down into two basic subcategories: data curve fitting and parame-
MECHANICAL TRADING SYSTEMS
ter curve fitting. Data curve fitting occurs when system developers eliminate a portion of their historical data or intentionally reduce their historical data series at the study’s outset to filter out losing trades.
Avoidance of data curve fitting in the system development process can be achieved through strict adherence to an objective data history criterion for backtesting of the trading system. As stated earlier, such data histories should include all types of market environments: bullish, bearish, trending, and mean reverting. If data histories do not contain all types of market environments, either we need to expand the data set to include more history or, if there is a lack of history for a particular trading vehicle, we should examine an asset that displays a strong positive correlation to the asset that we anticipate trading and whose history does include all types of market environments.
Parameter curve fitting can be defined as the practice of the system developer adapting trade criteria parameters to match or fit in-sample data. For example, let us return to the simple and robust moving average crossover system examined in Chapter 3. Because this system contained only two parameters (9- and 26-day moving averages), we were moderately confident that its future performance would display a strong positive correlation to its backtested data history. But suppose that system developers were dissatisfied with the poor win/loss ratio of the two moving average crossovers. They decide to search through a list of various indicators until discovering one that improves the win/loss ratio without negatively impacting other performance measures. Two things have now occurred: (1) the system gets fewer trading signals, and (2) the desired result is achieved— the remaining signals generate more winning trades than losers. Then the developers reason that if the addition of one new parameter made the system more successful, imagine what two, three or four more could do.
Eventually the addition of parameters results in the developers sacrificing a very robust and moderately successful trading system in favor of one that works perfectly in the past and terribly in the future. Remember, the more parameters added to a trading system, the more closely that system’s criteria has been fit to the data. The closer the parameters have been fit to a particular data history, the less likely that these criteria will be able to filter out randomness within the data series.8
The easiest method to eliminate parameter curve fitting is to test and trade a system containing only one parameter. Unfortunately, although such a system would be the most robust imaginable, the likelihood of a one-parameter trading system being profitable is slim. Therefore, system developers must seek to balance the need for the fewest parameters possible in their systems while still maintaining optimal performance.
Aside from the establishment of an objective limit to the number of
System Development and Analysis
parameters that any particular trading system can contain, various methods can be employed to prevent parameter curve fitting. Such methods include backtesting of the system on a diversified portfolio of assets over a statistically significant data series (e.g., the portfolio employed in Chapter 3 over 10 years), along with the utilization of an out-of-sample data series.
Both data and parameter curve fitting are the result of a psychological trading problem that I have termed the perfect trader syndrome. The motivation is a combination of the need to be perfect and a desire to eliminate uncertainty and gain control over an uncertain future. Often the perfect trader syndrome leads us to seek the holy grail of trading and spend thousands of dollars on bogus systems promising winning trading percentages in excess of 90 percent. Remember, our goal in trading is not perfection; it is simply to manage risk well enough to enable us to consistently employ a trading methodology that will be successful over the long-term.