Download (direct link):
In his book, Design, Testing and Optimization of Trading Systems, Robert Pardo discusses a phenomenon that he calls outlier curve fitting. Outlier curve fitting occurs when a single trade makes up a disproportionate percentage of a trading systemâ€™s profits (Pardo specifically warns against performance histories in which a single trade accounts for over 30 percent of a systemâ€™s profits.9) Certain types of trading systems are more susceptible to this problem than others. In general, because mean reversion trading systems exit with profits when the market reverts to its mean, outlier curve fitting is unlikely. By contrast, trend-following systems have a much greater tendency to contain single, disproportionately large profitable trades within their performance histories.
The problem of including such outliers within performance results is that a single trade could distort the systemâ€™s results, thereby leading us to believe that a system is robust and profitable, when in fact its success is due entirely to a single price shock event (e.g., the 1987 stock market crash). In extreme cases, such as those in which a single profit accounts for over 30 percent of a systemâ€™s profits, exclusion of the outlier is probably prudent. Although it is the simplest solution to this problem, in less extreme cases, exclusion probably is not the preferred response.
Elimination of outliers from the data series is usually justified with the myth that such occurrences are aberrations in market behavior and are unlikely to be repeated in the future. This belief in outliers as unrepeatable aberrations contradicts the whole premise behind trend trading, namely that trend-following systems enable participation in the amplified tails of the marketâ€™s distribution (see the discussion of stable Paretian distributions in Chapter 1). Intimately linked with this concept of outliers as aberrations is the myth that trend traders have a 50-50 chance of being caught on either side of the outlier. In reality, because outlier events typically are preceded
MECHANICAL TRADING SYSTEMS
by some type of technical breakout in the direction of the event, the odds of trend trader participation on the profitable side of the outlier are significantly greater than the 50 percent myth would lead us to believe. By the same token, a major drawback to the utilization of nondirectionally biased mean reversion systems is their greater propensity to suffering losses due to outlier events.
Finally, a subtler problem entailed in the exclusion of outliers from data history is that once the event has been removed, its reintroduction for price risk management purposes such as stress testing of the system (see Chapter 8) is often difficult to justify.
The Mechanics of Optimization
Now that we have outlined both the benefits and pitfalls of optimization studies, we can proceed with an examination of the preferred mechanics to employ in attempt to ensure the greatest utility of our optimization studies. To reiterate, the primary goal in our optimization studies is not pinpoint accuracy in forecasting of future performance. Instead, it is merely identification of historically robust parameters and parameter sets in the hope that such trade system criteria will continue to display positive correlations to past performance.
To review, consistency and data integrity in our optimization studies are crucial prerequisites in obtaining meaningful conclusions. I define â€śconsistencyâ€ť as the application of the same rules regarding entry, exit, and transaction costs throughout the entire backtested data series. Regarding data integrity, ideally our backtested data series should cover a diversified portfolio of assets and include all types of market environments: bullish, bearish, trending, choppy, neutral, and volatile.
Once this preliminary groundwork has been firmly established, we need to determine criteria for our choice of parameters and parameter sets. In choosing trading system parameters, we are seeking those that display the greatest propensity of enabling our participation in general principles of market behavior (e.g., mean reversion and/or trending). Regarding our testing of particular parameter sets in our optimization studies, the key here is inclusion of a broad and diverse group of parameter sets. There are two reasons for this:
1. Broad and diversified parameter sets improve our odds of identifying a robust set that will have a high probability of future positive correlations to our backtested history.10
2. Perhaps more important, the broader and more diversified our parameter sets, the greater the probability of our identification and subsequent elimination of suboptimal parameter sets.11
System Development and Analysis
Before examining the mechanics of optimization studies in detail, it is important to identify an objective measure of performance that will allow us to distinguish between more and less robust parameter sets and trading systems quickly and efficiently. Although readers are encouraged to experiment with the entire spectrum of performance measures, I feel that the profit to maximum drawdown ratio (P:MD) outlined is one of the most efficient tools for distinguishing between robust and suboptimal parameter sets and trading systems.