Download (direct link):
Confounding and selection bias can easily distort the findings from observational studies. Egger et al. [i998] note, An important criterion supporting causality of associations is a dose-response relation. In occupational epidemiology the quest to show such an association can lead to very different groups of employees being compared. In a meta-analysis that examined the link between exposure to formaldehyde and cancer, funeral directors and embalmers (high exposure) were compared with anatomists and pathologists (intermediate to high exposure) and with industrial workers (low to high exposure, depending on job assignment). There is a striking deficit of deaths from lung cancer among anatomists and pathologists [standardized mortality ratio 33 (95% confidence interval 22 to 47)], which is most likely to be due to a lower prevalence of smoking among
30 Reprinted with permission from the BMJ Publishing Group.
CHAPTER 6 LIMITATIONS OF SOME MISCELLANEOUS STATISTICAL PROCEDURES 87
this group. In this situation few would argue that formaldehyde protects against lung cancer. In other instances, however, such selection bias may be less obvious.31
On the other hand, much may be gained by a careful examination of possible sources of heterogeneity between the results from observational studies.
Publication and selection bias also plague the meta-analysis of completely randomized trials. Inconclusive or negative results seldom appear in print (Gotzsche, 1987; Chalmers et al., 1990; Easterbrook et al., 1991) and are unlikely even to be submitted for publication. One cant analyze what one doesnt know about.
Similarly, the decision as to which studies to incorporate can dramatically affect the results. Meta-analyses of the same issue may reach opposite conclusions, as shown by assessments of low-molecular-weight heparin in the prevention of perioperative thrombosis (Nurmohamed et al., 1992; Leizorovicz et al., 1992) and of second line antirheumatic drugs in the treatment of rheumatoid arthritis (Felson et al., 1990; Gotzsche et al., 1992). Meta-analyses showing benefit of statistical significance and clinical importance have been contradicted later by large randomized trials (Egger et al., 1997).
Where there are substantial differences between the different studies incorporated in a meta-analysis (their subjects or their environments), or substantial quantitative differences in the results from the different trials, a single overall summary estimate of treatment benefit has little practical applicability (Horowitz, 1995). Any analysis that ignores this heterogeneity is clinically misleading and scientifically naive (Thompson, 1994). Heterogeneity should be scrutinized, with an attempt to explain it (Bailey, 1987; Berkey et al., 1995; Chalmers, 1991; Victor, 1995).
Bayesian methods can be effective in meta-analyses; see, for example, Mosteller and Chalmers . In such situations the parameters of various trials are considered to be random samples from a distribution of trial parameters. The parameters of this higher-level distribution are called hyperparameters, and they also have distributions. The model is called hierarchical. The extent to which the various trials reinforce each other is determined by the data. If the trials are very similar, the variation of the hyperparameters will be small, and the analysis will be very close to a classical meta-analysis. If the trials do not reinforce each other, the conclusions of the hierarchical Bayesian analysis will show a very high variance in the results.
31 Reprinted with permission from the BMJ Publishing Group.
88 PART II HYPOTHESIS TESTING AND ESTIMATION
A hierarchical Bayesian analysis avoids the necessity of a prior decision as to whether or not the trials can be combined; the extent of the combination is determined purely by the data. This does not come for free; in contrast to the meta-analyses discussed above, all the original data (or at least the sufficient statistics) must be available for inclusion in the hierarchical model. The Bayesian method is also vulnerable to all the selection bias issues discussed above.
Guidelines For a Meta-Analysis
A detailed research protocol for the meta-analysis should be prepared in advance. Criteria for inclusion and statistical method employed should be documented in the materials and methods section of the subsequent report.
Meta-analysis should be restricted to randomized controlled trials.
Heterogeneity in the trial results should be documented and explained.
Do not attempt to compare treatments investigated in unrelated trials. (Suppose, by way of a counterexample, that Old were given as always to low-risk patients in one set of trials, while New was given to high-risk patients in another.)
Individual patient data, rather than published summary statistics, often are required for meaningful subgroup analyses. This is a major reason why we favor the modern trend of journals to insist that all data reported on within their pages be made available by website to all investigators.