# Common Errors in Statistics and How to Avoid Them - Good P.I

**Download**(direct link)

**:**

**38**> 39 40 41 42 43 44 .. 90 >> Next

If the Bayes factor is equal to 1/10th, it means that the study results have decreased the relative odds assigned to the primary hypothesis by tenfold. For example, suppose the probability of the primary hypothesis with respect to the alternate hypothesis was high to begin with, say 9 to 1. A tenfold decrease would mean a change to odds of 9 to 10, a probability of 47%. A further independent study with a Bayes factor of 1/10th would mean a change to a posteriori odds of 9 to 100, less than 9%.

The minimum Bayes factor is calculated from the same information used to determine the p value, and it can easily be derived from standard analytic results. In the words of Goodman [2001], “If a statistical test is based on a Gaussian approximation, the strongest Bayes factor against the null hypothesis is exp(-Z2/2), where Z is the number of standard errors from the null value. If the log-likelihood of a model is reported, the minimum Bayes factor is simply the exponential of the difference between

CHAPTER 6 LIMITATIONS OF SOME MISCELLANEOUS STATISTICAL PROCEDURES 85

the log-likelihoods of two competing models (i.e., the ratio of their maximum likelihoods).”

The minimum Bayes factor does not involve a specific prior probability distribution, rather, it is a global minimum over all prior distributions. Bayarri and Berger [1998] and Berger and Sellke [1987]] provide a simple formula for the minimum Bayes factor in the situation where the prior probability distribution is symmetric and descending around the null value. This is -exp p ln(p), where p is the fixed-sample-size p value.

As Goodman [2001] notes, “even the strongest evidence against the null hypothesis does not lower its odds as much as the p-value magnitude might lead people to believe. More importantly, the minimum Bayes factor makes it clear that we cannot estimate the credibility of the null hypothesis without considering evidence outside the study.”

For example, while a p value of 0.01 is usually termed “highly significant,” it actually represents evidence for the primary hypothesis of somewhere between 1/25 and 1/8.29 Put another way, the relative odds of the primary hypothesis versus any alternative given a p value of 0.01 are at most 8-25 times lower than they were before the study. If one is going to claim that a hypothesis is highly unlikely (e.g., less than 5%), one must already have evidence outside the study that the prior probability of the hypothesis is no greater than 60%. Conversely, even weak evidence in support of a highly plausible relationship may be enough for an author to make a convincing case.

Two Caveats

1. Bayesian methods cannot be used in support of after-the-fact-hypotheses because, by definition, an after-the-fact hypothesis has zero a priori probability and, thus, by Bayes’ rule, zero a posteriori probability.

2. One hypothesis proving of greater predictive value than another in a given instance may be suggestive but is far from definitive in the absence of collateral evidence and proof of causal mechanisms. See, for example, Hodges [1987].

When Using Bayesian Methods

Do not use an arbitrary prior.

Never report a p value.

Incorporate potential losses in the decision. Report the Bayes factor.

29 See Table B.1, Goodman [2001].

86 PART II HYPOTHESIS TESTING AND ESTIMATION

META-ANALYSIS

“Meta-analysis should be viewed as an observational study of the evidence. The steps involved are similar to any other research undertaking: formulation of the problem to be addressed, collection and analysis of the data, and reporting of the results. Researchers should write in advance a detailed research protocol that clearly states the objectives, the hypotheses to be tested, the subgroups of interest, and the proposed methods and criteria for identifying and selecting relevant studies and extracting and analysing information” (Egger, Smith, and Phillips, i997).30

Too many studies end with inconclusive results because of the relatively small number of observations that were made. The researcher can’t quite reject the null hypothesis, but isn’t quite ready to embrace the null hypothesis, either. As we saw in Chapter i, a post hoc subgroup analysis can suggest an additional relationship, but the relationship cannot be subject to statistical test in the absence of additional data.

Meta-analysis is a set of techniques that allow us to combine the results of a series of small trials and observational studies. With the appropriate meta-analysis, we can, in theory, obtain more precise estimates of main effects, test a priori hypotheses about subgroups, and determine the number of observations needed for large-scale randomized trials.

By putting together all available data, meta-analyses are also better placed than individual trials to answer questions about whether an overall study result varies among subgroups—for example, among men and women, older and younger patients, or subjects with different degrees of severity of disease.

In performing a meta-analysis, we need to distinguish between observational studies and randomized trials.

**38**> 39 40 41 42 43 44 .. 90 >> Next