Download (direct link):
Kepler was able to formulate his laws only because (1) Tycho Brahe had made over 30 years of precise (for the time) astronomical observations and (2) Kepler married Brahes daughter and, thus, gained access to his data.
Permutation tests are often lauded erroneously in the literature as assumption-free panaceas. Nothing could be further from the truth.
Permutation tests only yield exact significance levels if the labels on the observations are weakly exchangeable under the null hypothesis. Thus, they cannot be successfully applied to the coefficients in a multivariate regression.
On the other hand, if the observations are weakly exchangeable under the null hypothesis, then permutation tests are the method of choice for k-sample comparisons, multi-factor experimental designs, and contingency tables, whenever there are 12 or less observations in each subsample. Moreover, permutation methods can be used both to test hypotheses and to obtain interval estimates of parameters.
CHAPTER 6 LIMITATIONS OF SOME MISCELLANEOUS STATISTICAL PROCEDURES 89
TO LEARN MORE
Potential flaws in the bootstrap approach are considered by Schenker , Wu , Diciccio and Romano , Efron [1988, 1992], Knight , and Gine and Zinn . Canty et al.  provide a set of diagnostics for detecting and dealing with potential error sources.
Berry and Stangl  include a collection of case studies in Bayesian biostatistics. Kass and Raftery  discuss the problem of establishing priors along with a set of practical examples. The Bayes factor can be used as a test statistic; see Good .
For more on the strengths and limitations of meta-analysis, see Egger and Smith , Egger, Smith, and Phillips , Smith, Egger, and Phillips , Smith and Egger , Gillett , Gurevitch and Hedges , Horowitz , and Smeeth, Haines, and Ebrahim . To learn about the appropriate statistical procedures, see Adams, Gurevitch, and Rosenberg , Berlin et al. , and Hedges and Olkin .
For practical, worked-through examples of hierarchical Bayesian analysis, see Harley and Myers  and Su, Adkison, and Van Alen . Theoretical development may be found in Mosteller and Chalmers  and in Carlin and Louis .
The lack of access to the raw data underlying published studies is a matter of ongoing concern. See Moher et al. , Eysenbach and Sa , and Hutchon .
Permutation methods and their applications are described in Good , Manley , Mielke and Berry , and Pesarin . For a description of some robust permutation tests, see Lambert  and Maritz . Berger  reviews the pros and cons of permutation tests.
90 PART II HYPOTHESIS TESTING AND ESTIMATION
Reporting Your Results
Cut out the appropriate part of the computer output and paste it onto the draft of the paper. George Dyke (tongue in cheek) .
The FOCUS OF THIS CHAPTER IS ON WHAT to report and how to report it. Reportable elements include the experimental design and its objectives, its analysis, and the sources and amounts of missing data. Guidelines for table construction are provided. The bootstrap is proposed as an alternative to the standard error as a measure of precision. The value and limitations of p values and confidence intervals are summarized. Practical significance is distinguished from statistical significance and induction from deduction.
Few experimenters fail to list number of subjects, doses administered, and dose intervals in their reports. But many fail to provide the details of power and sample size calculations. Feng et al.  found that such careless investigators also report a higher proportion of nonsignificant intervention effects, indicating underpowered studies.
Too often inadequate attention is given to describing treatment allocation and the ones who got away. We consider both topics in what follows.
Allocation details should be fully described in your reports including dictated allocation versus allocation discretion, randomization, advance preparation of the allocation sequence, allocation concealment, fixed versus varying allocation proportions, restricted randomization, masking, simulta-
1 This material in this section relies heavily on a personal communication from Vance W. Berger and Costas A. Christophi.
CHAPTER 7 REPORTING YOUR RESULTS 91
neous versus sequential randomization, enrollment discretion, and the possibility of intent to treat.
Allocation discretion may be available to the investigator, the patient, both, or neither (dictated allocation). Were investigators permitted to assign treatment based on patient characteristics? Could patients select their own treatment from among a given set of choices?
Was actual (not virtual, quasi-, or pseudo-) randomization employed? Was the allocation sequence predictable? (For example, patients with even accession numbers or patients with odd accession numbers receive the active treatment; the others receive the control.)