# Common Errors in Statistics and How to Avoid Them - Good P.I

**Download**(direct link)

**:**

**12**> 13 14 15 16 17 18 .. 90 >> Next

For example, in clinical trials of a drug we might begin with some animal experiments, then progress to Phase I clinical trials in which, with the emphasis on safety, we look for the maximum tolerable dose. Phase I trials generally involve only a small number of subjects and a one-time or short-term intervention. An extended period of several months may be used for follow-up purposes. If no adverse effects are observed, we might decide to go ahead with a further or Phase II set of trials in the clinic in

22 PART I FOUNDATIONS

which our objective is to determine the minimum effective dose. Obviously, if the minimum effective dose is greater than the maximum tolerable dose, or if some dangerous side effects are observed that we didn’t observe in the first set of trials, we’ll abandon the drug and go on to some other research project. But if the signs are favorable, then and only then will we go to a set of Phase III trials involving a large number of subjects observed over an extended time period. Then, and only then, will we hope to get the answers to all our research questions.

Before you begin, list all the consequences of a study and all the actions you might take. Persist only if you can add to existing knowledge.

TO LEARN MORE

For more thorough accounts of decision theory, the interested reader is directed to Berger [1986], Blyth [1970], Cox [1958], DeGroot [1970], and Lehmann [1986]. For an applied perspective, see Clemen [1991], Berry [1995], and Sox et al. [1988].

Over 300 references warning of the misuse of null hypothesis testing can be accessed online at the URL http://www.cnr.colostate.edu/ ~anderson/thompson1.html. Alas, the majority of these warnings are ill informed, stressing errors that will not arise if you proceed as we recommend and place the emphasis on the why, not the what, of statistical procedures. Use statistics as a guide to decision making rather than a mandate.

Neyman and Pearson [1933] first formulated the problem of hypothesis testing in terms of two types of error. Extensions and analyses of their approach are given by Lehmann [1986] and Mayo [1996]. For more work along the lines proposed here, see Selike, Bayarri, and Berger

[2001].

Clarity in hypothesis formulation is essential; ambiguity can only yield controversy; see, for example, Kaplan [2001].

CHAPTER 2 HYPOTHESES: THE WHY OF YOUR RESEARCH 23

Chapter 3

Collecting Data

GIGO Garbage in, garbage out.

“Fancy statistical methods will not rescue garbage data.” Course notes of Raymond J. Carroll [2001].

The vast MAJORITY OF ERRORS IN STATISTICS—AND, not incidentally, in most human endeavors—arise from a reluctance (or even an inability) to plan. Some demon (or demonic manager) seems to be urging us to cross the street before we’ve had the opportunity to look both ways. Even on those rare occasions when we do design an experiment, we seem more obsessed with the mechanics than with the concepts that underlie it.

In this chapter we review the fundamental concepts of experimental design, the determination of sample size, the assumptions that underlie most statistical procedures, and the precautions necessary to ensure that they are satisfied and that the data you collect will be representative of the population as a whole. We do not intend to replace a text on experiment or survey design, but to supplement it, providing examples and solutions that are often neglected in courses on the subject.

preparation

The first step in data collection is to have a clear, preferably written statement of your objectives. In accordance with Chapter 1, you will have defined the population or populations from which you intend to sample and have identified the characteristics of these populations you wish to investigate.

You developed one or more well-formulated hypotheses (the topic of Chapter 2) and have some idea of the risks you will incur should your analysis of the collected data prove to be erroneous. You will need to

CHAPTER 3 COLLECTING DATA 25

decide what you wish to observe and measure and how you will go about observing it.

Good practice is to draft the analysis section of your final report based on the conclusions you would like to make. What information do you need to justify these conclusions? All such information must be collected.

The next section is devoted to the choice of measuring devices, followed by sections on determining sample size and preventive steps to ensure your samples will be analyzable by statistical methods.

MEASURING DEVICES

Know what you want to measure. Collect exact values whenever possible.

Know what you want to measure. Will you measure an endpoint such as death or measure a surrogate such as the presence of HIV antibodies? The regression slope describing the change in systolic blood pressure (in mm Hg) per 100 mg of calcium intake is strongly influenced by the approach used for assessing the amount of calcium consumed (Cappuccio et al., 1995). The association is small and only marginally significant with diet histories (slope -0.01 (-0.003 to -0.016)) but large and highly significant when food frequency questionnaires are used (-0.15 (-0.11 to -0.19)). With studies using 24-hour recall, an intermediate result emerges (-0.06 (-0.09 to -0.03)). Diet histories assess patterns of usual intake over long periods of time and require an extensive interview with a nutritionist, whereas 24-hour recall, and food frequency questionnaires are simpler methods that reflect current consumption (Block, 1982).

**12**> 13 14 15 16 17 18 .. 90 >> Next