Download (direct link):
They move without a forwarding address before a scheduled follow-up. Or simply don’t bother to show up for an appointment. We lost 30% of the patients in the follow-ups to a lifesaving cardiac procedure. (We can’t imagine not going in to see our surgeon, but then we guess we’re not typical.)
The key to a successful research program is to plan for such dropouts in advance and to start the trials with some multiple of the number required to achieve a given power and significance level.
1 Chapters 4-7 have more information on the use of the bootstrap and its limitations.
30 PART I FOUNDATIONS
FIGURE 3.1 A Typical Clinical Trial. Dropouts and noncompliant patients occur at every stage. Reprinted from the Manager’s Guide to Design and Conduct of Clinical Trials with the permission of John Wiley & Sons, Inc.
An analysis of those who did not respond to a survey or a treatment can sometimes be as informative as, or more informative than, the survey itself. See, for example, Mangel and Samaniego  as well as the sections on the Behrens-Fisher problem and on the premature drawing of conclusions in Chapter 5. Be sure to incorporate provisions for sampling nonresponders in your sample design and in your budget.
Sample from the Right Population
Be sure you are sampling from the population as a whole rather than from an unrepresentative subset of the population. The most famous blunder along these lines was basing the forecast of Landon over Roosevelt in the 1936 U.S. presidential election on a telephone survey; those who owned a
CHAPTER 3 COLLECTING DATA 31
telephone and responded to the survey favored Landon; those who voted did not. An economic study may be flawed because we have overlooked the homeless,2 and an astrophysical study may be flawed because of overlooking galaxies whose central surface brightness was very low.3
Most statistical procedures rely on two fundamental assumptions: that the observations are independent of one another and that they are identically distributed. If your methods of collection fail to honor these assumptions, then your analysis must fail also.
To ensure the independence of responses in a return-by-mail or return-by-web survey, no more than one form per household should be accepted. If a comparison of the responses within a household is desired, then the members of the household should be interviewed separately, outside of each other’s hearing, and with no opportunity to discuss the survey in between. People care what other people think and when asked about an emotionally charged topic may or may not tell the truth. In fact, they are unlikely to tell the truth if they feel that others may overhear or somehow learn of their responses.
To ensure independence of the observations in an experiment, determine in advance what constitutes the experimental unit.
In the majority of cases, the unit is obvious: One planet means one position in space, one container of gas means one volume and pressure to be recorded, and one runner on one fixed race course means one elapsed time.
In a clinical trial, each individual patient corresponds to a single set of observations or does she? Suppose we are testing the effects of a topical ointment on pinkeye. Is each eye a separate experimental unit, or each patient?
It is common in toxicology to examine a large number of slides. But regardless of how many are examined in the search for mutagenic and toxic effects, if all slides come from a single treated animal, then the total size of the sample is one.
We may be concerned with the possible effects a new drug might have on a pregnant woman and, as critically, on her children. In our preliminary tests, we’ll be working with mice. Is each fetus in the litter a separate experimental unit, or each mother?
2 City of New York v. Dept of Commerce, 822 F. Supp. 906 (E.D.N.Y., 1993).
3 Bothun [1998, p. 249].
32 PART I FOUNDATIONS
If the mother is the one treated with the drug, then the mother is the experimental unit, not the fetus. A litter of six or seven corresponds only to a sample of size one.
As for the topical ointment, while more precise results might be obtained by treating only one eye with the new ointment and recording the subsequent difference in appearance between the treated and untreated eyes, each patient still yields only one observation, not two.
Identically Distributed Observations
If you change measuring instruments during a study or change observers, then you will have introduced an additional source of variation and the resulting observations will not be identically distributed.
The same problems will arise if you discover during the course of a study (as is often the case) that a precise measuring instrument is no longer calibrated and readings have drifted. To forestall this, any measuring instrument should have been exposed to an extensive burn-in before the start of a set of experiments and should be recalibrated as frequently as the results of the burn-in or pre-study period dictate.