Download (direct link):
In our research efforts, the only statements we can make with God-like certainty are of the form “our conclusions fit the data.” The true nature of the real world is unknowable. We can speculate, but never conclude.
In our first advanced course in statistics, we read in the first chapter of Lehmann  that the “optimal” statistical procedure would depend on the losses associated with the various possible decisions. But on day one of our venture into the real world of practical applications, we were taught to ignore this principle.
At that time, the only computationally feasible statistical procedures were based on losses that were proportional to the square of the difference between estimated and actual values. No matter that the losses really might be proportional to the absolute value of those differences, or the cube, or the maximum over a certain range. Our options were limited by our ability to compute.
Computer technology has made a series of major advances in the past half century. What required days or weeks to calculate 40 years ago takes only milliseconds today. We can now pay serious attention to this long neglected facet of decision theory: the losses associated with the varying types of decision.
Suppose we are investigating a new drug: We gather data, perform a statistical analysis, and draw a conclusion. If chance alone is at work yielding exceptional values and we opt in favor of the new drug, we’ve made
20 PART I FOUNDATIONS
TABLE 2.2 Decision-Making Under Uncertainty
The Facts Our Decision
No difference. No difference. Drug is better. Type I error: Manufacturer wastes money developing ineffective drug.
Drug is better. Type II error: Manufacturer misses opportunity for profit. Public denied access to effective treatment.
TABLE 2.3 Decision-Making Under Uncertainty
The Facts Fears et al 's Decision
Compound not a Not a carcinogen. Compound a carcinogen.
carcinogen. Type I error: Manufacturer misses opportunity for profit. Public denied access to effective treatment.
Compound a carcinogen. Type II error: Patients die; families suffer; Manufacturer sued.
an error. We also make an error if we decide there is no difference and the new drug really is better. These decisions and the effects of making them are summarized in Table 2.2.
We distinguish the two types of error because they have the quite different implications described in Table 2.2. As a second example, Fears, Tarone, and Chu  use permutation methods to assess several standard screens for carcinogenicity. As shown in Table 2.3, their Type I error, a false positive, consists of labeling a relatively innocuous compound as carcinogenic. Such an action means economic loss for the manufacturer and the denial to the public of the compound’s benefits. Neither consequence is desirable. But a false negative, a Type II error, is much worse because it would mean exposing a large number of people to a potentially lethal compound.
What losses are associated with the decisions you will have to make? Specify them now before you begin.
The hypothesis/alternative duality is inadequate in most real-life situations. Consider the pressing problems of global warming and depletion of the ozone layer. We could collect and analyze yet another set of data and
CHAPTER 2 HYPOTHESES: THE WHY OF YOUR RESEARCH 21
TABLE 2.4 Effect of Global Warming
President's Decision on Emissions
No effect Burning of fossil fuels responsible
Reduce emissions Gather more data Change
Economy disrupted Sampling cost
Sampling cost Decline in quality
Decline in quality of of life
life (irreversible?) (irreversible?)
then, just as is done today, make one of three possible decisions: reduce emissions, leave emission standards alone, or sit on our hands and wait for more data to come in. Each decision has consequences as shown in Table 2.4.
As noted at the beginning of this chapter, it’s essential that we specify in advance the actions to be taken for each potential result. Always suspect are after-the-fact rationales that enable us to persist in a pattern of conduct despite evidence to the contrary. If no possible outcome of a study will be sufficient to change our mind, then perhaps we ought not undertake such a study in the first place.
Every research study involves multiple issues. Not only might we want to know whether a measurable, biologically (or medically, physically, or sociologically) significant effect takes place, but also what the size of the effect is and the extent to which the effect varies from instance to instance. We would also want to know what factors, if any, will modify the size of the effect or its duration.
We may not be able to address all these issues with a single data set. A preliminary experiment might tell us something about the possible existence of an effect, along with rough estimates of its size and variability. It is hoped that we will glean enough information to come up with doses, environmental conditions, and sample sizes to apply in collecting and evaluating the next data set. A list of possible decisions after the initial experiment includes “abandon this line of research,” “modify the environment and gather more data,” and “perform a large, tightly controlled, expensive set of trials.” Associated with each decision is a set of potential gains and losses. Common sense dictates that we construct a table similar to Table 2.2 or 2.3 before we launch a study.