Download (direct link):
Before we initiate data collection, we must have a firm idea of what we will measure.
A second fundamental principle is also applicable to both experiments and surveys: Collect exact values whenever possible. Worry about grouping them in interval or discrete categories later.
A long-term study of buying patterns in New South Wales illustrates some of the problems caused by grouping prematurely. At the beginning of the study, the decision was made to group the incomes of survey subjects into categories, under $20,000, $20,000 to $30,000, and so forth. Six years of steady inflation later, the organizers of the study realized that all the categories had to be adjusted. An income of $21,000 at the start of the study would only purchase $18,000 worth of goods and housing at the end. The problem was that those surveyed toward the end had filled out forms with exactly the same income categories. Had income been tabulated to the nearest dollar, it would have been easy to correct for increases in the cost of living and convert all responses to the same scale.
26 PART I FOUNDATIONS
But the study designers hadn’t considered these issues. A precise and costly survey was now a matter of guesswork.
You can always group your results (and modify your groupings) after a study is completed. If after-the-fact grouping is a possibility, your design should state how the grouping will be determined; otherwise there will be the suspicion that you chose the grouping to obtain desired results.
Measuring devices differ widely both in what they measure and in the precision with which they measure it. As noted in the next section of this chapter, the greater the precision with which measurements are made, the smaller the sample size required to reduce both Type I and Type II errors below specific levels.
Before you rush out and purchase the most expensive and precise measuring instruments on the market, consider that the total cost C of an experimental procedure is S + nc, where n is the sample size and c is the cost per unit sampled.
The startup cost S includes the cost of the measuring device. c is made up of the cost of supplies and personnel costs. The latter includes not only the time spent on individual measurements but also the time spent in preparing and calibrating the instrument for use.
Less obvious factors in the selection of a measuring instrument include impact on the subject, reliability (personnel costs continue even when an instrument is down), and reusability in future trials. For example, one of the advantages of the latest technology for blood analysis is that less blood needs to be drawn from patients. Less blood means happier subjects, fewer withdrawals, and a smaller initial sample size.
While no scientist would dream of performing an experiment without first mastering all the techniques involved, an amazing number will blunder into the execution of large-scale and costly surveys without a preliminary study of all the collateral issues a survey entails.
We know of one institute that mailed out some 20,000 questionnaires (didn’t the post office just raise its rates again?) before discovering that half the addresses were in error and that the vast majority of the remainder were being discarded unopened before prospective participants had even read the “sales pitch.”
Fortunately, there are texts such as Bly [1990, 1996] that will tell you how to word a “sales pitch” and the optimal colors and graphics to use along with the wording. They will tell you what “hooks” to use on the envelope to ensure attention to the contents and what premiums to offer to increase participation.
CHAPTER 3 COLLECTING DATA 27
There are other textbooks such as Converse and Presser , Fowler and Fowler , and Schroeder  to assist you in wording questionnaires and in pretesting questions for ambiguity before you begin. We have only two paragraphs of caution to offer:
1. Be sure your questions don’t reveal the purpose of your study; otherwise, respondents shape their answers to what they perceive to be your needs. Contrast “how do you feel about compulsory pregnancy?” with “how do you feel about abortions?”
2. With populations ever more heterogeneous, questions that work with some ethnic groups may repulse others (see, for example, Choi ).
Recommended are web-based surveys with initial solicitation by mail (letter or post card) and email. Not only are both costs and time to completion cut dramatically, but also the proportion of missing data and incomplete forms is substantially reduced. Moreover, web-based surveys are easier to monitor, and forms may be modified on the fly. Web-based entry also offers the possibility of displaying the individual’s prior responses during follow-up surveys.
Three other precautions can help ensure the success of your survey:
1. Award premiums only for fully completed forms.
2. Continuously tabulate and monitor submissions; don’t wait to be surprised.