# Common Errors in Statistics and How to Avoid Them - Good P.I

**Download**(direct link)

**:**

**74**> 75 76 77 78 79 80 .. 90 >> Next

180 APPENDIX B EXCESS ERROR ESTIMATION IN FORWARD LOGISTIC REGRESSION

excess error over the many training samples that Gregory might have observed and, therefore, over many realizations of this prediction rule. Because rcross, rjack, rboot average over many realizations, they are, strictly speaking, estimates of the expected excess error. Gregory, however, would much rather know the excess error of his particular realization.

It is perhaps unfair to think of qcross, rjack, rw as estimators of the excess error. A simple analogy may be helpful. Suppose X is an observation from the distribution F%, and T(X) estimates Z The bias is the expected difference E[T(X) - Z] and is analogous to the expected excess error. The difference T(X) - Z is analogous to the excess error. Getting a good estimate of the bias is sometimes possible, but getting a good estimate of the difference T(X) - Z would be equivalent to knowing Z

In the simulations, the underlying model was the logistic model that assumes x1 = (t1, y1),. . . , xn = (tn, yn) are independent and identically distributed such that y{ conditional on ti is Bernoulli with probability of success 0(ti), where

logit 0 (ti) = b 0 + ti b,

(4.1)

where ti = (ti1,. . ., tp) is p-variate normal with zero mean and a specified covariance structure S.

I performed two sets of simulations. In the first set (simulations 1.1,

1.2, 1.3) I let the sample sizes be, respectively, n = 20, 40, 60; the dimension of ti be p = 4; and

S =

(1 0 0 0'

0 0 t 0

0 t 1 0

V 0 0 0 1,

be = 0, b =

(^ 2 0 0

(4.2)

where t = 0.80. We would expect a good prediction rule to choose variables t1 and t2, and due to the correlation between variables t2 and t3, a prediction rule choosing t1 and t3 would probably not be too bad. In the second set of simulations (simulations 2.1, 2.2, 2.3, the sample sizes were again n = 20, 40, 60; the dimension of ti was increased to p = 6; and

S =

(1 0 0 0 0 0' (1'

0 1 0 0 0 0 1

0 0 1 0 0 0 , be = 0, b = 1

0 0 0 1 t 0 2

0 0 0 t 1 0 0

V0 0 0 0 0 1, V 0,

(4.3)

APPENDIX B EXCESS ERROR ESTIMATION IN FORWARD LOGISTIC REGRESSION 181

TABLE 1 The Results of 400 Experiments of Simulation 1.1

R E(R) SD( R) RMSEi(R) RMSE2(R)

apparent 0.0000 0.0000 0.1354 0.1006

cross 0.1039 0.1060 0.1381 0.1060

jack 0.0951 0.0864 0.1274 0.0865

boot 0.0786 0.0252 0.1078 0.0334

ideal 0.1006 0.0000 0.0906 0.0000

Note: RMSE1 is the root mean squared error about the true excess, and RMSE2 is that about the expected excess error. The expected excess error is E( R) for ideal.

Each of the six simulations consisted of 400 experiments. The results of all 400 experiments of simulation 1.1 are summarized in Table 1. In each experiment, we estimate the excess error R by evaluating the realized prediction rule on a large number (5,000) of new observations. We estimate the expected excess error by the sample average of the excess errors in the 400 experiments. To compare the three estimators, I first remark that in the 400 experiments, the bootstrap estimate was closest to the true excess error 210 times. From Table 1 we see that since

EFross ) = 0.1039, E(rjack ) = 0.0951, E(R) = 0.1006

are all close, rcross and fjack are nearly unbiased estimates of the expected excess error E(R), whereas rboot with expectation E( rboot) = 0.0786 is biased downwards. [Actually, since we are using the sample averages of the excess errors in 400 experiments as estimates of the expected excess errors, we are more correct in saying that a 95% confidence interval for E( rcross) is (0.0935), 0.1143), which contains E(R), and a 95% confidence interval for E( rjack) is (0.0866, 1036), which also contains E(R). On the other hand, a 95% confidence interval for E( rboot) is (0.0761, 0.0811), which does not contain E(R).] However, rcorss and rjack have enormous standard deviations, 0.1060 and 0.0864, respectively, compared to 0.0252, the standard deviation of rboot. From the column for RMSEb,

RMSE1 (ideal) < RMSE1 (boot ) < RMSE1 (app ) ~ RMSEb (Las ) ~ RMSEb (jack ),

with RMSEi( rboot) being about one-third of the distance between RMSEb( rideal) and RMSEb( rapp). The same ordering holds for RMSE2.

Recall that simulations 1.1, 1.2, and 1.3 had the same underlying distribution but differing sample sizes, n = 20, 40, and 60. As sample size increased, the expected excess error decreased, as did the mean squared error of the apparent error. We observed a similar pattern in simulations 2.1, 2.2, and 2.3, where the sample sizes were again n = 20, 40, and 60,

182 APPENDIX B EXCESS ERROR ESTIMATION IN FORWARD LOGISTIC REGRESSION

1.1

1.2

%

1.3

2.1

HH

2.2

2.3

0.00

0.04

0.08

0.12

0.16

0.20

FIGURE 2 95% (nonsimultaneous) Confidence Intervals for RMSE^ In each set of simulations, there are five confidence intervals for, respectively, apparent (A), cross-validation (C), jackknife (J), bootstrap (B), and ideal (1) estimates of the

excess error. Each confidence interval is indicated by . The middle vertical bar

**74**> 75 76 77 78 79 80 .. 90 >> Next