On the Harm that Pretesting Does
AbstractData in econometrics are, as a rule, non-experimental and hence we have to use the same data set to select the model and also to estimate the parameters in the selected model.In standard applied econometrics practice, however, one reports zero bias and some variance of the (pretest) estimators conditional on the selected model.In this paper we find the unconditional moments of the pretest estimator, taking full account of the fact that model selection and estimation are an integrated procedure.We derive the bias, variance, and mean squared error of the pretest estimator, and show what the error is in not reporting the correct moments.This error can be very substantial.We also show that there can be large differences in underreporting between different model selection procedures.Finally, we ask how the underreporting error increases when the number of auxiliary regressors increases.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Tilburg University, Center for Economic Research in its series Discussion Paper with number 2001-37.
Date of creation: 2001
Date of revision:
Contact details of provider:
Web page: http://center.uvt.nl
econometric models; testing; estimation;
This paper has been announced in the following NEP Reports:
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Lo, Andrew W & MacKinlay, A Craig, 1990.
"Data-Snooping Biases in Tests of Financial Asset Pricing Models,"
Review of Financial Studies,
Society for Financial Studies, vol. 3(3), pages 431-67.
- Andrew W. Lo & A. Craig MacKinlay, 1991. "Data-Snooping Biases in Tests of Financial Asset Pricing Models," NBER Working Papers 3001, National Bureau of Economic Research, Inc.
- Lo, Andrew W. (Andrew Wen-Chuan) & MacKinlay, Archie Craig, 1955-, 1989. "Data-snooping biases in tests of financial asset pricing models," Working papers 3020-89., Massachusetts Institute of Technology (MIT), Sloan School of Management.
- Hendry, David F., 2001. "Achievements and challenges in econometric methodology," Journal of Econometrics, Elsevier, vol. 100(1), pages 7-10, January.
- Mittelhammer, R.C., 1984. "Restricted least squares, pre-test, ols and stein rule estimators: Risk comparisons under model misspecification," Journal of Econometrics, Elsevier, vol. 25(1-2), pages 151-164.
- Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
- Lovell, Michael C, 1983. "Data Mining," The Review of Economics and Statistics, MIT Press, vol. 65(1), pages 1-12, February.
- Feldstein, Martin S, 1973. "Multicollinearity and the Mean Square Error of Alternative Estimators," Econometrica, Econometric Society, vol. 41(2), pages 337-46, March.
- Pötscher, B.M., 1991. "Effects of Model Selection on Inference," Econometric Theory, Cambridge University Press, vol. 7(02), pages 163-185, June.
- Giles, Judith A & Giles, David E A, 1993. " Pre-test Estimation and Testing in Econometrics: Recent Developments," Journal of Economic Surveys, Wiley Blackwell, vol. 7(2), pages 145-97, June.
- Judge, G.G. & Bock, M.E., 1983. "Biased estimation," Handbook of Econometrics, in: Z. Griliches† & M. D. Intriligator (ed.), Handbook of Econometrics, edition 1, volume 1, chapter 10, pages 599-649 Elsevier.
- Zaman, A., 1984. "Avoiding model selection by the use of shrinkage techniques," Journal of Econometrics, Elsevier, vol. 25(1-2), pages 73-85.
- Leeb, Hannes & P tscher, Benedikt M., 2003.
"The Finite-Sample Distribution Of Post-Model-Selection Estimators And Uniform Versus Nonuniform Approximations,"
Cambridge University Press, vol. 19(01), pages 100-142, February.
- Hannes Leeb & Benedikt M. Poetscher, 2000. "The Finite-Sample Distribution of Post-Model-Selection Estimators, and Uniform Versus Non-Uniform Approximations," Econometrics 0004001, EconWPA.
- Kenneth D. West, 1994.
"Asymptotic Inference About Predictive Ability,"
- Kevin D. Hoover & Stephen J. Perez, 1999.
"Data mining reconsidered: encompassing and the general-to-specific approach to specification search,"
Royal Economic Society, vol. 2(2), pages 167-191.
- Kevin Hoover & Stephen J. Perez, 2003. "Data Mining Reconsidered: Encompassing And The General-To-Specific Approach To Specification Search," Working Papers 9727, University of California, Davis, Department of Economics.
- Kevin D. Hoover & Stephen J. Perez, . "Data Mining Reconsidered: Encompassing And The General-To-Specific Approach To Specification Search," Department of Economics 97-27, California Davis - Department of Economics.
- repec:cup:etheor:v:7:y:1991:i:2:p:163-85 is not listed on IDEAS
- Thomson, Michael & Schmidt, Peter, 1982. "A Note on the Comparison of the Mean Square Error of Inequality Constrained Least Squares and Other Related Estimators," The Review of Economics and Statistics, MIT Press, vol. 64(1), pages 174-76, February.
- Francis X. Diebold & Robert S. Mariano, 1994.
"Comparing Predictive Accuracy,"
NBER Technical Working Papers
0169, National Bureau of Economic Research, Inc.
- Diebold, Francis X & Mariano, Roberto S, 1995. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(3), pages 253-63, July.
- Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-44, January.
- Jan R. Magnus & J. Durbin, 1999. "Estimation of Regression Coefficients of Interest When Other Regression Coefficients Are of No Interest," Econometrica, Econometric Society, vol. 67(3), pages 639-644, May.
- Roehrig, C.S., 1984. "Optimal critical regions for pre-test estimators using a Bayes risk criterion," Journal of Econometrics, Elsevier, vol. 25(1-2), pages 3-14.
- William E. Griffiths & Duangkamon Chotikapanich & D. S. Prasada Rao, 2005.
"Averaging Income Distributions,"
Bulletin of Economic Research,
Wiley Blackwell, vol. 57(4), pages 347-367, October.
- Danilov, D.L. & Magnus, J.R., 2002.
"Forecast Accuracy after Pretesting with an Application to the Stock Market,"
2002-76, Tilburg University, Center for Economic Research.
- Jan R. Magnus & Dmitry Danilov, 2004. "Forecast accuracy after pretesting with an application to the stock market," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(4), pages 251-274.
- C. L Chua & W. E. Griffiths & C. J O'Donnell, 2001.
"Bayesian Model Averaging in Consumer Demand Systems with Inequality Constraints,"
Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie,
Canadian Agricultural Economics Society/Societe canadienne d'agroeconomie, vol. 49(3), pages 269-291, November.
- Chua, C.L. & Griffiths, W.E. & O'Donnell, C.J., 2001. "Bayesian Model Averaging in Consumer Demand Systems with Inequality Constraints," Department of Economics - Working Papers Series 806, The University of Melbourne.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Richard Broekman).
If references are entirely missing, you can add them using this form.