This file is part of IDEAS, which uses RePEc data


[ Papers | Articles | Software | Books | Chapters | Authors | Institutions | JEL Classification | NEP reports | Search | New papers by email | Author registration | Rankings | Volunteers | FAQ | Blog | Help! ]

Formalized Data Snooping Based on Generalized Error Rates

Author info | Abstract | Publisher info | Download info | Related research | Statistics
Author Info
Joseph P
Romano
Azeem M. Shaikh
Michael Wolf

Additional information is available for the following registered author(s):

Abstract

It is common in econometric applications that several hypothesis tests are carried out at the same time. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. The classical approach is to control the familywise error rate (FWE), that is, the probability of one or more false rejections. But when the number of hypotheses under consideration is large, control of the FWE can become too demanding. As a result, the number of false hypotheses rejected may be small or even zero. This suggests replacing control of the FWE by a more liberal measure. To this end, we review a number of proposals from the statistical literature. We briefly discuss how these procedures apply to the general problem of model selection. A simulation study and two empirical applications illustrate the methods.

Download Info
To download:

If you experience problems downloading a file, check if you have the proper application to view it first. Information about this may be contained in the File-Format links below. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.iew.unizh.ch/wp/iewwp259.pdf
File Format: application/pdf
File Function:
Download Restriction: no

Publisher Info
Paper provided by Institute for Empirical Research in Economics - IEW in its series IEW - Working Papers with number iewwp259.

Download reference. The following formats are available: HTML (with abstract), plain text (with abstract), BibTeX, RIS (EndNote, RefMan, ProCite), ReDIF
Length:
Date of creation: Dec 2005
Date of revision:
Handle: RePEc:zur:iewwpx:259

Contact details of provider:

For technical questions regarding this item, or to correct its listing, contact: (Hanna Britt).

Related research
Keywords: Data snooping; false discovery proportion; false discovery rate; generalized familywise error rate; model selection; multiple testing; stepwise methods;

Other versions of this item:

Find related papers by JEL classification:
C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: General - - - Hypothesis Testing
C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: General - - - Semiparametric and Nonparametric Methods
C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation and Testing

This paper has been announced in the following NEP Reports:

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
  1. Peter Reinhard Hansen & Asger Lunde & James M. Nason, 2003. "Choosing the Best Volatility Models: The Model Confidence Set Approach," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 839-861, December. [Downloadable!] (restricted)
    Other versions:
  2. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, 07. [Downloadable!] (restricted)
  3. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, Elsevier. [Downloadable!] (restricted)
  4. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October. [Downloadable!] (restricted)
  5. Sullivan, Ryan & Timmermann, Allan G & White, Halbert, 1998. "Data-Snooping, Technical Trading Rule Performance and the Bootstrap," CEPR Discussion Papers 1976, C.E.P.R. Discussion Papers. [Downloadable!] (restricted)
    Other versions:
  6. Peter Reinhard Hansen & Asger Lunde & James M. Nason, 2005. "Model confidence sets for forecasting models," Working Paper 2005-07, Federal Reserve Bank of Atlanta. [Downloadable!]
  7. Sullivan, Ryan & Timmermann, Allan & White, Halbert, 2001. "Dangers of data mining: The case of calendar effects in stock returns," Journal of Econometrics, Elsevier, vol. 105(1), pages 249-286, November. [Downloadable!] (restricted)
  8. Kabaila, Paul & Leeb, Hannes, 2006. "On the Large-Sample Minimal Coverage Probability of Confidence Intervals After Model Selection," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 619-629, June. [Downloadable!] (restricted)
  9. Abramovich, Felix & Benjamini, Yoav, 1996. "Adaptive thresholding of wavelet coefficients," Computational Statistics & Data Analysis, Elsevier, vol. 22(4), pages 351-361, August. [Downloadable!] (restricted)
  10. Krolzig, Hans-Martin & Hendry, David F., 2001. "Computer automation of general-to-specific model selection procedures," Journal of Economic Dynamics and Control, Elsevier, vol. 25(6-7), pages 831-866, June. [Downloadable!] (restricted)
    Other versions:
  11. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
  12. Andrews, Donald W K & Monahan, J Christopher, 1992. "An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator," Econometrica, Econometric Society, vol. 60(4), pages 953-66, July. [Downloadable!] (restricted)
    Other versions:
  13. Hidetoshi Shimodaira, 1998. "An Application of Multiple Comparison Techniques to Model Selection," Annals of the Institute of Statistical Mathematics, Springer, vol. 50(1), pages 1-13, March. [Downloadable!] (restricted)
  14. Xiaotong Shen & Hsin-Cheng Huang & Jimmy Ye, 2004. "Inference After Model Selection," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 751-762, January. [Downloadable!] (restricted)
  15. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March. [Downloadable!] (restricted)
Full references

Cited by:
(explanations, Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.)

  1. Michael Wolf & Dan Wunderli, 2009. "Fund-of-funds construction by statistical multiple testing methods," IEW - Working Papers iewwp445, Institute for Empirical Research in Economics - IEW. [Downloadable!]
  2. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2008. "Control of the False Discovery Rate under Dependence using the Bootstrap and Subsampling," IEW - Working Papers iewwp337, Institute for Empirical Research in Economics - IEW. [Downloadable!]
  3. Hanck, Christoph, 2008. "Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach," MPRA Paper 12008, University Library of Munich, Germany. [Downloadable!]
  4. Deckers, Thomas & Hanck, Christoph, 2009. "Multiple Testing Techniques in Growth Econometrics," MPRA Paper 17843, University Library of Munich, Germany. [Downloadable!]
  5. Christopher J. Bennett, 2009. "p-Value Adjustments for Asymptotic Control of the Generalized Familywise Error Rate," Working Papers 0905, Department of Economics, Vanderbilt University. [Downloadable!]
  6. Oliver Ledoit & Michael Wolf, 2008. "Robust Performance Hypothesis Testing with the Sharpe Ratio," IEW - Working Papers iewwp320, Institute for Empirical Research in Economics - IEW. [Downloadable!]
  7. Joseph Romano & Azeem Shaikh & Michael Wolf, 2008. "Rejoinder on: Control of the false discovery rate under dependence using the bootstrap and subsampling," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer, vol. 17(3), pages 461-471, November. [Downloadable!] (restricted)
  8. Hanck, Christoph, 2008. "Nonstationary-Volatility Robust Panel Unit Root Tests and the Great Moderation," MPRA Paper 11988, University Library of Munich, Germany. [Downloadable!]
  9. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2009. "Hypothesis testing in econometrics," IEW - Working Papers iewwp444, Institute for Empirical Research in Economics - IEW. [Downloadable!]
Statistics
Access and download statistics

Did you know? All RePEc services are meant to be be free forever, as they are all run by volunteers.

This page was last updated on 2009-11-19.

This information is provided to you by IDEAS at the Department of Economics, College of Liberal Arts and Sciences, University of Connecticut using RePEc data on a server sponsored by the Society for Economic Dynamics.