This file is part of IDEAS, which uses RePEc data


[ Papers | Articles | Software | Books | Chapters | Authors | Institutions | JEL Classification | NEP reports | Search | New papers by email | Author registration | Rankings | Volunteers | FAQ | Blog | Help! ]

Formalized Data Snooping Based On Generalized Error Rates

Author info | Abstract | Publisher info | Download info | Related research | Statistics
Author Info
Romano, Joseph P.
Shaikh, Azeem M.
Wolf, Michael

Additional information is available for the following registered author(s):

Abstract

It is common in econometric applications that several hypothesis tests are carried out simultaneously. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. The classical approach is to control the familywise error rate (FWE), which is the probability of one or more false rejections. But when the number of hypotheses under consideration is large, control of the FWE can become too demanding. As a result, the number of false hypotheses rejected may be small or even zero. This suggests replacing control of the FWE by a more liberal measure. To this end, we review a number of recent proposals from the statistical literature. We briefly discuss how these procedures apply to the general problem of model selection. A simulation study and two empirical applications illustrate the methods.We thank three anonymous referees for helpful comments that have led to an improved presentation of the paper. The research of the third author has been partially supported by the Spanish Ministry of Science and Technology and FEDER, Grant BMF2003-03324.

Download Info
To download:

If you experience problems downloading a file, check if you have the proper application to view it first. Information about this may be contained in the File-Format links below. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://journals.cambridge.org/abstract_S0266466608080171
File Format: text/html
File Function: link to article abstract page
Download Restriction: no

Publisher Info
Article provided by Cambridge University Press in its journal Econometric Theory.

Volume (Year): 24 (2008)
Issue (Month): 02 (April)
Pages: 404-447
Download reference. The following formats are available: HTML (with abstract), plain text (with abstract), BibTeX, RIS (EndNote, RefMan, ProCite), ReDIF
Handle: RePEc:cup:etheor:v:24:y:2008:i:02:p:404-447_08

Contact details of provider:
Postal: The Edinburgh Building, Shaftesbury Road, Cambridge CB2 2RU UK
Fax: +44 (0)1223 325150
Email:
Web page: http://journals.cambridge.org/jid_ECT

For technical questions regarding this item, or to correct its listing, contact: (Mike Eden).

Related research
Keywords:

Other versions of this item:

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
  1. Peter Reinhard Hansen & Asger Lunde & James M. Nason, 2003. "Choosing the Best Volatility Models: The Model Confidence Set Approach," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 839-861, December. [Downloadable!] (restricted)
    Other versions:
  2. Krolzig, Hans-Martin & Hendry, David F., 2001. "Computer automation of general-to-specific model selection procedures," Journal of Economic Dynamics and Control, Elsevier, vol. 25(6-7), pages 831-866, June. [Downloadable!] (restricted)
    Other versions:
  3. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
  4. Andrews, Donald W K & Monahan, J Christopher, 1992. "An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator," Econometrica, Econometric Society, vol. 60(4), pages 953-66, July. [Downloadable!] (restricted)
    Other versions:
  5. Hidetoshi Shimodaira, 1998. "An Application of Multiple Comparison Techniques to Model Selection," Annals of the Institute of Statistical Mathematics, Springer, vol. 50(1), pages 1-13, March. [Downloadable!] (restricted)
  6. Xiaotong Shen & Hsin-Cheng Huang & Jimmy Ye, 2004. "Inference After Model Selection," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 751-762, January. [Downloadable!] (restricted)
  7. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, 07. [Downloadable!] (restricted)
  8. Allan Timmermann & Halbert White & Ryan Sullivan, 1998. "Data-Snooping, Technical Trading, Rule Performance and the Bootstrap," FMG Discussion Papers dp303, Financial Markets Group. [Downloadable!] (restricted)
    Other versions:
  9. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, Elsevier. [Downloadable!] (restricted)
  10. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October. [Downloadable!] (restricted)
  11. Peter Reinhard Hansen & Asger Lunde & James M. Nason, 2005. "Model confidence sets for forecasting models," Working Paper 2005-07, Federal Reserve Bank of Atlanta. [Downloadable!]
  12. Sullivan, Ryan & Timmermann, Allan & White, Halbert, 2001. "Dangers of data mining: The case of calendar effects in stock returns," Journal of Econometrics, Elsevier, vol. 105(1), pages 249-286, November. [Downloadable!] (restricted)
  13. Kabaila, Paul & Leeb, Hannes, 2006. "On the Large-Sample Minimal Coverage Probability of Confidence Intervals After Model Selection," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 619-629, June. [Downloadable!] (restricted)
  14. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March. [Downloadable!] (restricted)
  15. Abramovich, Felix & Benjamini, Yoav, 1996. "Adaptive thresholding of wavelet coefficients," Computational Statistics & Data Analysis, Elsevier, vol. 22(4), pages 351-361, August. [Downloadable!] (restricted)
Full references

Cited by:
(explanations, Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.)

  1. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2008. "Control of the False Discovery Rate under Dependence using the Bootstrap and Subsampling," IEW - Working Papers iewwp337, Institute for Empirical Research in Economics - IEW. [Downloadable!]
  2. Christopher J. Bennett, 2009. "p-Value Adjustments for Asymptotic Control of the Generalized Familywise Error Rate," Working Papers 0905, Department of Economics, Vanderbilt University. [Downloadable!]
  3. Hanck, Christoph, 2008. "Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach," MPRA Paper 12008, University Library of Munich, Germany. [Downloadable!]
  4. Oliver Ledoit & Michael Wolf, 2008. "Robust Performance Hypothesis Testing with the Sharpe Ratio," IEW - Working Papers iewwp320, Institute for Empirical Research in Economics - IEW. [Downloadable!]
  5. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2009. "Hypothesis testing in econometrics," IEW - Working Papers iewwp444, Institute for Empirical Research in Economics - IEW. [Downloadable!]
  6. Deckers, Thomas & Hanck, Christoph, 2009. "Multiple Testing Techniques in Growth Econometrics," MPRA Paper 17843, University Library of Munich, Germany. [Downloadable!]
  7. Hanck, Christoph, 2008. "Nonstationary-Volatility Robust Panel Unit Root Tests and the Great Moderation," MPRA Paper 11988, University Library of Munich, Germany. [Downloadable!]
  8. Joseph Romano & Azeem Shaikh & Michael Wolf, 2008. "Rejoinder on: Control of the false discovery rate under dependence using the bootstrap and subsampling," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer, vol. 17(3), pages 461-471, November. [Downloadable!] (restricted)
  9. Michael Wolf & Dan Wunderli, 2009. "Fund-of-funds construction by statistical multiple testing methods," IEW - Working Papers iewwp445, Institute for Empirical Research in Economics - IEW. [Downloadable!]
Statistics
Access and download statistics

Did you know? You can create a compilation of all publications of a group of people, say alumni of a program, your students or memers of an association.

This page was last updated on 2009-11-24.

This information is provided to you by IDEAS at the Department of Economics, College of Liberal Arts and Sciences, University of Connecticut using RePEc data on a server sponsored by the Society for Economic Dynamics.