IDEAS home Printed from https://ideas.repec.org/a/aea/aecrev/v102y2012i3p477-81.html

On the Use of Holdout Samples for Model Selection

Author

Listed:
  • Frank Schorfheide
  • Kenneth I. Wolpin

Abstract

Researchers often hold out data from the estimation of econometric models to use for external validation. However, the use of holdout samples is suboptimal from a Bayesian perspective, which prescribes using the entire sample to form posterior model weights. This paper examines a possible rationale for the use of holdout samples: data-inspired modifications of structural models are likely to lead to an exaggeration of model fit. The use of holdout samples can, in principle, set an incentive for the modeler not to exaggerate model fit.

Suggested Citation

  • Frank Schorfheide & Kenneth I. Wolpin, 2012. "On the Use of Holdout Samples for Model Selection," American Economic Review, American Economic Association, vol. 102(3), pages 477-481, May.
    • Handle: RePEc:aea:aecrev:v:102:y:2012:i:3:p:477-81
    as

    Download full text from publisher

    File URL: http://www.aeaweb.org/articles.php?doi=10.1257/aer.102.3.477
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Frank Smets & Rafael Wouters, 2007. "Shocks and Frictions in US Business Cycles: A Bayesian DSGE Approach," American Economic Review, American Economic Association, vol. 97(3), pages 586-606, June.
    2. Frank Smets & Raf Wouters, 2003. "An Estimated Dynamic Stochastic General Equilibrium Model of the Euro Area," Journal of the European Economic Association, MIT Press, vol. 1(5), pages 1123-1175, September.
    3. John Geweke, 2010. "Complete and Incomplete Econometric Models," Economics Books, Princeton University Press, edition 1, number 9218.
    4. Frank Schorfheide, 2000. "Loss function-based evaluation of DSGE models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(6), pages 645-670.
    5. Lawrence J. Christiano & Martin Eichenbaum & Charles L. Evans, 2005. "Nominal Rigidities and the Dynamic Effects of a Shock to Monetary Policy," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 1-45, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Emrah Arbak, 2017. "Identifying the provisioning policies of Belgian banks," Working Paper Research 326, National Bank of Belgium.
    2. Schorfheide, Frank & Wolpin, Kenneth I., 2016. "To hold out or not to hold out," Research in Economics, Elsevier, vol. 70(2), pages 332-345.
    3. Banghua Zhu & Sai Praneeth Karimireddy & Jiantao Jiao & Michael I. Jordan, 2023. "Online Learning in a Creator Economy," Papers 2305.11381, arXiv.org.
    4. Banghua Zhu & Stephen Bates & Zhuoran Yang & Yixin Wang & Jiantao Jiao & Michael I. Jordan, 2022. "The Sample Complexity of Online Contract Design," Papers 2211.05732, arXiv.org, revised May 2023.
    5. Paul Duetting & Michal Feldman & Tomasz Ponitka & Ermis Soumalias, 2025. "The Pseudo-Dimension of Contracts," Papers 2501.14474, arXiv.org.
    6. Chen, Xiaomeng Charlene & Jones, Stewart & Hasan, Mostafa Monzur & Zhao, Ruoyun & Alam, Nurul, 2023. "Does strategic deviation influence firms’ use of supplier finance?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 85(C).
    7. Hui Chen & Winston Wei Dou & Leonid Kogan, 2024. "Measuring “Dark Matter” in Asset Pricing Models," Journal of Finance, American Finance Association, vol. 79(2), pages 843-902, April.
    8. Antonio Merlo & Thomas R. Palfrey, 2018. "External validation of voter turnout models by concealed parameter recovery," Public Choice, Springer, vol. 176(1), pages 297-314, July.
    9. Joel da Costa & Tim Gebbie, 2020. "Learning low-frequency temporal patterns for quantitative trading," Papers 2008.09481, arXiv.org.
    10. Kanczuk, Fabio, 2015. "Brazil Through the Eyes of CHORINHO," Brazilian Review of Econometrics, Sociedade Brasileira de Econometria - SBE, vol. 35(2), March.
    11. de Bresser, Jochem, 2021. "Evaluating the Accuracy of Counterfactuals The Role of Heterogeneous Expectations in Life Cycle Models," Discussion Paper 2021-034, Tilburg University, Center for Economic Research.
    12. Emrah Arbak, 2017. "Identifying the provisioning policies of Belgian banks," Working Paper Research 326, National Bank of Belgium.
    13. Maibom, Jonas, 2021. "The Danish Labor Market Experiments: Methods and Findings," Nationaløkonomisk tidsskrift, Nationaløkonomisk Forening, vol. 2021(1), pages 1-21.
    14. Sebastian Galiani & Juan Pantano, 2021. "Structural Models: Inception and Frontier," NBER Working Papers 28698, National Bureau of Economic Research, Inc.
    15. Cepeda Carrión, Gabriel & Henseler, Jörg & Ringle, Christian M. & Roldán, José Luis, 2016. "Prediction-oriented modeling in business research by means of PLS path modeling: Introduction to a JBR special section," Journal of Business Research, Elsevier, vol. 69(10), pages 4545-4551.
    16. Wolfgang Drobetz & Tizian Otto, 2021. "Empirical asset pricing via machine learning: evidence from the European stock market," Journal of Asset Management, Palgrave Macmillan, vol. 22(7), pages 507-538, December.
    17. de Bresser, Jochem, 2021. "Evaluating the Accuracy of Counterfactuals The Role of Heterogeneous Expectations in Life Cycle Models," Other publications TiSEM a7e2b4d8-fed0-4e86-926f-d, Tilburg University, School of Economics and Management.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pablo A. Guerrón-Quintana & James M. Nason, 2013. "Bayesian estimation of DSGE models," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 21, pages 486-512, Edward Elgar Publishing.
    2. Fernández-Villaverde, J. & Rubio-Ramírez, J.F. & Schorfheide, F., 2016. "Solution and Estimation Methods for DSGE Models," Handbook of Macroeconomics, in: J. B. Taylor & Harald Uhlig (ed.), Handbook of Macroeconomics, edition 1, volume 2, chapter 0, pages 527-724, Elsevier.
    3. John B. Taylor & Volker Wieland, 2012. "Surprising Comparative Properties of Monetary Models: Results from a New Model Database," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 800-816, August.
    4. Faust, Jon & Gupta, Abhishek, 2010. "Posterior Predictive Analysis for Evaluating DSGE Models," MPRA Paper 26721, University Library of Munich, Germany.
    5. Ana Beatriz Galvão & Liudas Giraitis & George Kapetanios & Katerina Petrova, 2015. "A Bayesian Local Likelihood Method for Modelling Parameter Time Variation in DSGE Models," Working Papers 770, Queen Mary University of London, School of Economics and Finance.
    6. Christoffel, Kai & Kuester, Keith & Linzert, Tobias, 2009. "The role of labor markets for euro area monetary policy," European Economic Review, Elsevier, vol. 53(8), pages 908-936, November.
    7. Ippei Fujiwara & Yasuo Hirose & Mototsugu Shintani, 2011. "Can News Be a Major Source of Aggregate Fluctuations? A Bayesian DSGE Approach," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 43(1), pages 1-29, February.
    8. Nalan Basturk & Cem Cakmakli & S. Pinar Ceyhan & Herman K. van Dijk, 2014. "On the Rise of Bayesian Econometrics after Cowles Foundation Monographs 10, 14," Tinbergen Institute Discussion Papers 14-085/III, Tinbergen Institute, revised 04 Sep 2014.
    9. Özer Karagedikli & Troy Matheson & Christie Smith & Shaun P. Vahey, 2010. "RBCs AND DSGEs: THE COMPUTATIONAL APPROACH TO BUSINESS CYCLE THEORY AND EVIDENCE," Journal of Economic Surveys, Wiley Blackwell, vol. 24(1), pages 113-136, February.
    10. Aruoba, S. Borağan & Bocola, Luigi & Schorfheide, Frank, 2017. "Assessing DSGE model nonlinearities," Journal of Economic Dynamics and Control, Elsevier, vol. 83(C), pages 34-54.
    11. Rodríguez, Aldo, 2020. "Estimación Bayesiana de un Modelo de Economía Abierta con Sector Bancario," Dynare Working Papers 52, CEPREMAP.
    12. Yongsung Chang & Sun-Bin Kim & Frank Schorfheide, 2010. "Labor-Market Heterogeneity, Aggregation, and the Lucas Critique," RCER Working Papers 556, University of Rochester - Center for Economic Research (RCER).
    13. Warne, Anders & Coenen, Günter & Christoffel, Kai, 2010. "Forecasting with DSGE models," Working Paper Series 1185, European Central Bank.
    14. Ma, Yong & Li, Shushu, 2015. "Bayesian estimation of China's monetary policy transparency: A New Keynesian approach," Economic Modelling, Elsevier, vol. 45(C), pages 236-248.
    15. Marco Del Negro & Frank Schorfheide, 2009. "Monetary Policy Analysis with Potentially Misspecified Models," American Economic Review, American Economic Association, vol. 99(4), pages 1415-1450, September.
    16. Christian Müller-Kademann, 2018. "The Lucas Critique: A Lucas Critique," Economic Thought, World Economics Association, vol. 7(2), pages 54-62, November.
    17. S. Boragan Aruoba & Frank Schorfheide, 2011. "Sticky Prices versus Monetary Frictions: An Estimation of Policy Trade-Offs," American Economic Journal: Macroeconomics, American Economic Association, vol. 3(1), pages 60-90, January.
    18. Taylor, J.B., 2016. "The Staying Power of Staggered Wage and Price Setting Models in Macroeconomics," Handbook of Macroeconomics, in: J. B. Taylor & Harald Uhlig (ed.), Handbook of Macroeconomics, edition 1, volume 2, chapter 0, pages 2009-2042, Elsevier.
    19. Erceg, Christopher J. & Jakab, Zoltan & Lindé, Jesper, 2021. "Monetary policy strategies for the European Central Bank," Journal of Economic Dynamics and Control, Elsevier, vol. 132(C).
    20. Taylor, John & Wieland, Volker, 2009. "Surprising comparative properties of monetary models: Results from a new data base," CEPR Discussion Papers 7294, C.E.P.R. Discussion Papers.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:aecrev:v:102:y:2012:i:3:p:477-81. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.