IDEAS home Printed from https://ideas.repec.org/a/eee/reecon/v70y2016i2p332-345.html
   My bibliography  Save this article

To hold out or not to hold out

Author

Listed:
  • Schorfheide, Frank
  • Wolpin, Kenneth I.

Abstract

A recent literature has developed that combines two prominent empirical approaches to ex ante policy evaluation: randomized controlled trials (RCT) and structural estimation. The RCT provides a “gold-standard” estimate of a particular treatment, but only of that treatment. Structural estimation provides the capability to extrapolate beyond the experimental treatment, but is based on untestable assumptions and is subject to structural data mining. Combining the approaches by holding out from the structural estimation exercise either the treatment or control sample allows for external validation of the underlying behavioral model. Although intuitively appealing, this holdout methodology is not well grounded. For instance, it is easy to show that it is suboptimal from a Bayesian perspective. Using a stylized representation of a randomized controlled trial, we provide a formal rationale for the use of a holdout sample in an environment in which data mining poses an impediment to the implementation of the ideal Bayesian analysis and a numerical illustration of the potential benefits of holdout samples.

Suggested Citation

  • Schorfheide, Frank & Wolpin, Kenneth I., 2016. "To hold out or not to hold out," Research in Economics, Elsevier, vol. 70(2), pages 332-345.
  • Handle: RePEc:eee:reecon:v:70:y:2016:i:2:p:332-345
    DOI: 10.1016/j.rie.2016.02.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1090944316300436
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Petra E. Todd & Kenneth I. Wolpin, 2008. "Ex Ante Evaluation of Social Programs," Annals of Economics and Statistics, GENES, issue 91-92, pages 263-291.
    2. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    3. repec:adr:anecst:y:2008:i:91-92:p:13 is not listed on IDEAS
    4. Lo, Andrew W & MacKinlay, A Craig, 1990. "Data-Snooping Biases in Tests of Financial Asset Pricing Models," Review of Financial Studies, Society for Financial Studies, vol. 3(3), pages 431-467.
    5. Alvaro Sandroni, 2003. "The reproducible properties of correct forecasts," International Journal of Game Theory, Springer;Game Theory Society, vol. 32(1), pages 151-159, December.
    6. Christopher Ferrall, 2012. "Explaining and Forecasting Results of the Self-sufficiency Project," Review of Economic Studies, Oxford University Press, vol. 79(4), pages 1495-1526.
    7. Hidehiko Ichimura & Christopher Taber, 2000. "Direct estimation of policy impacts," IFS Working Papers W00/05, Institute for Fiscal Studies.
    8. Lamont, Owen A., 2002. "Macroeconomic forecasts and microeconomic forecasters," Journal of Economic Behavior & Organization, Elsevier, vol. 48(3), pages 265-280, July.
    9. Frank Schorfheide & Kenneth I. Wolpin, 2012. "On the Use of Holdout Samples for Model Selection," American Economic Review, American Economic Association, vol. 102(3), pages 477-481, May.
    10. Wolpin, Kenneth I., 2013. "The Limits of Inference without Theory," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262019086, September.
    11. David Laster & Paul Bennett & In Sun Geoum, 1999. "Rational Bias in Macroeconomic Forecasts," The Quarterly Journal of Economics, Oxford University Press, vol. 114(1), pages 293-318.
    12. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    13. repec:adr:anecst:y:2008:i:91-92 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. John Rust, 2014. "The Limits of Inference with Theory: A Review of Wolpin (2013)," Journal of Economic Literature, American Economic Association, vol. 52(3), pages 820-850, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marinovic, Iván & Ottaviani, Marco & Sorensen, Peter, 2013. "Forecasters’ Objectives and Strategies," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 690-720, Elsevier.
    2. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    3. Papastamos, Dimitrios & Matysiak, George & Stevenson, Simon, 2015. "Assessing the accuracy and dispersion of real estate investment forecasts," International Review of Financial Analysis, Elsevier, vol. 42(C), pages 141-152.
    4. Bajgrowicz, Pierre & Scaillet, Olivier, 2012. "Technical trading revisited: False discoveries, persistence tests, and transaction costs," Journal of Financial Economics, Elsevier, vol. 106(3), pages 473-491.
    5. Aleksejs Krecetovs & Pasquale Della Corte, 2016. "Macro uncertainty and currency premia," 2016 Meeting Papers 624, Society for Economic Dynamics.
    6. McCracken,M.W. & West,K.D., 2001. "Inference about predictive ability," Working papers 14, Wisconsin Madison - Social Systems.
    7. Stefan Nagel, 2013. "Empirical Cross-Sectional Asset Pricing," Annual Review of Financial Economics, Annual Reviews, vol. 5(1), pages 167-199, November.
    8. Pierdzioch, Christian & Reid, Monique B. & Gupta, Rangan, 2016. "Inflation forecasts and forecaster herding: Evidence from South African survey data," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 62(C), pages 42-50.
    9. R. Vincent Pohl, 2018. "Medicaid And The Labor Supply Of Single Mothers: Implications For Health Care Reform," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 59(3), pages 1283-1313, August.
    10. Christian Pierdzioch & Georg Stadtmann & Dirk Schäfer, 2011. "Fly with the Eagles or Scratch with the Chickens? – Zum Herdenverhalten von Wechselkursprognostikern," Credit and Capital Markets, Credit and Capital Markets, vol. 44(4), pages 465-490.
    11. Rianne Legerstee & Philip Hans Franses, 2015. "Does Disagreement Amongst Forecasters Have Predictive Value?," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 34(4), pages 290-302, July.
    12. Evan F. Koenig & Sheila Dolmas & Jeremy Piger, 2003. "The Use and Abuse of Real-Time Data in Economic Forecasting," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 618-628, August.
    13. Issler, João Victor & Lima, Luiz Renato, 2009. "A panel data approach to economic forecasting: The bias-corrected average forecast," Journal of Econometrics, Elsevier, vol. 152(2), pages 153-164, October.
    14. Pierdzioch, Christian & Rülke, Jan Christoph & Stadtmann, Georg, 2010. "New evidence of anti-herding of oil-price forecasters," Energy Economics, Elsevier, vol. 32(6), pages 1456-1459, November.
    15. Volker Wieland & Maik Wolters, 2011. "The diversity of forecasts from macroeconomic models of the US economy," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 47(2), pages 247-292, June.
    16. Köhler, Wolfgang R., 2004. "Optimal Incentive Contracts for Experts," Bonn Econ Discussion Papers 6/2004, University of Bonn, Bonn Graduate School of Economics (BGSE).
    17. Bizer, Kilian & Meub, Lukas & Proeger, Till & Spiwoks, Markus, 2014. "Strategic coordination in forecasting: An experimental study," Center for European, Governance and Economic Development Research Discussion Papers 195, University of Goettingen, Department of Economics.
    18. Wang, Shan & Jiang, Zhi-Qiang & Li, Sai-Ping & Zhou, Wei-Xing, 2015. "Testing the performance of technical trading rules in the Chinese markets based on superior predictive test," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 439(C), pages 114-123.
    19. Kothari, S. P., 2001. "Capital markets research in accounting," Journal of Accounting and Economics, Elsevier, vol. 31(1-3), pages 105-231, September.
    20. Michael Cooper & Huseyin Gulen, 2006. "Is Time-Series-Based Predictability Evident in Real Time?," The Journal of Business, University of Chicago Press, vol. 79(3), pages 1263-1292, May.

    More about this item

    Keywords

    Bayesian analysis; Model selection; Principal–agent models; Randomized controlled trials;
    All these keywords.

    JEL classification:

    • C11 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Bayesian Analysis: General
    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reecon:v:70:y:2016:i:2:p:332-345. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Nithya Sathishkumar). General contact details of provider: http://www.elsevier.com/locate/inca/622941 .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.