IDEAS home Printed from https://ideas.repec.org/p/ecl/riceco/14-018.html

To Hold Out or Not to Hold Out

Author

Listed:
  • Schorfheide, Frank

    (University of PA)

  • Wolpin, Kenneth I.

    (Rice University and University of PA)

Abstract

A recent literature has developed that combines two prominent empirical approaches to ex ante policy evaluation: randomized controlled trials (RCT) and structural estimation. The RCT provides a "gold standard" estimate of a particular treatment, but only of that treatment. Structural estimation provides the capability to extrapolate beyond the experimental treatment, but is based on untestable assumptions and is subject to structural data mining. Combining the approaches by holding out from the structural estimation exercise either the treatment or the control sample allows for external validation of the underlying behavioral model. Although intuitively appealing, this holdout methodology is not well grounded. For instance, it is easy to show that it is suboptimal from a Bayesian perspective. Using a stylized representation of a randomized controlled trial, we provide a formal rationale for the use of a holdout sample in an environment in which data mining poses an impediment to the implementation of the ideal Bayesian analysis and a numerical illustration of the potential benefits of holdout samples.

Suggested Citation

  • Schorfheide, Frank & Wolpin, Kenneth I., 2014. "To Hold Out or Not to Hold Out," Working Papers 14-018, Rice University, Department of Economics.
    • Handle: RePEc:ecl:riceco:14-018
    as

    Download full text from publisher

    File URL: http://economics.rice.edu/rise/working-papers/hold-out-or-not-hold-out
    Download Restriction: no
    ---><---

    Other versions of this item:

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. is not listed on IDEAS
    2. Jia, Zhiyang & Vattø, Trine Engh, 2021. "Predicting the path of labor supply responses when state dependence matters," Labour Economics, Elsevier, vol. 71(C).
    3. John Rust, 2014. "The Limits of Inference with Theory: A Review of Wolpin (2013)," Journal of Economic Literature, American Economic Association, vol. 52(3), pages 820-850, September.
    4. de Bresser, Jochem, 2021. "Evaluating the Accuracy of Counterfactuals The Role of Heterogeneous Expectations in Life Cycle Models," Discussion Paper 2021-034, Tilburg University, Center for Economic Research.
    5. Maibom, Jonas, 2021. "The Danish Labor Market Experiments: Methods and Findings," Nationaløkonomisk tidsskrift, Nationaløkonomisk Forening, vol. 2021(1), pages 1-21.
    6. Sebastian Galiani & Juan Pantano, 2021. "Structural Models: Inception and Frontier," NBER Working Papers 28698, National Bureau of Economic Research, Inc.
    7. Banghua Zhu & Sai Praneeth Karimireddy & Jiantao Jiao & Michael I. Jordan, 2023. "Online Learning in a Creator Economy," Papers 2305.11381, arXiv.org.
    8. Banghua Zhu & Stephen Bates & Zhuoran Yang & Yixin Wang & Jiantao Jiao & Michael I. Jordan, 2022. "The Sample Complexity of Online Contract Design," Papers 2211.05732, arXiv.org, revised May 2023.
    9. Paul Duetting & Michal Feldman & Tomasz Ponitka & Ermis Soumalias, 2025. "The Pseudo-Dimension of Contracts," Papers 2501.14474, arXiv.org.
    10. de Bresser, Jochem, 2021. "Evaluating the Accuracy of Counterfactuals The Role of Heterogeneous Expectations in Life Cycle Models," Other publications TiSEM a7e2b4d8-fed0-4e86-926f-d, Tilburg University, School of Economics and Management.

    More about this item

    JEL classification:

    • C11 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Bayesian Analysis: General
    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecl:riceco:14-018. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/dericus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.