IDEAS home Printed from https://ideas.repec.org/a/aea/jecper/v29y2015i3p61-80.html
   My bibliography  Save this article

Promises and Perils of Pre-analysis Plans

Author

Listed:
  • Benjamin A. Olken

Abstract

The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are greatly reduced. I begin by laying out the basics of what a statistical analysis plan actually contains so those researchers unfamiliar with it can better understand how it is done. In so doing, I have drawn both on standards used in clinical trials, which are clearly specified by the Food and Drug Administration, as well as my own practical experience from writing these plans in economics contexts. I then lay out some of the advantages of pre-specified analysis plans, both for the scientific community as a whole and also for the researcher. I also explore some of the limitations and costs of such plans. I then review a few pieces of evidence that suggest that, in many contexts, the benefits of using pre-specified analysis plans may not be as high as one might have expected initially. For the most part, I will focus on the relatively narrow issue of pre-analysis for randomized controlled trials.

Suggested Citation

  • Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
  • Handle: RePEc:aea:jecper:v:29:y:2015:i:3:p:61-80
    Note: DOI: 10.1257/jep.29.3.61
    as

    Download full text from publisher

    File URL: http://www.aeaweb.org/articles.php?doi=10.1257/jep.29.3.61
    Download Restriction: no

    File URL: http://www.aeaweb.org/jep/app/2903/29030061_app.pdf
    Download Restriction: no

    File URL: http://www.aeaweb.org/jep/ds/2903/29030061_ds.zip
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.

    References listed on IDEAS

    as
    1. repec:oup:qjecon:v:129:y:2014:i:2:p:697-752. is not listed on IDEAS
    2. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 128(2), pages 531-580.
    3. repec:pri:rpdevs:deaton_instruments_randomization_learning_all_04april_2010 is not listed on IDEAS
    4. Pascaline Dupas, 2014. "Short‐Run Subsidies and Long‐Run Adoption of New Health Products: Evidence From a Field Experiment," Econometrica, Econometric Society, vol. 82(1), pages 197-228, January.
    5. Loukas Balafoutas & Adrian Beck & Rudolf Kerschbamer & Matthias Sutter, 2013. "What Drives Taxi Drivers? A Field Experiment on Fraud in a Market for Credence Goods," Review of Economic Studies, Oxford University Press, vol. 80(3), pages 876-891.
    6. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    7. Dean Karlan & Robert Osei & Isaac Osei-Akoto & Christopher Udry, 2014. "Agricultural Decisions after Relaxing Credit and Risk Constraints," The Quarterly Journal of Economics, Oxford University Press, vol. 129(2), pages 597-652.
    8. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    9. Kory Kroft & Fabian Lange & Matthew J. Notowidigdo, 2013. "Duration Dependence and Labor Market Conditions: Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 128(3), pages 1123-1167.
    10. Christopher Blattman & Nathan Fiala & Sebastian Martinez, 2014. "Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda," The Quarterly Journal of Economics, Oxford University Press, vol. 129(2), pages 697-752.
    11. Nicholas Bloom & Benn Eifert & Aprajit Mahajan & David McKenzie & John Roberts, 2013. "Does Management Matter? Evidence from India," The Quarterly Journal of Economics, Oxford University Press, vol. 128(1), pages 1-51.
    12. Alessandro Tarozzi & Aprajit Mahajan & Brian Blackburn & Dan Kopf & Lakshmi Krishnan & Joanne Yoong, 2014. "Micro-loans, Insecticide-Treated Bednets, and Malaria: Evidence from a Randomized Controlled Trial in Orissa, India," American Economic Review, American Economic Association, vol. 104(7), pages 1909-1941, July.
    13. Pascaline Dupas & Jonathan Robinson, 2013. "Why Don't the Poor Save More? Evidence from Health Savings Experiments," American Economic Review, American Economic Association, vol. 103(4), pages 1138-1171, June.
    14. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    15. Francesco Avvisati & Marc Gurgand & Nina Guyon & Eric Maurin, 2014. "Getting Parents Involved: A Field Experiment in Deprived Schools," Review of Economic Studies, Oxford University Press, vol. 81(1), pages 57-83.
    16. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    17. Ernesto Dal Bó & Frederico Finan & Martín A. Rossi, 2013. "Strengthening State Capabilities: The Role of Financial Incentives in the Call to Public Service," The Quarterly Journal of Economics, Oxford University Press, vol. 128(3), pages 1169-1218.
    18. Michael Kremer & Edward Miguel & Rebecca Thornton, 2009. "Incentives to Learn," The Review of Economics and Statistics, MIT Press, vol. 91(3), pages 437-456, August.
    19. Katrina Jessoe & David Rapson, 2014. "Knowledge Is (Less) Power: Experimental Evidence from Residential Energy Use," American Economic Review, American Economic Association, vol. 104(4), pages 1417-1438, April.
    20. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, Oxford University Press, vol. 127(4), pages 1755-1812.
    21. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    22. Esther Duflo & Michael Greenstone & Nicholas Ryan, 2013. "Truth-telling by Third-party Auditors and the Response of Polluting Firms: Experimental Evidence from India," The Quarterly Journal of Economics, Oxford University Press, vol. 128(4), pages 1499-1545.
    23. Leonardo Bursztyn & Florian Ederer & Bruno Ferman & Noam Yuchtman, 2014. "Understanding Mechanisms Underlying Peer Effects: Evidence From a Field Experiment on Financial Decisions," Econometrica, Econometric Society, vol. 82(4), pages 1273-1301, July.
    24. Stefan Eriksson & Dan-Olof Rooth, 2014. "Do Employers Use Unemployment as a Sorting Criterion When Hiring? Evidence from a Field Experiment," American Economic Review, American Economic Association, vol. 104(3), pages 1014-1039, March.
    25. Erica Field & Rohini Pande & John Papp & Natalia Rigol, 2013. "Does the Classic Microfinance Model Discourage Entrepreneurship among the Poor? Experimental Evidence from India," American Economic Review, American Economic Association, vol. 103(6), pages 2196-2226, October.
    26. Nava Ashraf & Erica Field & Jean Lee, 2014. "Household Bargaining and Excess Fertility: An Experimental Study in Zambia," American Economic Review, American Economic Association, vol. 104(7), pages 2210-2237, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. ‘Metrics Monday: Robustness Check or Data Mining?
      by Marc F. Bellemare in Marc F. Bellemare on 2016-04-11 15:00:22

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Di Tillio, Alfredo & Ottaviani, Marco & Sørensen, Peter Norman, 2016. "Persuasion Bias in Science: Can Economics Help?," CEPR Discussion Papers 11343, C.E.P.R. Discussion Papers.
    2. Carneiro, Pedro & Lee, Sokbae & Wilhelm, Daniel, 2016. "Optimal Data Collection for Randomized Control Trials," IZA Discussion Papers 9908, Institute for the Study of Labor (IZA).
    3. Ivar Kolstad & Arne Wiig, 2016. "How do voters respond to information on self-serving elite behaviour? Evidence from a randomized survey experiment in Tanzania," CMI Working Papers 9, CMI (Chr. Michelsen Institute), Bergen, Norway.
    4. Garret S. Christensen & Edward Miguel, 2016. "Transparency, Reproducibility, and the Credibility of Economics Research," NBER Working Papers 22989, National Bureau of Economic Research, Inc.
    5. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    6. Michael L. Anderson & Jeremy Magruder, 2017. "Split-Sample Strategies for Avoiding False Discoveries," NBER Working Papers 23544, National Bureau of Economic Research, Inc.
    7. Knaus, Michael C. & Lechner, Michael & Strittmatter, Anthony, 2017. "Heterogeneous Employment Effects of Job Search Programmes: A Machine Learning Approach," IZA Discussion Papers 10961, Institute for the Study of Labor (IZA).
    8. Muhammad Haseeb & Kate Vyborny, 2016. "Imposing institutions: Evidence from cash transfer reform in Pakistan," CSAE Working Paper Series 2016-36, Centre for the Study of African Economies, University of Oxford.
    9. Abebe, Girum & Caria, Stefano & Fafchamps, Marcel & Falco, Paolo & Franklin, Simon & Quinn, Simon, 2017. "Anonymity of distance? Job search and labour market exclusion in a growing African city," LSE Research Online Documents on Economics 86573, London School of Economics and Political Science, LSE Library.
    10. Eble,Alex & Boone,Peter & Elbourne,Diana, 2016. "On minimizing the risk of bias in randomized controlled trials in economics," Policy Research Working Paper Series 7746, The World Bank.
    11. repec:aea:aecrev:v:107:y:2017:i:5:p:41-45 is not listed on IDEAS
    12. Martin Dufwenberg Jr. & Martin Dufwenberg, 2016. "Lies in Disguise - A Theoretical Analysis of Cheating," CESifo Working Paper Series 6208, CESifo Group Munich.
    13. Ivar Kolstad & Arne Wiig, 2016. "How do voters respond to information on self-serving elite behaviour? Evidence from a randomized survey experiment in Tanzania," CMI Working Papers 9, CMI (Chr. Michelsen Institute), Bergen, Norway.

    More about this item

    JEL classification:

    • C38 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Classification Methdos; Cluster Analysis; Principal Components; Factor Analysis

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:jecper:v:29:y:2015:i:3:p:61-80. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jane Voros) or (Michael P. Albert). General contact details of provider: http://edirc.repec.org/data/aeaaaea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.