IDEAS home Printed from https://ideas.repec.org/a/aea/jecper/v35y2021i3p175-92.html
   My bibliography  Save this article

Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It

Author

Listed:
  • Maximilian Kasy

Abstract

A key challenge for interpreting published empirical research is the fact that published findings might be selected by researchers or by journals. Selection might be based on criteria such as significance, consistency with theory, or the surprisingness of findings or their plausibility. Selection leads to biased estimates, reduced coverage of confidence intervals, and distorted posterior beliefs. I review methods for detecting and quantifying selection based on the distribution of p-values, systematic replication studies, and meta-studies. I then discuss the conflicting recommendations regarding selection resulting from alternative objectives, in particular, the validity of inference versus the relevance of findings for decision-makers. Based on this discussion, I consider various reform proposals, such as deemphasizing significance, pre-analysis plans, journals for null results and replication studies, and a functionally differentiated publication system. In conclusion, I argue that we need alternative foundations of statistics that go beyond the single-agent model of decision theory.

Suggested Citation

  • Maximilian Kasy, 2021. "Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 175-192, Summer.
    • Handle: RePEc:aea:jecper:v:35:y:2021:i:3:p:175-92
    DOI: 10.1257/jep.35.3.175
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/doi/10.1257/jep.35.3.175
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/jep.35.3.175.data
    Download Restriction: no
    File URL: https://libkey.io/10.1257/jep.35.3.175?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marianne Saam, 2021. "Überlegungen zu Open Science in der Volkswirtschaftslehre [Reflections on Open Science in Economics]," Wirtschaftsdienst, Springer;ZBW - Leibniz Information Centre for Economics, vol. 101(10), pages 746-747, October.
    2. Millán-Quijano, Jaime & Pulgarín, Sebastián, 2023. "Oiling up the field. Forced internal displacement and the expansion of palm oil in Colombia," World Development, Elsevier, vol. 162(C).
    3. Joakim A. Weill & Matthieu Stigler & Olivier Deschenes & Michael R. Springborn, 2021. "Researchers' Degrees-of-Flexibility and the Credibility of Difference-in-Differences Estimates: Evidence From the Pandemic Policy Evaluations," NBER Working Papers 29550, National Bureau of Economic Research, Inc.
    4. Oriana Bandiera & Nidhi Parekh & Barbara Petrongolo & Michelle Rao, 2022. "Men are from Mars, and Women Too: A Bayesian Meta‐analysis of Overconfidence Experiments," Economica, London School of Economics and Political Science, vol. 89(S1), pages 38-70, June.
    5. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    6. Lionel Page & Charles N. Noussair & Robert Slonim, 2021. "The replication crisis, the rise of new research practices and what it means for experimental economics," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(2), pages 210-225, December.
    7. Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2022. "The Power of Tests for Detecting $p$-Hacking," Papers 2205.07950, arXiv.org, revised Apr 2024.
    8. Dražanová, Lenka & Gonnot, Jérôme & Heidland, Tobias & Krüger, Finja, 2023. "Which individual-level factors explain public attitudes toward immigration? A meta-analysis," Open Access Publications from Kiel Institute for the World Economy 287732, Kiel Institute for the World Economy (IfW Kiel).
    9. Patrick Vu, 2022. "Can the Replication Rate Tell Us About Publication Bias?," Papers 2206.15023, arXiv.org, revised Jul 2022.
    10. Weill, Joakim A. & Stigler, Matthieu & Deschenes, Olivier & Springborn, Michael R., 2021. "COVID-19 Mobility Policies Impacts: How Credible Are Difference-in-Differences Estimates?," IZA Discussion Papers 14682, Institute of Labor Economics (IZA).
    11. Dražanová, Lenka & Gonnot, Jérôme & Heidland, Tobias & Krüger, Finja, 2022. "Understanding differences in attitudes to immigration: A meta-analysis of individual-level factors," Kiel Working Papers 2235, Kiel Institute for the World Economy (IfW Kiel).
    12. Grosch, Kerstin & Fischer, Sabine, 2024. "Gender equivalence in overconfidence A large-scale experimental study in a non-WEIRD country," Department for Strategy and Innovation Working Paper Series 02/2024, WU Vienna University of Economics and Business.
    13. Kyle Myers & Wei Yang Tham, 2023. "Money, Time, and Grant Design," Papers 2312.06479, arXiv.org.
    14. Donovan, Stuart & de Graaff, Thomas & Grimes, Arthur & de Groot, Henri L.F. & Maré, David C., 2022. "Cities with forking paths? Agglomeration economies in New Zealand 1976–2018," Regional Science and Urban Economics, Elsevier, vol. 95(C).
    15. Vu, Patrick, 2022. "Can the Replication Rate Tell Us About Selective Publication?," I4R Discussion Paper Series 3, The Institute for Replication (I4R).

    More about this item

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • L82 - Industrial Organization - - Industry Studies: Services - - - Entertainment; Media

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:jecper:v:35:y:2021:i:3:p:175-92. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.