IDEAS home Printed from https://ideas.repec.org/p/cpr/ceprdp/17331.html

The Null Result Penalty

Author

Listed:
  • Chopra, Felix
  • Haaland, Ingar
  • Roth, Christopher
  • Stegmann, Andreas

Abstract

In experiments with economists, we measure how the evaluation of research studies depends on whether the study yielded a null result. Studies with null results are perceived to be less publishable, of lower quality, less important, and less precisely estimated than studies with statistically significant results, even when holding constant all other study features, including the precision of estimates. The penalty for null results is of similar magnitude for various subgroups of researchers, from PhD students to editors. The null result penalty is larger when experts predict a non-null result and when statistical uncertainty is communicated in terms of p-values rather than standard errors. Our findings have implications for understanding mechanisms underlying publication bias and the communication of research findings.

Suggested Citation

  • Chopra, Felix & Haaland, Ingar & Roth, Christopher & Stegmann, Andreas, 2022. "The Null Result Penalty," CEPR Discussion Papers 17331, C.E.P.R. Discussion Papers.
    • Handle: RePEc:cpr:ceprdp:17331
    as

    Download full text from publisher

    File URL: https://cepr.org/publications/DP17331
    Download Restriction: CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or

    for a different version of it.

    Other versions of this item:

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. is not listed on IDEAS
    2. Dylong, Patrick & Setzepfand, Paul & Uebelmesser, Silke, 2024. "Priming attitudes toward immigrants: Implications for migration research and survey design," European Journal of Political Economy, Elsevier, vol. 85(C).
    3. Bobasu, Alina & Ciccarelli, Matteo & Notarpietro, Alessandro & Ambrocio, Gene & Auer, Simone & Bonfim, Diana & Bottero, Margherita & Brázdik, František & Buss, Ginters & Byrne, David & Casalis, André , 2025. "Monetary policy transmission: a reference guide through ESCB models and empirical benchmarks," Occasional Paper Series 377, European Central Bank.
    4. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    5. Fang, Ximeng & Innocenti, Stefania, 2023. "Increasing the acceptability of carbon taxation: The role of social norms and economic reasoning," INET Oxford Working Papers 2023-25, Institute for New Economic Thinking at the Oxford Martin School, University of Oxford.
    6. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    7. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    8. Garg, Prashant & Fetzer, Thiemo, 2024. "Causal Claims in Economics," I4R Discussion Paper Series 183, The Institute for Replication (I4R).
    9. repec:osf:osfxxx:jq57n_v1 is not listed on IDEAS
    10. Burro, Giovanni & Castagnetti, Alessandro, 2022. "Will I tell you that you are smart (dumb)? Deceiving Others about their IQ or about a Random Draw," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    11. Rémi Suchon & Vincent Theroude, 2025. "Cooperation across the inequality divide: the effect of the magnitude of inequality," Working Papers halshs-05042251, HAL.
    12. Bruno Ferman & Lucas Finamor, 2025. "There must be an error here! Experimental evidence on coding errors' biases," Papers 2508.20069, arXiv.org, revised Sep 2025.
    13. repec:osf:osfxxx:u4vgs_v1 is not listed on IDEAS
    14. Costanza Naguib, 2025. "Does single-blind review encourage or discourage p-hacking?," Diskussionsschriften dp2504, Universitaet Bern, Departement Volkswirtschaft.
    15. Helena Antonie Baier & Philipp Lergetporer & Thomas Rittmannsberger, 2025. "Firms’ expectations about skill shortages," Small Business Economics, Springer, vol. 65(2), pages 1095-1112, August.
    16. Francesco Capozza & Krishna Srinivasan & Mattie Toma, 2025. "Science by Consensus: Eliciting Citizens’ and Experts’ R&D Spending Priorities," CESifo Working Paper Series 12235, CESifo.
    17. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    18. Fischer, Mira & Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Zeidler, Helen, 2024. "The E-word – On the public acceptance of experiments," Economics Letters, Elsevier, vol. 235(C).
    19. Costanza Naguib, 2024. "P-hacking and Significance Stars," Diskussionsschriften dp2409, Universitaet Bern, Departement Volkswirtschaft.
    20. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony & Wright, Taylor, 2025. "Media Stars: Statistical Significance and Research Impact," IZA Discussion Papers 18034, Institute of Labor Economics (IZA).
    21. Sebastian Gechert & Bianka Mey & Matej Opatrny & Tomas Havranek & T. D. Stanley & Pedro R. D. Bom & Hristos Doucouliagos & Philipp Heimberger & Zuzana Irsova & Heiko J. Rachinger, 2025. "Conventional wisdom, meta‐analysis, and research revision in economics," Journal of Economic Surveys, Wiley Blackwell, vol. 39(3), pages 980-999, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cpr:ceprdp:17331. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://www.cepr.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.