IDEAS home Printed from https://ideas.repec.org/a/oup/econjl/v134y2024i657p193-219..html

The Null Result Penalty

Author

Listed:
  • Felix Chopra
  • Ingar Haaland
  • Christopher Roth
  • Andreas Stegmann

Abstract

We examine how the evaluation of research studies in economics depends on whether a study yielded a null result. Studies with null results are perceived to be less publishable, of lower quality, less important and less precisely estimated than studies with large and statistically significant results, even when holding constant all other study features, including the sample size and the precision of the estimates. The null result penalty is of similar magnitude among PhD students and journal editors. The penalty is larger when experts predict a large effect and when statistical uncertainty is communicated with p-values rather than standard errors. Our findings highlight the value of a pre-result review.

Suggested Citation

  • Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2024. "The Null Result Penalty," The Economic Journal, Royal Economic Society, vol. 134(657), pages 193-219.
    • Handle: RePEc:oup:econjl:v:134:y:2024:i:657:p:193-219.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/ej/uead060
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or

    for a different version of it.

    Other versions of this item:

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dylong, Patrick & Setzepfand, Paul & Uebelmesser, Silke, 2024. "Priming attitudes toward immigrants: Implications for migration research and survey design," European Journal of Political Economy, Elsevier, vol. 85(C).
    2. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    3. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    4. Garg, Prashant & Fetzer, Thiemo, 2024. "Causal Claims in Economics," I4R Discussion Paper Series 183, The Institute for Replication (I4R).
    5. repec:osf:osfxxx:jq57n_v1 is not listed on IDEAS
    6. repec:osf:osfxxx:u4vgs_v1 is not listed on IDEAS
    7. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    8. Costanza Naguib, 2025. "Does single-blind review encourage or discourage p-hacking?," Diskussionsschriften dp2504, Universitaet Bern, Departement Volkswirtschaft.
    9. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    10. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony & Wright, Taylor, 2025. "Media Stars: Statistical Significance and Research Impact," IZA Discussion Papers 18034, Institute of Labor Economics (IZA).
    11. Bobasu, Alina & Ciccarelli, Matteo & Notarpietro, Alessandro & Ambrocio, Gene & Auer, Simone & Bonfim, Diana & Bottero, Margherita & Brázdik, František & Buss, Ginters & Byrne, David & Casalis, André , 2025. "Monetary policy transmission: a reference guide through ESCB models and empirical benchmarks," Occasional Paper Series 377, European Central Bank.
    12. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    13. Fang, Ximeng & Innocenti, Stefania, 2023. "Increasing the acceptability of carbon taxation: The role of social norms and economic reasoning," INET Oxford Working Papers 2023-25, Institute for New Economic Thinking at the Oxford Martin School, University of Oxford.
    14. Burro, Giovanni & Castagnetti, Alessandro, 2022. "Will I tell you that you are smart (dumb)? Deceiving Others about their IQ or about a Random Draw," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    15. Rémi Suchon & Vincent Theroude, 2025. "Cooperation across the inequality divide: the effect of the magnitude of inequality," Working Papers halshs-05042251, HAL.
    16. Helena Antonie Baier & Philipp Lergetporer & Thomas Rittmannsberger, 2025. "Firms’ expectations about skill shortages," Small Business Economics, Springer, vol. 65(2), pages 1095-1112, August.
    17. Francesco Capozza & Krishna Srinivasan & Mattie Toma, 2025. "Science by Consensus: Eliciting Citizens’ and Experts’ R&D Spending Priorities," CESifo Working Paper Series 12235, CESifo.
    18. Fischer, Mira & Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Zeidler, Helen, 2024. "The E-word – On the public acceptance of experiments," Economics Letters, Elsevier, vol. 235(C).
    19. Costanza Naguib, 2024. "P-hacking and Significance Stars," Diskussionsschriften dp2409, Universitaet Bern, Departement Volkswirtschaft.
    20. Sebastian Gechert & Bianka Mey & Matej Opatrny & Tomas Havranek & T. D. Stanley & Pedro R. D. Bom & Hristos Doucouliagos & Philipp Heimberger & Zuzana Irsova & Heiko J. Rachinger, 2025. "Conventional wisdom, meta‐analysis, and research revision in economics," Journal of Economic Surveys, Wiley Blackwell, vol. 39(3), pages 980-999, July.

    More about this item

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. Meta-Research in Economics

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:econjl:v:134:y:2024:i:657:p:193-219.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press or the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/resssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.