IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/31.html
   My bibliography  Save this paper

Selective Reporting of Placebo Tests in Top Economics Journals

Author

Listed:
  • Dreber, Anna
  • Johannesson, Magnus
  • Yang, Yifan

Abstract

Placebo tests, where a null result is used to support the validity of the research design, is common in economics. Such tests provide an incentive to underreport statistically significant tests, a form of reversed p-hacking. Based on a pre-registered analysis plan, we test for such underreporting in all papers meeting our inclusion criteria (n=377) published in 11 top economics journals between 2009-2021. If the null hypothesis is true in all tests, 2.5% of them should be statistically significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]). Our results provide strong evidence of selective underreporting of statistically significant placebo tests in top economics journals.

Suggested Citation

  • Dreber, Anna & Johannesson, Magnus & Yang, Yifan, 2023. "Selective Reporting of Placebo Tests in Top Economics Journals," I4R Discussion Paper Series 31, The Institute for Replication (I4R).
    • Handle: RePEc:zbw:i4rdps:31
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/271197/1/I4R-DP031.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Quentin Lippmann & Alexandre Georgieff & Claudia Senik, 2020. "Undoing Gender with Institutions: Lessons from the German Division and Reunification," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1445-1470.
    2. Emmanuel Saez & Manos Matsaganis & Panos Tsakloglou, 2012. "Earnings Determination and Taxes: Evidence From a Cohort-Based Payroll Tax Reform in Greece," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 493-533.
    3. Oana Borcan & Mikael Lindahl & Andreea Mitrut, 2017. "Fighting Corruption in Education: What Works and Who Benefits?," American Economic Journal: Economic Policy, American Economic Association, vol. 9(1), pages 180-209, February.
    4. David Card & Alexandre Mas & Enrico Moretti & Emmanuel Saez, 2012. "Inequality at Work: The Effect of Peer Salaries on Job Satisfaction," American Economic Review, American Economic Association, vol. 102(6), pages 2981-3003, October.
    5. Pedro S. Martins, 2009. "Dismissals for Cause: The Difference That Just Eight Paragraphs Can Make," Journal of Labor Economics, University of Chicago Press, vol. 27(2), pages 257-279, April.
    6. Derek Neal & Diane Whitmore Schanzenbach, 2010. "Left Behind by Design: Proficiency Counts and Test-Based Accountability," The Review of Economics and Statistics, MIT Press, vol. 92(2), pages 263-283, May.
    7. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    8. Quentin Lippmann & Alexandre Georgieff & Claudia Senik, 2020. "Undoing Gender with Institutions: Lessons from the German Division and Reunification," Post-Print halshs-03247392, HAL.
    9. Bryan Bollinger & Jesse Burkhardt & Kenneth T. Gillingham, 2020. "Peer Effects in Residential Water Conservation: Evidence from Migration," American Economic Journal: Economic Policy, American Economic Association, vol. 12(3), pages 107-133, August.
    10. Quentin Lippmann & Alexandre Georgieff & Claudia Senik, 2020. "Undoing Gender with Institutions: Lessons from the German Division and Reunification," PSE-Ecole d'économie de Paris (Postprint) halshs-03247392, HAL.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Estefanía Galván, 2022. "Gender Identity and Quality of Employment," Economica, London School of Economics and Political Science, vol. 89(354), pages 409-436, April.
    2. Jaanika Meriküll & Maryna Tverdostup, 2020. "The Gap That Survived The Transition: The Gender Wage Gap Over Three Decades In Estonia," University of Tartu - Faculty of Economics and Business Administration Working Paper Series 127, Faculty of Economics and Business Administration, University of Tartu (Estonia).
    3. Natalia Danzer & Mathias Huebener & Astrid Pape & C. Katharina Spieß & Nico A. Siegel & Gert G. Wagner, 2021. "Cracking under Pressure? Gender Role Attitudes toward Maternal Employment in Times of a Pandemic," Discussion Papers of DIW Berlin 1951, DIW Berlin, German Institute for Economic Research.
    4. Pronkina, Elizaveta & Berniell, Inés & Fawaz, Yarine & Laferrère, Anne & Mira, Pedro, 2023. "The COVID-19 curtain: Can past communist regimes explain the vaccination divide in Europe?," Social Science & Medicine, Elsevier, vol. 321(C).
    5. Dilmaghani, Maryam, 2021. "The gender gap in competitive chess across countries: Commanding queens in command economies," Journal of Comparative Economics, Elsevier, vol. 49(2), pages 425-441.
    6. Friedman-Sokuler, Naomi & Senik, Claudia, 2020. "From Pink-Collar to Lab Coat: Cultural Persistence and Diffusion of Socialist Gender Norms," IZA Discussion Papers 13385, Institute of Labor Economics (IZA).
    7. José-Ignacio Antón & Rafael Grande & Rafael Muñoz de Bustillo & Fernando Pinto, 2023. "Gender Gaps in Working Conditions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 166(1), pages 53-83, February.
    8. Sascha O. Becker & Lukas Mergele & Ludger Woessmann, 2020. "The Separation and Reunification of Germany: Rethinking a Natural Experiment Interpretation of the Enduring Effects of Communism," Journal of Economic Perspectives, American Economic Association, vol. 34(2), pages 143-171, Spring.
    9. Juliane Hennecke & Astrid Pape, 2022. "Suddenly a stay-at-home dad? Short- and long-term consequences of fathers’ job loss on time investment in the household," Review of Economics of the Household, Springer, vol. 20(2), pages 579-607, June.
    10. Estefanía Galván & Cecilia García-Peñalosa, 2021. "Interactions amongst gender norms: Evidence from US couples," Documentos de Trabajo (working papers) 21-15, Instituto de Economía - IECON.
    11. Bondar Mariia & Fuchs-Schündeln Nicola, 2023. "Good Bye Lenin Revisited: East-West Preferences Three Decades after German Reunification," German Economic Review, De Gruyter, vol. 24(1), pages 97-119, February.
    12. Patricia Cortés & Gizem Koşar & Jessica Pan & Basit Zafar, 2022. "Should Mothers Work? How Perceptions of the Social Norm Affect Individual Attitudes Toward Work in the U.S," NBER Working Papers 30606, National Bureau of Economic Research, Inc.
    13. Wei Si, 2022. "Higher education expansion and gender norms: evidence from China," Journal of Population Economics, Springer;European Society for Population Economics, vol. 35(4), pages 1821-1858, October.
    14. Karin Hederos & Anders Stenberg, 2022. "Gender identity and relative income within households: evidence from Sweden," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 744-772, July.
    15. Batinti, Alberto & Costa-Font, Joan, 2022. "Does democracy make taller men? Cross-country European evidence," Economics & Human Biology, Elsevier, vol. 45(C).
    16. Tattarini, Giulia & Grotti, Raffaele, 2022. "Gender roles and selection mechanisms across contexts: a comparative analysis of the relationship between unemployment, self‐perceived health and gender," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 44(3), pages 641-662.
    17. de la Vega, Noa, 2022. "The differential effect of childbirth on men's and women's careers," Labour Economics, Elsevier, vol. 78(C).
    18. Maéva Doumbia & Marion Goussé, 2021. "Gender identity and relative income within households: Evidence from Canada," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 54(4), pages 1667-1683, November.
    19. Batinti, Alberto & Costa-Font, Joan, 2022. "Does democracy make taller men? Cross-country European evidence," LSE Research Online Documents on Economics 113745, London School of Economics and Political Science, LSE Library.
    20. Sascha O. Becker & Lukas Mergele & Ludger Wößmann, 2021. "Es liegt nicht alles am Sozialismus — über Ost-West-Unterschiede und ihre Ursprünge [It Is Not All Because of Socialism — On East-West Differences and Their Origins]," Wirtschaftsdienst, Springer;ZBW - Leibniz Information Centre for Economics, vol. 101(1), pages 32-36, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:31. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.