IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/227.html
   My bibliography  Save this paper

Opinion on the Replication Debate over Heyes and Saberian (2019)

Author

Listed:
  • Roodman, David

Abstract

Heyes and Saberian (2019) finds that U.S. immigration judges are less likely to grant asylum in cases heard on warmer days. Spamann (2022) corrects errors in that paper, enlarges the sample, proposes additional revisions, and strongly challenges the conclusion. In a rejoinder, Heyes and Saberian (2022) incorporates many of these comments, yet maintains that "results…are qualitatively un-changed." Experimenting with a new academic-literary form, I review the case as a judge might, to offer a take that is more independent and legible than the partisans can offer. I agree with Spamann (2022): the only viable explanation for the combined evidence is publication bias or other forms of result filtration.

Suggested Citation

  • Roodman, David, 2025. "Opinion on the Replication Debate over Heyes and Saberian (2019)," I4R Discussion Paper Series 227, The Institute for Replication (I4R).
    • Handle: RePEc:zbw:i4rdps:227
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/316399/1/I4R-DP227.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joshua B. Fischman, 2014. "Measuring Inconsistency, Indeterminacy, and Error in Adjudication," American Law and Economics Review, American Law and Economics Association, vol. 16(1), pages 40-85.
    2. Thompson, Samuel B., 2011. "Simple formulas for standard errors that cluster by both firm and time," Journal of Financial Economics, Elsevier, vol. 99(1), pages 1-10, January.
    3. Anthony Heyes & Nicholas Rivers & Brandon Schaufele, 2019. "Pollution and Politician Productivity: The Effect of PM on MPs," Land Economics, University of Wisconsin Press, vol. 95(2), pages 157-173.
    4. Denton, Frank T, 1985. "Data Mining as an Industry," The Review of Economics and Statistics, MIT Press, vol. 67(1), pages 124-127, February.
    5. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    6. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    7. repec:osf:metaar:a9vhr_v1 is not listed on IDEAS
    8. Abel Brodeur & Nikolai M. Cook & Jonathan S. Hartley & Anthony Heyes, 2024. "Do Preregistration and Preanalysis Plans Reduce p-Hacking and Publication Bias? Evidence from 15,992 Test Statistics and Suggestions for Improvement," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 2(3), pages 527-561.
    9. Anthony Heyes & Soodeh Saberian, 2022. "Correction to "Temperature and Decisions: Evidence from 207,000 Court Cases" and Reply to Spamann," American Economic Journal: Applied Economics, American Economic Association, vol. 14(4), pages 529-533, October.
    10. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    11. David Roodman, 2007. "The Anarchy of Numbers: Aid, Development, and Cross-Country Empirics," The World Bank Economic Review, World Bank, vol. 21(2), pages 255-277, May.
    12. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "A Proposed Specification Check for p-Hacking," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 66-69, May.
    13. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    14. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    15. James Archsmith & Anthony Heyes & Soodeh Saberian, 2018. "Air Quality and Error Quantity: Pollution and Performance in a High-Skilled, Quality-Focused Occupation," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 5(4), pages 827-863.
    16. MacKinnon, James G. & Webb, Matthew D., 2015. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Queen's Economics Department Working Papers 274639, Queen's University - Department of Economics.
    17. Evan Herrnstadt & Anthony Heyes & Erich Muehlegger & Soodeh Saberian, 2021. "Air Pollution and Criminal Activity: Microgeographic Evidence from Chicago," American Economic Journal: Applied Economics, American Economic Association, vol. 13(4), pages 70-100, October.
    18. Heyes, Anthony & Zhu, Mingying, 2019. "Air pollution as a cause of sleeplessness: Social media evidence from a panel of Chinese cities," Journal of Environmental Economics and Management, Elsevier, vol. 98(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    2. Luis Sarmiento & Adam Nowakowski, 2023. "Court Decisions and Air Pollution: Evidence from Ten Million Penal Cases in India," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 86(3), pages 605-644, November.
    3. Wang, Wenjie, 2021. "Wild Bootstrap for Instrumental Variables Regression with Weak Instruments and Few Clusters," MPRA Paper 106227, University Library of Munich, Germany.
    4. Webb, Matthew D., 2014. "Reworking Wild Bootstrap Based Inference for Clustered Errors," Queen's Economics Department Working Papers 274640, Queen's University - Department of Economics.
    5. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.
    6. Luis Sarmiento, 2022. "Air pollution and the productivity of high‐skill labor: evidence from court hearings," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(1), pages 301-332, January.
    7. Brodeur, Abel & Cook, Nikolai M. & Heyes, Anthony & Wright, Taylor, 2025. "Media Stars: Statistical Significance and Research Impact," I4R Discussion Paper Series 254, The Institute for Replication (I4R).
    8. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2021. "Wild Bootstrap and Asymptotic Inference With Multiway Clustering," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 505-519, March.
    9. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    10. Wang, Chunchao & Lin, Qianqian & Qiu, Yun, 2022. "Productivity loss amid invisible pollution," Journal of Environmental Economics and Management, Elsevier, vol. 112(C).
    11. Wang, Chunchao & Lin, Qianqian & Qiu, Yun, 2020. "Productivity Loss amid Invisible Pollution," GLO Discussion Paper Series 722, Global Labor Organization (GLO).
    12. Bellani, Luna & Ceolotto, Stefano & Elsner, Benjamin & Pestel, Nico, 2021. "Air Pollution Affects Decision-Making: Evidence from the Ballot Box," IZA Discussion Papers 14718, Institute of Labor Economics (IZA).
    13. Vu, Patrick, 2024. "Why are replication rates so low?," Journal of Econometrics, Elsevier, vol. 245(1).
    14. Lohmann, Paul M. & Gsottbauer, Elisabeth & You, Jing & Kontoleon, Andreas, 2023. "Air pollution and anti-social behaviour: Evidence from a randomised lab-in-the-field experiment," Social Science & Medicine, Elsevier, vol. 320(C).
    15. Nikolai Cook, Anthony Heyes, Nicholas Rivers, 2023. "Clean Air and Cognitive Productivity: Effect and Adaptation," LCERPA Working Papers bm0137, Laurier Centre for Economic Research and Policy Analysis.
    16. Aloys Prinz & David J. Richter, 2021. "Feinstaubbelastung und Lebenserwartung in Deutschland," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 15(3), pages 237-272, December.
    17. James G. MacKinnon & Morten {O}rregaard Nielsen & Matthew D. Webb, 2024. "Jackknife inference with two-way clustering," Papers 2406.08880, arXiv.org, revised Nov 2025.
    18. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    19. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    20. Su, Yuandong & Lu, Xinjie & Zeng, Qing & Huang, Dengshi, 2022. "Good air quality and stock market returns," Research in International Business and Finance, Elsevier, vol. 62(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:227. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.