IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/34050.html
   My bibliography  Save this paper

Beyond Bonferroni: Hierarchical Multiple Testing in Empirical Research

Author

Listed:
  • Sebastian Calónico
  • Sebastian Galiani

Abstract

This paper offers a general, practical review of methods for dealing with multiple hypothesis testing, aimed at applied researchers who must draw inference on multiple outcomes, subgroups, or model specifications but often lack clear guidance on how to proceed. We distill key lessons from the methodological literature, explain why several common practices are flawed, and clarify the guarantees provided by alternative approaches. Beyond standard corrections such as Bonferroni, Holm, and false discovery rate controls, we emphasize the value of hierarchical procedures that exploit causal or logical structure to deliver more powerful and interpretable results. By linking these tools to real applications, we translate methodological advances into accessible guidance and provide researchers with a clear roadmap for producing credible, transparent, and policy-relevant empirical work.

Suggested Citation

  • Sebastian Calónico & Sebastian Galiani, 2025. "Beyond Bonferroni: Hierarchical Multiple Testing in Empirical Research," NBER Working Papers 34050, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:34050
    Note: DEV
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w34050.pdf
    Download Restriction: Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or

    for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Galiani, Sebastián & Gertler, Paul J. & Undurraga, Raimundo & Cooper, Ryan & Martínez, Sebastián & Ross, Adam, 2017. "Shelter from the storm: Upgrading housing infrastructure in Latin American slums," Journal of Urban Economics, Elsevier, vol. 98(C), pages 187-213.
    2. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    3. Matias D. Cattaneo & Sebastian Galiani & Paul J. Gertler & Sebastian Martinez & Rocio Titiunik, 2009. "Housing, Health, and Happiness," American Economic Journal: Economic Policy, American Economic Association, vol. 1(1), pages 75-105, February.
    4. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    5. Miguel Angel Carpio & Lucero Gomez & Pablo Lavado, 2025. "The Impact of Social Health Insurance on Student Performance: Evidence From an RDD in Peru," Health Economics, John Wiley & Sons, Ltd., vol. 34(7), pages 1309-1325, July.
    6. Sebastian Galiani & Paul Gertler & Nicolas Ajzenman & Alexandra Orsola‐Vidal, 2016. "Promoting Handwashing Behavior: The Effects of Large‐scale Community and School‐level Interventions," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1545-1559, December.
    7. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    8. Sebastian Galiani & Martín A. Rossi & Ernesto Schargrodsky, 2011. "Conscription and Crime: Evidence from the Argentine Draft Lottery," American Economic Journal: Applied Economics, American Economic Association, vol. 3(2), pages 119-136, April.
    9. Alwyn Young, 2019. "Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 557-598.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cardim, Joana & Molina-Millán, Teresa & Vicente, Pedro C., 2023. "Can technology improve the classroom experience in primary education? An African experiment on a worldwide program," Journal of Development Economics, Elsevier, vol. 164(C).
    2. Henning Hermesifo & Philipp Lergetporer & Frauke Peter & Simon Wiederhold, 2025. "Application Barriers and the Socioeconomic Gap in Child Care Enrollment," Journal of the European Economic Association, European Economic Association, vol. 23(3), pages 1133-1172.
    3. Lubega, Patrick & Nakakawa, Frances & Narciso, Gaia & Newman, Carol & Kaaya, Archileo N. & Kityo, Cissy & Tumuhimbise, Gaston A., 2021. "Body and mind: Experimental evidence from women living with HIV," Journal of Development Economics, Elsevier, vol. 150(C).
    4. Sevim, Dilek & Baranov, Victoria & Bhalotra, Sonia R. & Maselko, Joanna & Biroli, Pietro, 2023. "Socioemotional Skills in Early Childhood: Evidence from a Maternal Psychosocial Intervention," IZA Discussion Papers 15925, Institute of Labor Economics (IZA).
    5. Hermes, Henning & Lergetporer, Philipp & Peter, Frauke & Wiederhold, Simon, 2021. "Behavioral Barriers and the Socioeconomic Gap in Child Care Enrollment," IZA Discussion Papers 14698, Institute of Labor Economics (IZA).
    6. Abu Siddique & Tabassum Rahman & Debayan Pakrashi & Asad Islam & Firoz Ahmed, 2020. "Raising COVID-19 Awareness in Rural Communities: A Randomized Experiment in Bangladesh and India," Munich Papers in Political Economy 09, Munich School of Politics and Public Policy and the School of Management at the Technical University of Munich.
    7. Nava Ashraf & Oriana Bandiera & Edward Davenport & Scott S. Lee, 2020. "Losing Prosociality in the Quest for Talent? Sorting, Selection, and Productivity in the Delivery of Public Services," American Economic Review, American Economic Association, vol. 110(5), pages 1355-1394, May.
    8. Rafkin, Charlie & Shreekumar, Advik & Vautrey, Pierre-Luc, 2021. "When guidance changes: Government stances and public beliefs," Journal of Public Economics, Elsevier, vol. 196(C).
    9. Anell, Anders & Dietrichson, Jens & Ellegård, Lina Maria & Kjellsson, Gustav, 2021. "Information, switching costs, and consumer choice: Evidence from two randomised field experiments in Swedish primary health care," Journal of Public Economics, Elsevier, vol. 196(C).
    10. Sevim, Dilek & Baranov, Victoria & Bhalotra, Sonia & Maselko, Joanna & Biroli, Pietro, 2023. "Trajectories of Early Childhood Skill Development and Maternal Mental Health," The Warwick Economics Research Paper Series (TWERPS) 1469, University of Warwick, Department of Economics.
    11. Laura Jaitman, 2015. "Urban infrastructure in Latin America and the Caribbean: public policy priorities," Latin American Economic Review, Springer;Centro de Investigaciòn y Docencia Económica (CIDE), vol. 24(1), pages 1-57, December.
    12. Jun Li, 2022. "Value‐Based Payments in Health Care: Evidence from a Nationwide Randomized Experiment in the Home Health Sector," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 41(4), pages 1090-1117, September.
    13. Adhvaryu, Achyuta & Nyshadham, Anant & Xu, Huayu, 2023. "Hostel takeover: Living conditions, reference dependence, and the well-being of migrant workers," Journal of Public Economics, Elsevier, vol. 226(C).
    14. Jeffrey D. Michler & Anna Josephson, 2022. "Recent developments in inference: practicalities for applied economics," Chapters, in: A Modern Guide to Food Economics, chapter 11, pages 235-268, Edward Elgar Publishing.
    15. Siddique, Abu & Islam, Asad & Mozumder, Tanvir Ahmed & Rahman, Tabassum & Shatil, Tanvir, 2022. "Forced Displacement, Mental Health, and Child Development: Evidence from the Rohingya Refugees," SocArXiv b4fc7, Center for Open Science.
    16. Michael Vlassopoulos & Abu Siddique & Tabassum Rahman & Debayan Pakrashi & Asad Islam & Firoz Ahmed, 2024. "Improving Women's Mental Health during a Pandemic," American Economic Journal: Applied Economics, American Economic Association, vol. 16(2), pages 422-455, April.
    17. Sevim, Dilek & Baranov, Victoria & Bhalotra, Sonia & Maselko, Joanna & Biroli, Pietro, 2023. "Trajectories of Early Childhood Skill Development and Maternal Mental Health," CAGE Online Working Paper Series 674, Competitive Advantage in the Global Economy (CAGE).
    18. Aksoy, Cevat Giray & Poutvaara, Panu & Schikora, Felicitas, 2023. "First time around: Local conditions and multi-dimensional integration of refugees," Journal of Urban Economics, Elsevier, vol. 137(C).
    19. Diegmann, André & Pohlan, Laura & Weber, Andrea, 2024. "Do Politicians Affect Firm Outcomes? Evidence from Connections to the German Federal Parliament," IZA Discussion Papers 17031, Institute of Labor Economics (IZA).
    20. Eichengreen, Barry & Aksoy, Cevat Giray & Saka, Orkun, 2021. "Revenge of the experts: Will COVID-19 renew or diminish public trust in science?," Journal of Public Economics, Elsevier, vol. 193(C).

    More about this item

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • C11 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Bayesian Analysis: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:34050. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.