IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2205.10478.html
   My bibliography  Save this paper

The Power of Prognosis: Improving Covariate Balance Tests with Outcome Information

Author

Listed:
  • Clara Bicalho
  • Adam Bouyamourn
  • Thad Dunning

Abstract

Scholars frequently use covariate balance tests to test the validity of natural experiments and related designs. Unfortunately, when measured covariates are unrelated to potential outcomes, balance is uninformative about key identification conditions. We show that balance tests can then lead to erroneous conclusions. To build stronger tests, researchers should identify covariates that are jointly predictive of potential outcomes; formally measure and report covariate prognosis; and prioritize the most individually informative variables in tests. Building on prior research on ``prognostic scores," we develop bootstrap balance tests that upweight covariates associated with the outcome. We adapt this approach for regression-discontinuity designs and use simulations to compare weighting methods based on linear regression and more flexible methods, including machine learning. The results show how prognosis weighting can avoid both false negatives and false positives. To illustrate key points, we study empirical examples from a sample of published studies, including an important debate over close elections.

Suggested Citation

  • Clara Bicalho & Adam Bouyamourn & Thad Dunning, 2022. "The Power of Prognosis: Improving Covariate Balance Tests with Outcome Information," Papers 2205.10478, arXiv.org, revised Oct 2025.
    • Handle: RePEc:arx:papers:2205.10478
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2205.10478
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Caughey, Devin & Sekhon, Jasjeet S., 2011. "Elections and the Regression Discontinuity Design: Lessons from Close U.S. House Races, 1942–2008," Political Analysis, Cambridge University Press, vol. 19(4), pages 385-408.
    2. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    3. Christopher R. Genovese & Kathryn Roeder & Larry Wasserman, 2006. "False discovery control with p-value weighting," Biometrika, Biometrika Trust, vol. 93(3), pages 509-524, September.
    4. Erin Hartman & F. Daniel Hidalgo, 2018. "An Equivalence Approach to Balance and Placebo Tests," American Journal of Political Science, John Wiley & Sons, vol. 62(4), pages 1000-1013, October.
    5. Dunning,Thad, 2012. "Natural Experiments in the Social Sciences," Cambridge Books, Cambridge University Press, number 9781107017665, November.
    6. Dunning,Thad, 2012. "Natural Experiments in the Social Sciences," Cambridge Books, Cambridge University Press, number 9781107698000, November.
    7. Kost, James T. & McDermott, Michael P., 2002. "Combining dependent P-values," Statistics & Probability Letters, Elsevier, vol. 60(2), pages 183-190, November.
    8. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    9. Hartman, Erin, 2021. "Equivalence Testing for Regression Discontinuity Designs," Political Analysis, Cambridge University Press, vol. 29(4), pages 505-521, October.
    10. Ben B. Hansen, 2008. "The prognostic analogue of the propensity score," Biometrika, Biometrika Trust, vol. 95(2), pages 481-488.
    11. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    12. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gregory J. Wawro & Ira Katznelson, 2020. "American political development and new challenges of causal inference," Public Choice, Springer, vol. 185(3), pages 299-314, December.
    2. Blair, Graeme & Cooper, Jasper & Coppock, Alexander & Humphreys, Macartan, 2019. "Declaring and Diagnosing Research Designs," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 113(3), pages 838-859.
    3. Parker Hevron, 2018. "Judicialization and Its Effects: Experiments as a Way Forward," Laws, MDPI, vol. 7(2), pages 1-21, May.
    4. Haoge Chang & Joel Middleton & P. M. Aronow, 2021. "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials," Papers 2110.08425, arXiv.org, revised Oct 2021.
    5. Arzi Adbi, 2023. "Financial Sustainability of For-Profit Versus Non-Profit Microfinance Organizations Following a Scandal," Journal of Business Ethics, Springer, vol. 188(1), pages 57-74, November.
    6. Adel Daoud, 2020. "The wealth of nations and the health of populations: A quasi-experimental design of the impact of sovereign debt crises on child mortality," Papers 2012.14941, arXiv.org.
    7. Anustubh Agnihotri & Rahul Verma, 2016. "Design-based Approach in Social Science Research," Studies in Indian Politics, , vol. 4(2), pages 241-248, December.
    8. Adam Ploszaj, 2025. "Air travel and research collaboration: a quasi-experimental insight," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(4), pages 2167-2183, April.
    9. Andrew Bertoli & Allan Dafoe & Robert F. Trager, 2019. "Is There a War Party? Party Change, the Left–Right Divide, and International Conflict," Journal of Conflict Resolution, Peace Science Society (International), vol. 63(4), pages 950-975, April.
    10. Ian D. Gow & David F. Larcker & Peter C. Reiss, 2016. "Causal Inference in Accounting Research," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 54(2), pages 477-523, May.
    11. Aaron Reeves & Martin McKee & Johan Mackenbach & Margaret Whitehead & David Stuckler, 2017. "Introduction of a National Minimum Wage Reduced Depressive Symptoms in Low‐Wage Workers: A Quasi‐Natural Experiment in the UK," Health Economics, John Wiley & Sons, Ltd., vol. 26(5), pages 639-655, May.
    12. Shige Song & Lu Zheng, 2016. "The impact of sent-down movement on Chinese women's age at first marriage," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 34(28), pages 797-826.
    13. Arzi Adbi & Chirantan Chatterjee & Matej Drev & Anant Mishra, 2019. "When the Big One Came: A Natural Experiment on Demand Shock and Market Structure in India's Influenza Vaccine Markets," Production and Operations Management, Production and Operations Management Society, vol. 28(4), pages 810-832, April.
    14. Reeves, Aaron & McKee, Martin & Mackenbach, Johan & Whitehead, Margaret & Stuckler, David, 2017. "Introduction of a national minimum wage reduceddepressive symptoms in low-wage workers:a quasi-natural experiment in the UK," LSE Research Online Documents on Economics 66485, London School of Economics and Political Science, LSE Library.
    15. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.
    16. Rocio Titiunik, 2020. "Natural Experiments," Papers 2002.00202, arXiv.org.
    17. Kirk Bansak, 2021. "Estimating causal moderation effects with randomized treatments and non‐randomized moderators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(1), pages 65-86, January.
    18. Arzi Adbi & Chirantan Chatterjee & Anant Mishra, 2022. "How Do MNEs and Domestic Firms Respond Locally to a Global Demand Shock? Evidence from a Pandemic," Management Science, INFORMS, vol. 68(12), pages 9003-9025, December.
    19. Richard Aviles-Lopez & Juan de Dios Luna del Castillo & Miguel Ángel Montero-Alonso, 2023. "Exploratory Matching Model Search Algorithm (EMMSA) for Causal Analysis: Application to the Cardboard Industry," Mathematics, MDPI, vol. 11(21), pages 1-34, October.
    20. Marie Bjørneby & Annette Alstadsæter & Kjetil Telle, 2018. "Collusive tax evasion by employers and employees. Evidence from a randomized fi eld experiment in Norway," Discussion Papers 891, Statistics Norway, Research Department.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2205.10478. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.