IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2604.00504.html

Conformal Inference for Experimental Attrition in Social Science Research

Author

Listed:
  • Xiangyu Song

Abstract

Attrition in survey and field experiments presents a challenge for social science research. Common approaches to deal with this problem -- such as complete case analysis, multiple imputation, and weighting methods -- rely on strong assumptions that may not hold in practice. This paper introduces a new method that combines recent advances in statistical inference with established tools for handling missing data. The approach produces prediction intervals for treatment effects that are both robust and precise. Evidence from simulation studies shows that the method achieves better coverage and produces narrower intervals than common alternatives. The reanalysis of two recently published experiment studies illustrates how this framework allows researchers to compare treatment effects across participants who remain in the study, those who drop out, and the full sample. Taken together, these results highlight how the proposed approach provides a stronger foundation for causal inference in the presence of attrition.

Suggested Citation

  • Xiangyu Song, 2026. "Conformal Inference for Experimental Attrition in Social Science Research," Papers 2604.00504, arXiv.org, revised Apr 2026.
  • Handle: RePEc:arx:papers:2604.00504
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2604.00504
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hausman, Jerry A & Wise, David A, 1979. "Attrition Bias in Experimental and Panel Data: The Gary Income Maintenance Experiment," Econometrica, Econometric Society, vol. 47(2), pages 455-473, March.
    2. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2021. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1849-1864, October.
    3. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    4. Horowitz, Joel L. & Manski, Charles F., 1998. "Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and imputations," Journal of Econometrics, Elsevier, vol. 84(1), pages 37-58, May.
    5. Honaker, James & King, Gary & Blackwell, Matthew, 2011. "Amelia II: A Program for Missing Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 45(i07).
    6. Kalla, Joshua L. & Broockman, David E., 2018. "The Minimal Persuasive Effects of Campaign Contact in General Elections: Evidence from 49 Field Experiments," American Political Science Review, Cambridge University Press, vol. 112(1), pages 148-166, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicolaj N. Mühlbach, 2020. "Tree-based Synthetic Control Methods: Consequences of moving the US Embassy," CREATES Research Papers 2020-04, Department of Economics and Business Economics, Aarhus University.
    2. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.
    3. Martin Huber, 2012. "Identification of Average Treatment Effects in Social Experiments Under Alternative Forms of Attrition," Journal of Educational and Behavioral Statistics, , vol. 37(3), pages 443-474, June.
    4. Kapteyn, Arie & Michaud, Pierre-Carl & Smith, James P. & van Soest, Arthur, 2006. "Effects of Attrition and Non-Response in the Health and Retirement Study," IZA Discussion Papers 2246, IZA Network @ LISER.
    5. John Fitzgerald & Peter Gottschalk & Robert Moffitt, 1998. "An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 251-299.
    6. Martin Huber, 2010. "Identification of average treatment effects in social experiments under different forms of attrition," University of St. Gallen Department of Economics working paper series 2010 2010-22, Department of Economics, University of St. Gallen.
    7. Nicolaj S{o}ndergaard Muhlbach & Mikkel Slot Nielsen, 2019. "Tree-based Synthetic Control Methods: Consequences of moving the US Embassy," Papers 1909.03968, arXiv.org, revised Feb 2021.
    8. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    9. Victor Chernozhukov & Mert Demirer & Esther Duflo & Iván Fernández‐Val, 2025. "Fisher–Schultz Lecture: Generic Machine Learning Inference on Heterogeneous Treatment Effects in Randomized Experiments, With an Application to Immunization in India," Econometrica, Econometric Society, vol. 93(4), pages 1121-1164, July.
    10. Kapteyn, Arie & Michaud, Pierre-Carl & Smith, James P. & van Soest, Arthur, 2006. "Effects of Attrition and Non-Response in the Health and Retirement Study," IZA Discussion Papers 2246, Institute for the Study of Labor (IZA).
    11. Inkmann, J., 2005. "Inverse Probability Weighted Generalised Empirical Likelihood Estimators : Firm Size and R&D Revisited," Other publications TiSEM c39cff1f-16c1-4446-a83f-c, Tilburg University, School of Economics and Management.
    12. Christopher J. Gerry & Georgios Papadopoulos, 2015. "Sample attrition in the RLMS, 2001–10," The Economics of Transition, The European Bank for Reconstruction and Development, vol. 23(2), pages 425-468, April.
    13. Yan Zhang & Zudi Lu, 2024. "A Time Series Synthetic Control Causal Evaluation of the UK’s Mini-Budget Policy on Stock Market," Mathematics, MDPI, vol. 12(20), pages 1-25, October.
    14. Michela Bia & Martin Huber & Lukáš Lafférs, 2024. "Double Machine Learning for Sample Selection Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 42(3), pages 958-969, July.
    15. Andrew M. Jones & Xander Koolman & Nigel Rice, 2006. "Health‐related non‐response in the British Household Panel Survey and European Community Household Panel: using inverse‐probability‐weighted estimators in non‐linear models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 169(3), pages 543-569, July.
    16. Keisuke Hirano & Guido W. Imbens & Geert Ridder & Donald B. Rubin, 2001. "Combining Panel Data Sets with Attrition and Refreshment Samples," Econometrica, Econometric Society, vol. 69(6), pages 1645-1659, November.
    17. Emre Ekinci, 2009. "Dealing with Attrition When Refreshment Samples are Available: An Application to the Turkish Household Labor Force Survey," 2009 Meeting Papers 353, Society for Economic Dynamics.
    18. Vivek F. Farias & Andrew A. Li & Tianyi Peng, 2021. "Learning Treatment Effects in Panels with General Intervention Patterns," Papers 2106.02780, arXiv.org, revised Mar 2023.
    19. Inkmann, Joachim, 2001. "Accounting for Nonresponse Heterogeneity in Panel Data," CoFE Discussion Papers 01/03, University of Konstanz, Center of Finance and Econometrics (CoFE).
    20. Douglas Kiarelly Godoy de Araujo, 2024. "Synthetic controls with machine learning: application on the effect of labour deregulation on worker productivity in Brazil," BIS Working Papers 1181, Bank for International Settlements.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2604.00504. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.