IDEAS home Printed from https://ideas.repec.org/p/ucr/wpaper/201919.html
   My bibliography  Save this paper

Testing for Attrition Bias in Field Experiments

Author

Listed:
  • Sarojini Hirshleifer

    (Department of Economics, University of California Riverside)

  • Dalia Ghanem

    (UC Davis)

  • Karen Ortiz-Becerra

    (UC Davis)

Abstract

We approach attrition in field experiments with baseline outcome data as an identification problem in a panel model. A systematic review of the literature indicates that there is no consensus on how to test for attrition bias. We establish identifying assumptions for treatment effects for both the respondent subpopulation and the study population. We then derive their sharp testable implications on the baseline outcome distribution and propose randomization procedures to test them. We demonstrate that the most commonly used test does not control size in general when internal validity holds. Simulations and applications illustrate the empirical relevance of our analysis.

Suggested Citation

  • Sarojini Hirshleifer & Dalia Ghanem & Karen Ortiz-Becerra, 2019. "Testing for Attrition Bias in Field Experiments," Working Papers 201919, University of California at Riverside, Department of Economics, revised Aug 2019.
    • Handle: RePEc:ucr:wpaper:201919
    as

    Download full text from publisher

    File URL: https://economics.ucr.edu/repec/ucr/wpaper/201919.pdf
    File Function: First version, 2019
    Download Restriction: no
    File URL: https://economics.ucr.edu/repec/ucr/wpaper/201919R.pdf
    File Function: Revised version, 2019
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Tarek Azzam & Michael Bates & David Fairris, 2019. "Do Learning Communities Increase First Year College Retention? Testing Sample Selection and External Validity of Randomized Control Trials," Working Papers 202002, University of California at Riverside, Department of Economics.
    2. Molina Millán, Teresa & Macours, Karen, 2017. "Attrition in Randomized Control Trials: Using Tracking Information to Correct Bias," IZA Discussion Papers 10711, Institute of Labor Economics (IZA).
    3. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    4. Hausman, Jerry A & Wise, David A, 1979. "Attrition Bias in Experimental and Panel Data: The Gary Income Maintenance Experiment," Econometrica, Econometric Society, vol. 47(2), pages 455-473, March.
    5. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492, National Bureau of Economic Research, Inc.
    6. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    7. Patrick Kline & Andres Santos, 2013. "Sensitivity to missing data assumptions: Theory and an evaluation of the U.S. wage structure," Quantitative Economics, Econometric Society, vol. 4(2), pages 231-267, July.
    8. Ahn, Hyungtaik & Powell, James L., 1993. "Semiparametric estimation of censored selection models with a nonparametric selection mechanism," Journal of Econometrics, Elsevier, vol. 58(1-2), pages 3-29, July.
    9. Rachel Glennerster & Kudzai Takavarasha, 2013. "Running Randomized Evaluations: A Practical Guide," Economics Books, Princeton University Press, edition 1, number 10085.
    10. Joseph G. Altonji & Rosa L. Matzkin, 2005. "Cross Section and Panel Data Estimators for Nonseparable Models with Endogenous Regressors," Econometrica, Econometric Society, vol. 73(4), pages 1053-1102, July.
    11. Hoderlein, Stefan & White, Halbert, 2012. "Nonparametric identification in nonseparable panel data models with generalized fixed effects," Journal of Econometrics, Elsevier, vol. 168(2), pages 300-314.
    12. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    13. Dufour, Jean-Marie, 2006. "Monte Carlo tests with nuisance parameters: A general approach to finite-sample inference and nonstandard asymptotics," Journal of Econometrics, Elsevier, vol. 133(2), pages 443-477, August.
    14. Wooldridge, Jeffrey M., 1995. "Selection corrections for panel data models under conditional mean independence assumptions," Journal of Econometrics, Elsevier, vol. 68(1), pages 115-132, July.
    15. James J. Heckman, 1976. "Introduction to "Annals of Economic and Social Measurement, Volume 5, number 4"," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, National Bureau of Economic Research, Inc.
    16. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    17. Francis Vella, 1998. "Estimating Models with Sample Selection Bias: A Survey," Journal of Human Resources, University of Wisconsin Press, vol. 33(1), pages 127-169.
    18. Jean-Marie Dufour & Abdeljelil Farhat & Lucien Gardiol & Lynda Khalaf, 1998. "Simulation-based finite sample normality tests in linear regressions," Econometrics Journal, Royal Economic Society, vol. 1(Conferenc), pages 154-173.
    19. Teresa Molina Millán & Karen Macours, 2017. "Attrition in randomized control trials: Using tracking information to correct bias," FEUNL Working Paper Series novaf:wp1702, Universidade Nova de Lisboa, Faculdade de Economia.
    20. Bester, C. Alan & Hansen, Christian, 2009. "Identification of Marginal Effects in a Nonparametric Correlated Random Effects Model," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(2), pages 235-250.
    21. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    22. David S. Lee, 2009. "Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(3), pages 1071-1102.
    23. Ghanem, Dalia, 2017. "Testing identifying assumptions in nonseparable panel data models," Journal of Econometrics, Elsevier, vol. 197(2), pages 202-217.
    24. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tarek Azzam & Michael Bates & David Fairris, 2019. "Do Learning Communities Increase First Year College Retention? Testing Sample Selection and External Validity of Randomized Control Trials," Working Papers 202002, University of California at Riverside, Department of Economics.
    2. Fulya Ersoy, 2021. "Returns to effort: experimental evidence from an online language platform," Experimental Economics, Springer;Economic Science Association, vol. 24(3), pages 1047-1073, September.
    3. Guigonan S. Adjognon & Daan van Soest & Jonas Guthoff, 2021. "Reducing Hunger with Payments for Environmental Services (PES): Experimental Evidence from Burkina Faso," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(3), pages 831-857, May.
    4. Ben Weidmann & Luke Miratrix, 2021. "Missing, presumed different: Quantifying the risk of attrition bias in education evaluations," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 732-760, April.
    5. Rafkin, Charlie & Shreekumar, Advik & Vautrey, Pierre-Luc, 2021. "When guidance changes: Government stances and public beliefs," Journal of Public Economics, Elsevier, vol. 196(C).
    6. Annie Alcid & Erwin Bulte & Robert Lensink & Aussi Sayinzoga & Mark Treurniet, 2023. "Short- and Medium-term Impacts of Employability Training: Evidence from a Randomised Field Experiment in Rwanda," Journal of African Economies, Centre for the Study of African Economies, vol. 32(3), pages 296-328.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dalia Ghanem & Sarojini Hirshleifer & Karen Ortiz-Becerra, 2019. "Testing Attrition Bias in Field Experiments," Working Papers 202218, University of California at Riverside, Department of Economics, revised Oct 2022.
    2. Guigonan S. Adjognon & Daan van Soest & Jonas Guthoff, 2021. "Reducing Hunger with Payments for Environmental Services (PES): Experimental Evidence from Burkina Faso," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(3), pages 831-857, May.
    3. Martin Huber, 2014. "Treatment Evaluation in the Presence of Sample Selection," Econometric Reviews, Taylor & Francis Journals, vol. 33(8), pages 869-905, November.
    4. Martin Huber, 2012. "Identification of Average Treatment Effects in Social Experiments Under Alternative Forms of Attrition," Journal of Educational and Behavioral Statistics, , vol. 37(3), pages 443-474, June.
    5. Irene Botosaru & Chris Muris, 2017. "Binarization for panel models with fixed effects," CeMMAP working papers 31/17, Institute for Fiscal Studies.
    6. Glenn W. Harrison & Morten I. Lau & Hong Il Yoo, 2020. "Risk Attitudes, Sample Selection, and Attrition in a Longitudinal Field Experiment," The Review of Economics and Statistics, MIT Press, vol. 102(3), pages 552-568, July.
    7. Lewbel, Arthur, 2007. "Endogenous selection or treatment model estimation," Journal of Econometrics, Elsevier, vol. 141(2), pages 777-806, December.
    8. Simon Calmar Andersen & Louise Beuchert & Phillip Heiler & Helena Skyt Nielsen, 2023. "A Guide to Impact Evaluation under Sample Selection and Missing Data: Teacher's Aides and Adolescent Mental Health," Papers 2308.04963, arXiv.org.
    9. Louise Laage, 2020. "A Correlated Random Coefficient Panel Model with Time-Varying Endogeneity," Papers 2003.09367, arXiv.org, revised Nov 2022.
    10. Fatouh, Mahmoud & Neamțu, Ioana & van Wijnbergen, Sweder, 2021. "Risk-taking and uncertainty: do contingent convertible (CoCo) bonds increase the risk appetite of banks?," Bank of England working papers 938, Bank of England.
    11. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    12. Dalia Ghanem & Pedro H. C. Sant'Anna & Kaspar Wüthrich, 2022. "Selection and Parallel Trends," CESifo Working Paper Series 9910, CESifo.
    13. Martin Huber & Anna Solovyeva, 2020. "Direct and Indirect Effects under Sample Selection and Outcome Attrition," Econometrics, MDPI, vol. 8(4), pages 1-25, December.
    14. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    15. Irene Botosaru & Chris Muris, 2022. "Identification of time-varying counterfactual parameters in nonlinear panel models," Papers 2212.09193, arXiv.org, revised Nov 2023.
    16. Mahmoud Fatouh & Ioana Neamtu & Sweder van Wijnbergen, 2022. "Risk-Taking, Competition and Uncertainty: Do Contingent Convertible (CoCo) Bonds Increase the Risk Appetite of Banks?," Tinbergen Institute Discussion Papers 22-017/IV, Tinbergen Institute.
    17. Nicoletti, Cheti, 2006. "Nonresponse in dynamic panel data models," Journal of Econometrics, Elsevier, vol. 132(2), pages 461-489, June.
    18. D'Addio, Anna Cristina & De Greef, Isabelle & Rosholm, Michael, 2002. "Assessing Unemployment Traps in Belgium Using Panel Data Sample Selection Models," IZA Discussion Papers 669, Institute of Labor Economics (IZA).
    19. Chernozhukov, Victor & Fernández-Val, Iván & Hoderlein, Stefan & Holzmann, Hajo & Newey, Whitney, 2015. "Nonparametric identification in panels using quantiles," Journal of Econometrics, Elsevier, vol. 188(2), pages 378-392.
    20. María laura Alzúa & Guillermo Cruces & Carolina Lopez, 2016. "Long-Run Effects Of Youth Training Programs: Experimental Evidence From Argentina," Economic Inquiry, Western Economic Association International, vol. 54(4), pages 1839-1859, October.

    More about this item

    Keywords

    attrition; field experiments; randomized experiments; randomized controlled trials; internal validity; Kolmogorov-Smirnov; Cramer-von-Mises; randomization tests;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ucr:wpaper:201919. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kelvin Mac (email available below). General contact details of provider: https://edirc.repec.org/data/deucrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.