IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp10711.html
   My bibliography  Save this paper

Attrition in Randomized Control Trials: Using Tracking Information to Correct Bias

Author

Listed:
  • Molina Millán, Teresa

    () (Universidade Nova de Lisboa)

  • Macours, Karen

    () (Paris School of Economics)

Abstract

This paper starts from a review of RCT studies in development economics, and documents many studies largely ignore attrition once attrition rates are found balanced between treatment arms. The paper analyzes the implications of attrition for the internal and external validity of the results of a randomized experiment with balanced attrition rates, and proposes a new method to correct for attrition bias. We rely on a 10-years longitudinal data set with a final attrition rate of 10 percent, obtained after intensive tracking of migrants, and document the sensitivity of ITT estimates for schooling gains and labour market outcomes for a social program in Nicaragua. We find that not including those found during the intensive tracking leads to an overestimate of the ITT effects for the target population by more than 35 percent, and that selection into attrition is driven by observable baseline characteristics. We propose to correct for attrition using inverse probability weighting with estimates of weights that exploit the similarities between missing individuals and those found during an intensive tracking phase. We compare these estimates with alternative strategies using regression adjustment, standard weights, bounds or proxy information.

Suggested Citation

  • Molina Millán, Teresa & Macours, Karen, 2017. "Attrition in Randomized Control Trials: Using Tracking Information to Correct Bias," IZA Discussion Papers 10711, Institute for the Study of Labor (IZA).
  • Handle: RePEc:iza:izadps:dp10711
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp10711.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Pedro C. Vicente, 2014. "Is Vote Buying Effective? Evidence from a Field Experiment in West Africa," Economic Journal, Royal Economic Society, vol. 124(574), pages 356-387, February.
    2. Esther Duflo & Michael Kremer & Jonathan Robinson, 2011. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 101(6), pages 2350-2390, October.
    3. Jobiba Chinkhumba & Susan Godlonton & Rebecca Thornton, 2014. "The Demand for Medical Male Circumcision," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 152-177, April.
    4. Jeffrey E. Zabel, 1998. "An Analysis of Attrition in the Panel Study of Income Dynamics and the Survey of Income and Program Participation with an Application to a Model of Labor Market Behavior," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 479-506.
    5. Karen Macours & Norbert Schady & Renos Vakis, 2012. "Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment," American Economic Journal: Applied Economics, American Economic Association, vol. 4(2), pages 247-273, April.
    6. Edmonds, Eric V. & Shrestha, Maheshwor, 2014. "You get what you pay for: Schooling incentives and child labor," Journal of Development Economics, Elsevier, pages 196-211.
    7. Hidrobo, Melissa & Hoddinott, John & Peterman, Amber & Margolies, Amy & Moreira, Vanessa, 2014. "Cash, food, or vouchers? Evidence from a randomized experiment in northern Ecuador," Journal of Development Economics, Elsevier, vol. 107(C), pages 144-156.
    8. Powell-Jackson, Timothy & Hanson, Kara & Whitty, Christopher J.M. & Ansah, Evelyn K., 2014. "Who benefits from free healthcare? Evidence from a randomized experiment in Ghana," Journal of Development Economics, Elsevier, vol. 107(C), pages 305-319.
    9. Valdivia, Martín, 2015. "Business training plus for female entrepreneurship? Short and medium-term experimental evidence from Peru," Journal of Development Economics, Elsevier, pages 33-51.
    10. Kazianga, Harounan & de Walque, Damien & Alderman, Harold, 2014. "School feeding programs, intrahousehold allocation and the nutrition of siblings: Evidence from a randomized trial in rural Burkina Faso," Journal of Development Economics, Elsevier, vol. 106(C), pages 15-34.
    11. Seshan, Ganesh & Yang, Dean, 2014. "Motivating migrants: A field experiment on financial decision-making in transnational households," Journal of Development Economics, Elsevier, vol. 108(C), pages 119-127.
    12. John Fitzgerald & Peter Gottschalk & Robert Moffitt, 1998. "An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 251-299.
    13. Jessica Cohen & Pascaline Dupas & Simone Schaner, 2015. "Price Subsidies, Diagnostic Tests, and Targeting of Malaria Treatment: Evidence from a Randomized Controlled Trial," American Economic Review, American Economic Association, vol. 105(2), pages 609-645, February.
    14. Meredith, Jennifer & Robinson, Jonathan & Walker, Sarah & Wydick, Bruce, 2013. "Keeping the doctor away: Experimental evidence on investment in preventative health products," Journal of Development Economics, Elsevier, pages 196-210.
    15. Alejandro Drexler & Greg Fischer & Antoinette Schoar, 2014. "Keeping It Simple: Financial Literacy and Rules of Thumb," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 1-31, April.
    16. Taryn Dinkelman & Claudia Martínez A., 2014. "Investing in Schooling In Chile: The Role of Information about Financial Aid for Higher Education," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 244-257, May.
    17. Shawn Cole & Daniel Stein & Jeremy Tobacman, 2014. "Dynamics of Demand for Index Insurance: Evidence from a Long-Run Field Experiment," American Economic Review, American Economic Association, vol. 104(5), pages 284-290, May.
    18. Luc Behaghel & Bruno Crépon & Marc Gurgand & Thomas Le Barbanchon, 2015. "Please Call Again: Correcting Nonresponse Bias in Treatment Effect Models," The Review of Economics and Statistics, MIT Press, vol. 97(5), pages 1070-1080, December.
    19. Thomas, Duncan & Witoelar, Firman & Frankenberg, Elizabeth & Sikoki, Bondan & Strauss, John & Sumantri, Cecep & Suriastini, Wayan, 2012. "Cutting the costs of attrition: Results from the Indonesia Family Life Survey," Journal of Development Economics, Elsevier, vol. 98(1), pages 108-123.
    20. Manuela Angelucci & Dean Karlan & Jonathan Zinman, 2015. "Microcredit Impacts: Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 151-182, January.
    21. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    22. Nava Ashraf & Erica Field & Jean Lee, 2014. "Household Bargaining and Excess Fertility: An Experimental Study in Zambia," American Economic Review, American Economic Association, vol. 104(7), pages 2210-2237, July.
    23. repec:oup:qjecon:v:129:y:2014:i:2:p:697-752. is not listed on IDEAS
    24. Deon Filmer & Lant Pritchett, 2001. "Estimating Wealth Effects Without Expenditure Data—Or Tears: An Application To Educational Enrollments In States Of India," Demography, Springer;Population Association of America (PAA), vol. 38(1), pages 115-132, February.
    25. Jesse M. Cunha, 2014. "Testing Paternalism: Cash versus In-Kind Transfers," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 195-230, April.
    26. Bruno Crépon & Florencia Devoto & Esther Duflo & William Parienté, 2015. "Estimating the Impact of Microcredit on Those Who Take It Up: Evidence from a Randomized Experiment in Morocco," American Economic Journal: Applied Economics, American Economic Association, pages 123-150.
    27. Falaris, Evangelos M., 2003. "The effect of survey attrition in longitudinal surveys: evidence from Peru, Cote d'Ivoire and Vietnam," Journal of Development Economics, Elsevier, vol. 70(1), pages 133-157, February.
    28. Xavier Gine & Jessica Goldberg & Dean Yang, 2012. "Credit Market Consequences of Improved Personal Identification: Field Experimental Evidence from Malawi," American Economic Review, American Economic Association, vol. 102(6), pages 2923-2954, October.
    29. Tania Barham & Karen Macours & John A. Maluccio, 2013. "Boys' Cognitive Skill Formation and Physical Growth: Long-Term Experimental Evidence on Critical Ages for Early Childhood Interventions," American Economic Review, American Economic Association, vol. 103(3), pages 467-471, May.
    30. Susan Godlonton & Rebecca L. Thornton, 2013. "Learning from Others' HIV Testing: Updating Beliefs and Responding to Risk," American Economic Review, American Economic Association, vol. 103(3), pages 439-444, May.
    31. Karen Macours & Renos Vakis, 2014. "Changing Households' Investment Behaviour through Social Interactions with Local Leaders: Evidence from a Randomised Transfer Programme," Economic Journal, Royal Economic Society, vol. 0(576), pages 607-633, May.
    32. Jere R. Behrman & Susan W. Parker & Petra E. Todd & Kenneth I. Wolpin, 2015. "Aligning Learning Incentives of Students and Teachers: Results from a Social Experiment in Mexican High Schools," Journal of Political Economy, University of Chicago Press, vol. 123(2), pages 325-364.
    33. Barham, Tania & Macours, Karen & Maluccio, John, 2017. "Are Conditional Cash Transfers Fulfilling Their Promise? Schooling, Learning, and Earnings After 10 Years," CEPR Discussion Papers 11937, C.E.P.R. Discussion Papers.
    34. Angus Deaton & Nancy Cartwright, 2016. "Understanding and Misunderstanding Randomized Controlled Trials," Working Papers august_25.pdf, Princeton University, Woodrow Wilson School of Public and International Affairs, Research Program in Development Studies..
    35. Erica Field & Seema Jayachandran & Rohini Pande, 2010. "Do Traditional Institutions Constrain Female Entrepreneurship? A Field Experiment on Business Training in India," American Economic Review, American Economic Association, vol. 100(2), pages 125-129, May.
    36. Yi, Hongmei & Song, Yingquan & Liu, Chengfang & Huang, Xiaoting & Zhang, Linxiu & Bai, Yunli & Ren, Baoping & Shi, Yaojiang & Loyalka, Prashant & Chu, James & Rozelle, Scott, 2015. "Giving kids a head start: The impact and mechanisms of early commitment of financial aid on poor students in rural China," Journal of Development Economics, Elsevier, vol. 113(C), pages 1-15.
    37. Christopher Blattman & Nathan Fiala & Sebastian Martinez, 2014. "Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda," The Quarterly Journal of Economics, Oxford University Press, vol. 129(2), pages 697-752.
    38. Bruhn, Miriam & Lara Ibarra, Gabriel & McKenzie, David, 2014. "The minimal impact of a large-scale financial education program in Mexico City," Journal of Development Economics, Elsevier, vol. 108(C), pages 184-189.
    39. repec:idb:brikps:publication-detail,7101.html?id=32886 is not listed on IDEAS
    40. Karen Macours & Norbert Schady & Renos Vakis, 2012. "Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment," American Economic Journal: Applied Economics, American Economic Association, vol. 4(2), pages 247-273, April.
    41. Nava Ashraf & James Berry & Jesse M. Shapiro, 2010. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia," American Economic Review, American Economic Association, vol. 100(5), pages 2383-2413, December.
    42. Abhijit Banerjee & Esther Duflo & Richard Hornbeck, 2014. "Bundling Health Insurance and Microfinance in India: There Cannot Be Adverse Selection If There Is No Demand," American Economic Review, American Economic Association, vol. 104(5), pages 291-297, May.
    43. Shane A. Norris & Linda M. Richter & Stella A. Fleetwood, 2007. "Panel studies in developing countries: case analysis of sample attrition over the past 16 years within the birth to twenty cohort in Johannesburg, South Africa," Journal of International Development, John Wiley & Sons, Ltd., vol. 19(8), pages 1143-1150.
    44. Abhijit Banerjee & Esther Duflo & Rachel Glennerster & Cynthia Kinnan, 2015. "The Miracle of Microfinance? Evidence from a Randomized Evaluation," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 22-53, January.
    45. Jessica Cohen & Pascaline Dupas, 2010. "Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 1-45.
    46. Alessandro Tarozzi & Aprajit Mahajan & Brian Blackburn & Dan Kopf & Lakshmi Krishnan & Joanne Yoong, 2014. "Micro-loans, Insecticide-Treated Bednets, and Malaria: Evidence from a Randomized Controlled Trial in Orissa, India," American Economic Review, American Economic Association, vol. 104(7), pages 1909-1941, July.
    47. David Bravo & Jere R. Behrman & Olivia s. Mitchell & Petra E. Todd. & Javiera Vásquez, 2008. "Encuesta de protección social 2006: Presentación general y principales resultados," Working Papers wp273, University of Chile, Department of Economics.
    48. Prina, Silvia, 2015. "Banking the poor via savings accounts: Evidence from a field experiment," Journal of Development Economics, Elsevier, vol. 115(C), pages 16-31.
    49. Benjamin A. Olken & Junko Onishi & Susan Wong, 2014. "Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia," American Economic Journal: Applied Economics, American Economic Association, vol. 6(4), pages 1-34, October.
    50. Tania Barham & Karen Macours & John A. Maluccio, 2013. "Boys' Cognitive Skill Formation and Physical Growth: Long-Term Experimental Evidence on Critical Ages for Early Childhood Interventions," American Economic Review, American Economic Association, vol. 103(3), pages 467-471, May.
    51. Wilson, Nicholas L. & Xiong, Wentao & Mattson, Christine L., 2014. "Is sex like driving? HIV prevention and risk compensation," Journal of Development Economics, Elsevier, vol. 106(C), pages 78-91.
    52. Stein, Daniel, 2014. "Dynamics of demand for rainfall index insurance : evidence from a commercial product in India," Policy Research Working Paper Series 7035, The World Bank.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Barham, Tania & Macours, Karen & Maluccio, John, 2017. "Are Conditional Cash Transfers Fulfilling Their Promise? Schooling, Learning, and Earnings After 10 Years," CEPR Discussion Papers 11937, C.E.P.R. Discussion Papers.

    More about this item

    Keywords

    survey non response; sample selectivity; randomized controlled trial; inverse probability weights;

    JEL classification:

    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp10711. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.