IDEAS home Printed from https://ideas.repec.org/a/sae/envira/v53y2021i5p992-1011.html
   My bibliography  Save this article

A rusting gold standard: Failures in an Indonesian RCT, and the implications for poverty reduction

Author

Listed:
  • Vikram Tyagi
  • Sophie Webber

Abstract

Randomised Control Trials (RCT) are both widely used in development, reaching hundreds of millions through RCT informed policy, and highly regarded, receiving a Nobel Prize in Economics. Proponents, largely academic economists, position RCTs as a scientific and ideologically neutral way to get to the heart of ‘what works’ in development. However, this new and radical micro-experimental approach to poverty reduction has sat uneasily within the broader development sphere, including critical geographers. We contribute to this debate by empirically examining the practical manifestation of the methodology within ‘PNPM Generasi’ – an innovative Indonesian cash transfer program evaluated through the largest RCT to date. Drawing primarily on field interviews, our examination finds three departures from the gold standard proclamations of the RCT methodology in practice. First, we find that randomising aid delivery breached ethical guidelines and compromised the effectiveness of the program being studied, largely as a result of RCTs being run in secret from local communities and politicians. Second, we find applying the RCT methodology on the ground was technically complex and financially costly, requiring not only an excessive use of scarce program funds, but also undermining the validity of the evaluation results. Finally, we find RCT results were not widely used in practice. We demonstrate these findings are not restricted to our specific case, but rather, reflect systemic deficiencies with RCT evaluations. Our results suggest that RCTs may have value if applied judiciously, and as part of multi-pronged approaches, but we caution against their growing monopoly influence on poverty reduction.

Suggested Citation

  • Vikram Tyagi & Sophie Webber, 2021. "A rusting gold standard: Failures in an Indonesian RCT, and the implications for poverty reduction," Environment and Planning A, , vol. 53(5), pages 992-1011, August.
  • Handle: RePEc:sae:envira:v:53:y:2021:i:5:p:992-1011
    DOI: 10.1177/0308518X20971638
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0308518X20971638
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0308518X20971638?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Adrien Bouguen & Yue Huang & Michael Kremer & Edward Miguel, 2018. "Using RCTs to Estimate Long-Run Impacts in Development Economics," NBER Working Papers 25356, National Bureau of Economic Research, Inc.
    2. Rudolf, K. & Romero, M. & Wollni, M., 2018. "Planting Trees in Oil Palm Plantations: Results from a Randomized Controlled Trial," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 277366, International Association of Agricultural Economists.
    3. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2019. "All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development," Development and Change, International Institute of Social Studies, vol. 50(3), pages 735-762, May.
    4. Naila Kabeer, 2019. "Randomized Control Trials and Qualitative Evaluations of a Multifaceted Programme for Women in Extreme Poverty: Empirical Findings and Methodological Reflections," Journal of Human Development and Capabilities, Taylor & Francis Journals, vol. 20(2), pages 197-217, April.
    5. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    6. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    7. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    8. Christopher B. Barrett & Michael R. Carter, 2010. "The Power and Pitfalls of Experiments in Development Economics: Some Non-random Reflections," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 32(4), pages 515-548.
    9. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.
    10. Stevano, Sara, 2020. "Small development questions are important, but they require big answers," World Development, Elsevier, vol. 127(C).
    11. World Bank, 2011. "Indonesia's PNPM Generasi Program : Final Impact Evaluation Report," World Bank Publications - Reports 21595, The World Bank Group.
    12. Martin Ravallion, 2018. "Should the Randomistas (Continue to) Rule?," Working Papers 492, Center for Global Development, revised 17 Jan 2019.
    13. Donovan, Kevin P., 2018. "The rise of the randomistas: on the experimental turn in international aid," SocArXiv xygzb, Center for Open Science.
    14. Kendra Strauss, 2008. "Re-engaging with rationality in economic geography: behavioural approaches and the importance of context in decision-making," Journal of Economic Geography, Oxford University Press, vol. 8(2), pages 137-156, March.
    15. Sophie Webber, 2015. "Randomising Development: Geography, Economics and the Search for Scientific Rigour," Tijdschrift voor Economische en Sociale Geografie, Royal Dutch Geographical Society KNAG, vol. 106(1), pages 36-52, February.
    16. Sophie Webber & Carolyn Prouse, 2018. "The New Gold Standard: The Rise of Randomized Control Trials and Experimental Development," Economic Geography, Taylor & Francis Journals, vol. 94(2), pages 166-187, March.
    17. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    18. Christian Berndt & Manuel Wirth, 2019. "Struggling for the Moral Market: Economic Knowledge, Diverse Markets, and Market Borders," Economic Geography, Taylor & Francis Journals, vol. 95(3), pages 288-309, May.
    19. Benjamin A. Olken & Junko Onishi & Susan Wong, 2014. "Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia," American Economic Journal: Applied Economics, American Economic Association, vol. 6(4), pages 1-34, October.
    20. Hoffmann, Nimi, 2020. "Involuntary experiments in former colonies: The case for a moratorium," World Development, Elsevier, vol. 127(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wintrup, James, 2022. "Promising careers? A critical analysis of a randomised control trial in community health worker recruitment in Zambia," Social Science & Medicine, Elsevier, vol. 299(C).
    2. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank Group, vol. 33(1), pages 34-64.
    3. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2019. "All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development," Development and Change, International Institute of Social Studies, vol. 50(3), pages 735-762, May.
    4. Basu, Kaushik, 2013. "The method of randomization and the role of reasoned intuition," Policy Research Working Paper Series 6722, The World Bank.
    5. Baldwin Kate & Bhavnani Rikhil R., 2015. "Ancillary Studies of Experiments: Opportunities and Challenges," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 113-146, June.
    6. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    7. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    8. Hanushek, Eric A., 2021. "Addressing cross-national generalizability in educational impact evaluation," International Journal of Educational Development, Elsevier, vol. 80(C).
    9. Kate Baldwin & Rikhil R. Bhavnani, 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series wp-2013-024, World Institute for Development Economic Research (UNU-WIDER).
    10. George W. Norton, 2020. "Lessons from a Career in Agricultural Development and Research Evaluation," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 42(2), pages 151-167, June.
    11. Yonatan Eyal, 2020. "Self-Assessment Variables as a Source of Information in the Evaluation of Intervention Programs: A Theoretical and Methodological Framework," SAGE Open, , vol. 10(1), pages 21582440198, January.
    12. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    13. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    14. Vu, Linh Hoang & Tran, Tuyen Quang, 2021. "Sibship composition, birth order and education: Evidence from Vietnam," International Journal of Educational Development, Elsevier, vol. 85(C).
    15. Kabeer, Naila, 2020. "Women's Empowerment and Economic Development: A Feminist Critique of Storytelling Practices in "Randomista" Economics: a feminist critique of storytelling practices in “randomista” economics," LSE Research Online Documents on Economics 104600, London School of Economics and Political Science, LSE Library.
    16. Kabeer, Naila, 2020. "‘Misbehaving’ RCTs: The confounding problem of human agency," World Development, Elsevier, vol. 127(C).
    17. Ravallion, Martin, 2020. "Highly prized experiments," World Development, Elsevier, vol. 127(C).
    18. Baldwin, Kate & Bhavnani, Rikhil R., 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series 024, World Institute for Development Economic Research (UNU-WIDER).
    19. Bravo-Ureta, Boris E. & Higgins, Daniel & Arslan, Aslihan, 2020. "Irrigation infrastructure and farm productivity in the Philippines: A stochastic Meta-Frontier analysis," World Development, Elsevier, vol. 135(C).
    20. Kabeer, Naila, 2020. "Women’s empowerment and economic development: a feminist critique of story telling practices in ‘Randomista' economics," LSE Research Online Documents on Economics 103880, London School of Economics and Political Science, LSE Library.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:envira:v:53:y:2021:i:5:p:992-1011. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.