IDEAS home Printed from https://ideas.repec.org/a/oup/wbecrv/v28y2014i3p432-445..html
   My bibliography  Save this article

Evaluation of Development Programs: Randomized Controlled Trials or Regressions?

Author

Listed:
  • Chris Elbers
  • Jan Willem Gunning

Abstract

Can project evaluation methods be used to evaluate programs: complex interventions involving multiple activities? A program evaluation cannot be based simply on separate evaluations of its components if interactions between the activities are important. In this paper a measure is proposed, the total program effect (TPE), which is an extension of the average treatment effect on the treated (ATET). It explicitly takes into account that in the real world (with heterogeneous treatment effects) individual treatment effects and program assignment are often correlated. The TPE can also deal with the common situation in which such a correlation is the result of decisions on (intended) program participation not being taken centrally. In this context RCTs are less suitable even for the simplest interventions. The TPE can be estimated by applying regression techniques to observational data from a representative sample from the targeted population. The approach is illustrated with an evaluation of a health insurance program in Vietnam.

Suggested Citation

  • Chris Elbers & Jan Willem Gunning, 2014. "Evaluation of Development Programs: Randomized Controlled Trials or Regressions?," The World Bank Economic Review, World Bank, vol. 28(3), pages 432-445.
    • Handle: RePEc:oup:wbecrv:v:28:y:2014:i:3:p:432-445.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/wber/lht025
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Das, J. & Dercon, S. & Habyarimana, J. & Krishnan, P., 2004. "‘When Can School Inputs Improve Test Scores?’," Cambridge Working Papers in Economics 0437, Faculty of Economics, University of Cambridge.
    2. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    3. Alain de Janvry & Frederico Finan & Elisabeth Sadoulet, 2012. "Local Electoral Incentives and Decentralized Program Performance," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 672-685, August.
    4. White, Howard, 2006. "Impact evaluation: the experience of the Independent Evaluation Group of the World Bank," MPRA Paper 1111, University Library of Munich, Germany.
    5. Chen, Shaohua & Mu, Ren & Ravallion, Martin, 2009. "Are there lasting impacts of aid to poor areas?," Journal of Public Economics, Elsevier, vol. 93(3-4), pages 512-528, April.
    6. Jishnu Das & Stefan Dercon & James Habyarimana & Pramila Krishnan, 2007. "Teacher Shocks and Student Learning: Evidence from Zambia," Journal of Human Resources, University of Wisconsin Press, vol. 42(4).
    7. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    8. Elbers, Chris & Gunning, Jan Willem & de Hoop, Kobus, 2009. "Assessing Sector-wide Programs with Statistical Impact Evaluation: A Methodological Proposal," World Development, Elsevier, vol. 37(2), pages 513-520, February.
    9. Martin Ravallion, 2012. "Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty : Review Essay," Journal of Economic Literature, American Economic Association, vol. 50(1), pages 103-114, March.
    10. repec:pri:rpdevs:deaton_instruments_randomization_learning_all_04april_2010 is not listed on IDEAS
    11. Shahidur R. Khandker & Zaid Bakht & Gayatri B. Koolwal, 2009. "The Poverty Impact of Rural Roads: Evidence from Bangladesh," Economic Development and Cultural Change, University of Chicago Press, vol. 57(4), pages 685-722, July.
    12. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    13. Martin Ravallion, 2009. "Evaluation in the Practice of Development," The World Bank Research Observer, World Bank, vol. 24(1), pages 29-53, March.
    14. James Heckman, 1997. "Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations," Journal of Human Resources, University of Wisconsin Press, vol. 32(3), pages 441-462.
    15. Almeida, Rita K. & Galasso, Emanuela, 2010. "Jump-starting Self-employment? Evidence for Welfare Participants in Argentina," World Development, Elsevier, vol. 38(5), pages 742-755, May.
    16. Rodrik, Dani, 2008. "The New Development Economics: We Shall Experiment, but How Shall We Learn?," Working Paper Series rwp08-055, Harvard University, John F. Kennedy School of Government.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Headey, Derek D. & Hoddinott, John, 2016. "Agriculture, nutrition and the green revolution in Bangladesh," Agricultural Systems, Elsevier, vol. 149(C), pages 122-131.
    2. Yoshino, Naoyuki & Abidhadjaev, Umid, 2017. "An impact evaluation of investment in infrastructure: The case of a railway connection in Uzbekistan," Journal of Asian Economics, Elsevier, vol. 49(C), pages 1-11.
    3. Tsiboe, Francis & Nalley, Lawton Lanier & Dixon, Bruce L. & Popp, Jennie S. & Luckstead, Jeff, 2014. "Cost-Benefit Analysis of the Cocoa Livelihoods Program in Sub-Saharan Africa," 2015 Annual Meeting, January 31-February 3, 2015, Atlanta, Georgia 195775, Southern Agricultural Economics Association.
    4. Headey, Derek D. & Hoddinott, John F., 2014. "Understanding the rapid reduction of undernutrition in Nepal, 2001-2011:," IFPRI discussion papers 1384, International Food Policy Research Institute (IFPRI).
    5. Melinda Vigh & Chris Elbers, 2017. "Picking Winners: Measuring the Effectiveness of Selectively Placed Policy Interventions," Tinbergen Institute Discussion Papers 17-110/V, Tinbergen Institute.
    6. repec:fpr:2020cp:1(1 is not listed on IDEAS
    7. Bigsten, Arne, 2016. "The Development of Development Economics," Working Papers in Economics 653, University of Gothenburg, Department of Economics, revised Mar 2017.
    8. Fan, Shenggen & Brzeska, Joanna, 2011. "The nexus between agriculture and nutrition: Do growth patterns and conditional factors matter?," 2020 conference briefs 1, International Food Policy Research Institute (IFPRI).
    9. Basu, Kaushik, 2013. "The method of randomization and the role of reasoned intuition," Policy Research Working Paper Series 6722, The World Bank.
    10. Boulier, Bryan & Emran, M. Shahe & Hoque, Nazmul, 2021. "Access to Credit, Education, and Women’s Say in the Household: Evidence from Bangladesh," MPRA Paper 109009, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chris Elbers & Jan Willem Gunning, 2012. "Evaluation of Development Programs: Using Regressions to assess the Impact of Complex Interventions," Tinbergen Institute Discussion Papers 12-081/2, Tinbergen Institute.
    2. Chris Elbers & Jan Willem Gunning, 2009. "Evaluation of Development Policy: Treatment versus Program Effects," Tinbergen Institute Discussion Papers 09-073/2, Tinbergen Institute.
    3. Elbers, Chris & Gunning, Jan Willem, 2014. "Evaluation of non-governmental development organizations," WIDER Working Paper Series 026, World Institute for Development Economic Research (UNU-WIDER).
    4. Chris Elbers & Jan Willem Gunning, 2014. "Evaluation of Non-Governmental Development Organizations," WIDER Working Paper Series wp-2014-026, World Institute for Development Economic Research (UNU-WIDER).
    5. Donovan, Kevin P., 2018. "The rise of the randomistas: on the experimental turn in international aid," SocArXiv xygzb, Center for Open Science.
    6. Onur Altindag & Theodore J. Joyce & Julie A. Reeder, 2015. "Effects of Peer Counseling to Support Breastfeeding: Assessing the External Validity of a Randomized Field Experiment," NBER Working Papers 21013, National Bureau of Economic Research, Inc.
    7. David K. Evans & Anna Popova, 2016. "What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews," The World Bank Research Observer, World Bank, vol. 31(2), pages 242-270.
    8. Breen, Richard & Ermisch, John, 2021. "Instrumental Variable Estimation in Demographic Studies: The LATE interpretation of the IV estimator with heterogenous effects," SocArXiv vx9m7, Center for Open Science.
    9. Temple, Jonathan R.W., 2010. "Aid and Conditionality," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4415-4523, Elsevier.
    10. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    11. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    12. Ayako Wakano & Hiroyuki Yamada & Daichi Shimamoto, 2017. "Does the Heterogeneity of Project Implementers Affect the Programme Participation of Beneficiaries?: Evidence from Rural Cambodia," Journal of Development Studies, Taylor & Francis Journals, vol. 53(1), pages 49-67, January.
    13. Glewwe, Paul & Kremer, Michael, 2006. "Schools, Teachers, and Education Outcomes in Developing Countries," Handbook of the Economics of Education, in: Erik Hanushek & F. Welch (ed.), Handbook of the Economics of Education, edition 1, volume 2, chapter 16, pages 945-1017, Elsevier.
    14. Martin Ravallion, 2012. "Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty : Review Essay," Journal of Economic Literature, American Economic Association, vol. 50(1), pages 103-114, March.
    15. Masino, Serena & Niño-Zarazúa, Miguel, 2016. "What works to improve the quality of student learning in developing countries?," International Journal of Educational Development, Elsevier, vol. 48(C), pages 53-65.
    16. Hammer, Jeffrey & Spears, Dean, 2016. "Village sanitation and child health: Effects and external validity in a randomized field experiment in rural India," Journal of Health Economics, Elsevier, vol. 48(C), pages 135-148.
    17. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2019. "All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development," Development and Change, International Institute of Social Studies, vol. 50(3), pages 735-762, May.
    18. Ralitza Dimova, 2019. "A Debate that Fatigues…: To Randomise or Not to Randomise; What’s the Real Question?," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 31(2), pages 163-168, April.
    19. Arthur Lewbel, 2019. "The Identification Zoo: Meanings of Identification in Econometrics," Journal of Economic Literature, American Economic Association, vol. 57(4), pages 835-903, December.
    20. Duo Qin & Yanqun Zhang, 2013. "A History of Polyvalent Structural Parameters: the Case of Instrument Variable Estimators," Working Papers 183, Department of Economics, SOAS University of London, UK.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:wbecrv:v:28:y:2014:i:3:p:432-445.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://edirc.repec.org/data/wrldbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.