IDEAS home Printed from https://ideas.repec.org/p/idb/spdwps/1003.html
   My bibliography  Save this paper

Program Evaluation and Spillover Effects

Author

Listed:
  • Manuela Angelucci
  • Vincenzo Di Maro

Abstract

This note defines what spillover effects are, why it is important to measure them, and how to design a field experiment that will enable researchers to measure the average effects of the treatment in the presence of spillover effects on subjects both eligible and ineligible for the program. In addition, it discusses how to use nonexperimental methods for estimating spillover effects when the experimental design is not a viable option. Several practical examples are provided to show how spillover effects can be estimated. Evaluations that account for spillover effects should be designed in such a way that they explain both the cause of these effects and who is affected by them. Failure to have such an evaluation design can result in wrong policy recommendations and in the neglect of important mechanisms through which the program operates. To estimate the direct and indirect effect of a program, one has to use control groups that are not affected by the program either directly or indirectly. This often means selecting the control groups from different geographic units (e.g. the village or school). In order to understand the mechanisms that cause spillover effects one has to think about competing explanations and collect data on relevant outcomes. In many cases, unveiling the mechanisms behind the spillover effects results in a better understanding of how the program works in general.

Suggested Citation

  • Manuela Angelucci & Vincenzo Di Maro, 2010. "Program Evaluation and Spillover Effects," SPD Working Papers 1003, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
  • Handle: RePEc:idb:spdwps:1003
    as

    Download full text from publisher

    File URL: http://www.iadb.org/document.cfm?id=35173297
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 128(2), pages 531-580.
    2. Pierre Dubois & Bruno Jullien & Thierry Magnac, 2008. "Formal and Informal Risk Sharing in LDCs: Theory and Empirical Evidence," Econometrica, Econometric Society, vol. 76(4), pages 679-725, July.
    3. Hirano, Keisuke & Hahn, Jinyong, 2010. "Design of randomized experiments to measure social interaction effects," Economics Letters, Elsevier, vol. 106(1), pages 51-53, January.
    4. Townsend, Robert M, 1994. "Risk and Insurance in Village India," Econometrica, Econometric Society, vol. 62(3), pages 539-591, May.
    5. David S. Lee & Thomas Lemieux, 2009. "Regression Discontinuity Designs In Economics," Working Papers 1118, Princeton University, Department of Economics, Industrial Relations Section..
    6. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    7. Ciro Avitabile, 2011. "Spillover Effects in Healthcare Programs: Evidence on Social Norms and Information Sharing," CSEF Working Papers 271, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy, revised 14 Mar 2011.
    8. Dean Yang, 2008. "Can Enforcement Backfire? Crime Displacement in the Context of Customs Reform in the Philippines," The Review of Economics and Statistics, MIT Press, vol. 90(1), pages 1-14, February.
    9. Sarah Baird & Aislinn Bohren & Berk Ozler & Craig McIntosh, 2014. "Designing Experiments to Measure Spillover Effects," Working Papers 2014-11, The George Washington University, Institute for International Economic Policy.
    10. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    11. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    12. Gustavo J. Bobonis & Frederico Finan, 2009. "Neighborhood Peer Effects in Secondary School Enrollment Decisions," The Review of Economics and Statistics, MIT Press, vol. 91(4), pages 695-716, November.
    13. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    14. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    15. Ciro Avitabile, 2011. "Spillover Effects in Healthcare Programs: Evidence on Social Norms and Information Sharing," CSEF Working Papers 271, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy, revised 14 Mar 2011.
    16. Jeffrey I. Steinfeld, 1999. "Book," Journal of Industrial Ecology, Yale University, vol. 3(4), pages 145-147, October.
    17. Rafael Di Tella & Ernesto Schargrodsky, 2004. "Do Police Reduce Crime? Estimates Using the Allocation of Police Forces After a Terrorist Attack," American Economic Review, American Economic Association, vol. 94(1), pages 115-133, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Huber, Martin & Steinmayr, Andreas, 2017. "A Framework for Separating Individual Treatment Effects From Spillover, Interaction, and General Equilibrium Effects," Rationality and Competition Discussion Paper Series 21, CRC TRR 190 Rationality and Competition.
    2. Angelucci, Manuela & Prina, Silvia & Royer, Heather & Samek, Anya, 2015. "When Incentives Backfire: Spillover Effects in Food Choice," IZA Discussion Papers 9288, Institute of Labor Economics (IZA).
    3. Beliyou Haile & Carlo Azzarri & Cleo Roberts & David J. Spielman, 2017. "Targeting, bias, and expected impact of complex innovations on developing-country agriculture: evidence from Malawi," Agricultural Economics, International Association of Agricultural Economists, vol. 48(3), pages 317-326, May.
    4. Manan Roy & Daniel Millimet & Rusty Tchernis, 2012. "Federal nutrition programs and childhood obesity: inside the black box," Review of Economics of the Household, Springer, vol. 10(1), pages 1-38, March.
    5. Giuliani, Elisa & Maffioli, Alessandro & Pacheco, Manuel & Pietrobelli, Carlo & Stucchi, Rodolfo, 2014. "Evaluating the Impact of Cluster Development Programs," Papers in Innovation Studies 2014/10, Lund University, CIRCLE - Centre for Innovation Research.
    6. Clarke, Damian, 2017. "Estimating Difference-in-Differences in the Presence of Spillovers," MPRA Paper 81604, University Library of Munich, Germany.
    7. Castro, Marcelo Araújo & Mattos, Enlinson & Regatieri, Rebeca, 2015. "Fiscal interactions and spillover effects of a federal grant to Brazilian municipalities," Textos para discussão 388, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    8. Li, Chunying & Zhang, Jinning & Lyu, Yanwei, 2022. "Does the opening of China railway express promote urban total factor productivity? New evidence based on SDID and SDDD model," Socio-Economic Planning Sciences, Elsevier, vol. 80(C).
    9. Viviana Celli, 2022. "Causal mediation analysis in economics: Objectives, assumptions, models," Journal of Economic Surveys, Wiley Blackwell, vol. 36(1), pages 214-234, February.
    10. Davidson, Angus Alexander & Young, Michael Denis & Leake, John Espie & O’Connor, Patrick, 2022. "Aid and forgetting the enemy: A systematic review of the unintended consequences of international development in fragile and conflict-affected situations," Evaluation and Program Planning, Elsevier, vol. 92(C).
    11. Schmidt, Emily & Tadesse, Fanaye, 2017. "The sustainable land management program in the Ethiopian highlands: An evaluation of its impact on crop production:," ESSP working papers 103, International Food Policy Research Institute (IFPRI).
    12. Henrik Jungaberle & Ede Nagy, 2015. "Pilot Evaluation Study of the Life Skills Program REBOUND," SAGE Open, , vol. 5(4), pages 21582440156, November.
    13. Victoria Castillo & Lucas Figal Garone & Alessandro Maffioli & Lina Salazar, 2015. "Tourism Policy, a Big Push to Employment: Evidence from a Multiple Synthetic Control Approach," IDB Publications (Working Papers) 87973, Inter-American Development Bank.
    14. Ruben Fotso, 2020. "Evaluation of indirect effects of place-based science-industry transfer policies: Case of French Technological Research Institutes," Working Papers 2031, Groupe d'Analyse et de Théorie Economique Lyon St-Étienne (GATE Lyon St-Étienne), Université de Lyon.
    15. George Mgendi & Shiping Mao & Fangbin Qiao, 2021. "Is a Training Program Sufficient to Improve the Smallholder Farmers’ Productivity in Africa? Empirical Evidence from a Chinese Agricultural Technology Demonstration Center in Tanzania," Sustainability, MDPI, vol. 13(3), pages 1-23, February.
    16. Ibanez, Marcela & Blackman, Allen, 2016. "Is Eco-Certification a Win–Win for Developing Country Agriculture? Organic Coffee Certification in Colombia," World Development, Elsevier, vol. 82(C), pages 14-27.
    17. Gustavo A. Crespi & Alessandro Maffioli & Pierre Mohnen & Gonzalo Vázquez, 2011. "Evaluating the Impact of Science, Technology and Innovation Programs: a Methodological Toolkit," SPD Working Papers 1104, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
    18. Ruben Fotso, 2020. "Evaluation of indirect effects of place-based science-industry transfer policies: Case of French Technological Research Institutes," Working Papers halshs-02998262, HAL.
    19. Githiomi, Caroline & Muriithi, Beatrice & Irungu, Patrick & Mwungu, Chris M. & Diiro, Gracious & Affognon, Hippolyte & Mburu, John & Ekesi, Sunday, 2019. "Economic analysis of spillover effects of an integrated pest management (IPM) strategy for suppression of mango fruit fly in Kenya," Food Policy, Elsevier, vol. 84(C), pages 121-132.
    20. Mohamed Buheji, 2020. "Socio-Economic Projects Spillovers and Their Influence on Communities Development," Review of European Studies, Canadian Center of Science and Education, vol. 12(1), pages 1-47, March.
    21. Martin Persson, U. & Alpízar, Francisco, 2013. "Conditional Cash Transfers and Payments for Environmental Services—A Conceptual Framework for Explaining and Judging Differences in Outcomes," World Development, Elsevier, vol. 43(C), pages 124-137.
    22. O'Shaughnessy, Eric, 2022. "Rooftop solar incentives remain effective for low- and moderate-income adoption," Energy Policy, Elsevier, vol. 163(C).
    23. Peter Griffiths, 2015. "Marketing by Controlling Social Discourse: The Fairtrade Case," Economic Affairs, Wiley Blackwell, vol. 35(2), pages 256-271, June.
    24. Arthur Alik-Lagrange & Martin Ravallion, 2016. "Social Frictions to Knowledge Diffusion: Evidence from an Information Intervention," NBER Working Papers 21877, National Bureau of Economic Research, Inc.
    25. Pascaline Dupas & Edward Miguel, 2016. "Impacts and Determinants of Health Levels in Low-Income Countries," NBER Working Papers 22235, National Bureau of Economic Research, Inc.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Manuela Angelucci & Vincenzo Di Maro, 2010. "Program Evaluation and Spillover Effects," SPD Working Papers 1003, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
    2. repec:idb:brikps:259 is not listed on IDEAS
    3. Arthur Alik-Lagrange & Martin Ravallion, 2016. "Social Frictions to Knowledge Diffusion: Evidence from an Information Intervention," NBER Working Papers 21877, National Bureau of Economic Research, Inc.
    4. Rafael Perez Ribas & Fabio Veras Soares & Clarissa Gondim Teixeira & Elydia Silva & Guilherme Issamu Hirata, 2010. "Beyond Cash: Assessing Externality and Behaviour Effects of Non-Experimental Cash Transfers," Working Papers 65, International Policy Centre for Inclusive Growth.
    5. List, John & Momeni, Fatemeh & Zenou, Yves, 2019. "Are Estimates of Early Education Programs Too Pessimistic? Evidence from a Large-Scale Field Experiment that Causally Measures Neighbor Effects," CEPR Discussion Papers 13725, C.E.P.R. Discussion Papers.
    6. Huber, Martin & Steinmayr, Andreas, 2017. "A Framework for Separating Individual Treatment Effects from Spillover, Interaction, and General Equilibrium Effects," IZA Discussion Papers 10648, Institute of Labor Economics (IZA).
    7. Christian Volpe Martincus, 2010. "Odyssey in International Markets: An Assessment of the Effectiveness of Export Promotion in Latin America and the Caribbean," IDB Publications (Books), Inter-American Development Bank, number 16458, February.
    8. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    9. Olivia D'Aoust & Olivier Sterck & Philip Verwimp, 2013. "Buying Peace: The Mirage of Demobilizing Rebels," Working Papers ECARES ECARES 2013-22, ULB -- Universite Libre de Bruxelles.
    10. Bobonis, Gustavo J. & Castro, Roberto & Morales, Juan S., 2018. "Conditional Cash Transfers for Women and Spousal Violence: Evidence of the Long-Term Relationship from the Oportunidades Program in Rural Mexico," IDB Publications (Working Papers) 7267, Inter-American Development Bank.
    11. Matteo Bobba & Jérémie Gignoux, 2019. "Neighborhood Effects in Integrated Social Policies," The World Bank Economic Review, World Bank Group, vol. 33(1), pages 116-139.
    12. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    13. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    14. Daniel Mejía & Pascual Restrepo & Sandra V. Rozo, 2017. "On the Effects of Enforcement on Illegal Markets: Evidence from a Quasi-Experiment in Colombia," The World Bank Economic Review, World Bank Group, vol. 31(2), pages 570-594.
    15. Gonzalo Vazquez-Bare, 2017. "Identification and Estimation of Spillover Effects in Randomized Experiments," Papers 1711.02745, arXiv.org, revised Jan 2022.
    16. Sarah Baird & Aislinn Bohren & Craig McIntosh & Berk Ozler, 2017. "Optimal Design of Experiments in the Presence of Interference*, Second Version," PIER Working Paper Archive 16-025, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania, revised 30 Nov 2017.
    17. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    18. Christian Volpe Martincus & Jerónimo Carballo, 2010. "Is Export Promotion Effective in Developing Countries? Firm-Level Evidence on the Intensive and Extensive Margins of Exports," IDB Publications (Working Papers) 36763, Inter-American Development Bank.
    19. Julie Le Gallo & Yannick L'Horty & Pascale Petit, 2014. "Does subsidising young people to learn to drive promote social inclusion? Evidence from a large controlled experiment in France," TEPP Working Paper 2014-15, TEPP.
    20. Bandiera, Oriana & Gulesci, Selim & Rasul, Imran & Burgess, Robin, 2009. "Community networks and poverty reduction programmes: evidence from Bangladesh," LSE Research Online Documents on Economics 58054, London School of Economics and Political Science, LSE Library.
    21. Bonan, Jacopo & Battiston, Pietro & Bleck, Jaimie & LeMay-Boucher, Philippe & Pareglio, Stefano & Sarr, Bassirou & Tavoni, Massimo, 2021. "Social interaction and technology adoption: Experimental evidence from improved cookstoves in Mali," World Development, Elsevier, vol. 144(C).

    More about this item

    Keywords

    Impact Evaluation; Spillover Effects; Field Experiments; Data Collection; Indirect Treatment Effect; Program Mechanisms;
    All these keywords.

    JEL classification:

    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • D62 - Microeconomics - - Welfare Economics - - - Externalities

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:idb:spdwps:1003. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/iadbbus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Felipe Herrera Library (email available below). General contact details of provider: https://edirc.repec.org/data/iadbbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.