IDEAS home Printed from https://ideas.repec.org/p/wbk/wbrwps/6296.html
   My bibliography  Save this paper

Learning from the experiments that never happened : lessons from trying to conduct randomized evaluations of matching grant programs in Africa

Author

Listed:
  • Campos, Francisco
  • Coville, Aidan
  • Fernandes, Ana M.
  • Goldstein, Markus
  • McKenzie, David

Abstract

Matching grants are one of the most common policy instruments used by developing country governments to try to foster technological upgrading, innovation, exports, use of business development services and other activities leading to firm growth. However, since they involve subsidizing firms, the risk is that they could crowd out private investment, subsidizing activities that firms were planning to undertake anyway, or lead to pure private gains, rather than generating the public gains that justify government intervention. As a result, rigorous evaluation of the effects of such programs is important. The authors attempted to implement randomized experiments to evaluate the impact of seven matching grant programs offered in six African countries, but in each case were unable to complete an experimental evaluation. One critique of randomized experiments is publication bias, whereby only those experiments with"interesting"results get published. The hope is to mitigate this bias by learning from the experiments that never happened. This paper describes the three main proximate reasons for lack of implementation: continued project delays, politicians not willing to allow random assignment, and low program take-up; and then delves into the underlying causes of these occurring. Political economy, overly stringent eligibility criteria that do not take account of where value-added may be highest, a lack of attention to detail in"last mile"issues, incentives facing project implementation staff, and the way impact evaluations are funded, and all help explain the failure of randomization. Lessons are drawn from these experiences for both the implementation and the possible evaluation of future projects.

Suggested Citation

  • Campos, Francisco & Coville, Aidan & Fernandes, Ana M. & Goldstein, Markus & McKenzie, David, 2012. "Learning from the experiments that never happened : lessons from trying to conduct randomized evaluations of matching grant programs in Africa," Policy Research Working Paper Series 6296, The World Bank.
  • Handle: RePEc:wbk:wbrwps:6296
    as

    Download full text from publisher

    File URL: http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2012/12/21/000158349_20121221103809/Rendered/PDF/wps6296.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. David McKenzie, 2010. "Impact Assessments in Finance and Private Sector Development: What Have We Learned and What Should We Learn?," The World Bank Research Observer, World Bank Group, vol. 25(2), pages 209-233, August.
    2. Dean Karlan, Ryan Knight, and Christopher Udry, 2012. "Hoping to Win, Expected to Lose: Theory and Lessons on Microenterprise Development," Working Papers 312, Center for Global Development.
    3. Miriam Bruhn & Dean Karlan & Antoinette Schoar, 2018. "The Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico," Journal of Political Economy, University of Chicago Press, vol. 126(2), pages 635-687.
    4. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    5. David McKenzie & Christopher Woodruff, 2014. "What Are We Learning from Business Training and Entrepreneurship Evaluations around the Developing World?," The World Bank Research Observer, World Bank Group, vol. 29(1), pages 48-82.
    6. Nicholas Bloom & Benn Eifert & Aprajit Mahajan & David McKenzie & John Roberts, 2013. "Does Management Matter? Evidence from India," The Quarterly Journal of Economics, Oxford University Press, vol. 128(1), pages 1-51.
    7. Gladys Lopez-Acevedo & Hong W. Tan, 2011. "Impact Evaluation of Small and Medium Enterprise Programs in Latin America and the Caribbean," World Bank Publications - Books, The World Bank Group, number 2298, December.
    8. Ole Dahl Rasmussen & Nikolaj Malchow-Møller & Thomas Barnebeck Andersen, 2011. "Walking the talk: the need for a trial registry for development interventions," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 3(4), pages 502-519, December.
    9. Robert E. Hall & Susan E. Woodward, 2010. "The Burden of the Nondiversifiable Risk of Entrepreneurship," American Economic Review, American Economic Association, vol. 100(3), pages 1163-1194, June.
    10. David McKenzie, 2011. "How Can We Learn Whether Firm Policies Are Working in Africa? Challenges (and Solutions?) For Experiments and Structural Models -super-†," Journal of African Economies, Centre for the Study of African Economies (CSAE), vol. 20(4), pages 600-625, August.
    11. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    12. Ravallion Martin, 2009. "Should the Randomistas Rule?," The Economists' Voice, De Gruyter, vol. 6(2), pages 1-5, February.
    13. Phillips,David A., 2001. "Implementing the market approach to enterprise support - an evaluation of ten matching grant schemes," Policy Research Working Paper Series 2589, The World Bank.
    14. McKenzie, David, 2011. "How can we learn whether firm policies are working in africa ? challenges (and solutions?) for experiments and structural models," Policy Research Working Paper Series 5632, The World Bank.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. 4 visiones sobre retos en evaluación
      by Guest blogger in Hacia el desarrollo efectivo on 2013-05-14 21:36:05
    2. Learning from the experiments that didn’t happen: Part I
      by David McKenzie in Development Impact on 2013-01-07 18:43:21
    3. Learning from the experiments that didn't happen: Part II
      by Markus Goldstein in Development Impact on 2013-01-09 21:09:03
    4. Leaning against publication bias: about the experiments that do not work out
      by Economic Logician in Economic Logic on 2013-01-29 21:44:00
    5. Development that Works: 4 views on evaluation challenges
      by Guest blogger in Eval Central on 2013-05-14 21:39:55

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Grover,Arti Goswami & Imbruno,Michele, 2020. "Using Experimental Evidence to Inform Firm Support Programs in Developing Countries," Policy Research Working Paper Series 9461, The World Bank.
    2. Nathan Fiala & Cormac Mangan, 2013. "Improving Development Effectiveness through R&D: Dynamic Learning and Evaluation," Discussion Papers of DIW Berlin 1325, DIW Berlin, German Institute for Economic Research.
    3. Túlio A. Cravo & Caio Piza, 2019. "The impact of business-support services on firm performance: a meta-analysis," Small Business Economics, Springer, vol. 53(3), pages 753-770, October.
    4. Tulio Cravo & Caio Piza, 2016. "The Impact of Business Support Services for Small and Medium Enterprises on Firm Performance in Low -and Middle- Income Countries: A Meta-Analysis," IDB Publications (Working Papers) 94938, Inter-American Development Bank.
    5. Grimm, Michael & Soubeiga, Sidiki & Weber, Michael, 2021. "Short-Term Impacts of Targeted Cash Grants and Business Development Services: Experimental Evidence from Entrepreneurs in Burkina Faso," IZA Discussion Papers 14892, Institute of Labor Economics (IZA).
    6. David McKenzie & Nabila Assaf & Ana Paula Cusolito, 2017. "The additionality impact of a matching grant programme for small firms: experimental evidence from Yemen," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 9(1), pages 1-14, January.
    7. Hossain, Marup & Mabiso, Athur & Garbero, Alessandra, 2022. "Matching grants and economic activities among horticultural entrepreneurs: Long-term evidence from Rwanda," World Development, Elsevier, vol. 150(C).
    8. David McKenzie & Nabila Assaf & Ana Paula Cusolito, 2016. "The Additionality Impact of a Matching Grant Program for Small Firms," World Bank Publications - Reports 23755, The World Bank Group.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David McKenzie & Christopher Woodruff, 2014. "What Are We Learning from Business Training and Entrepreneurship Evaluations around the Developing World?," The World Bank Research Observer, World Bank Group, vol. 29(1), pages 48-82.
    2. Axel Demenet, 2016. "Does Managerial Capital also Matter Among Micro and Small Firms in Developing Countries?," Working Papers DT/2016/12, DIAL (Développement, Institutions et Mondialisation).
    3. Grimm, Michael & Paffhausen, Anna Luisa, 2015. "Do interventions targeted at micro-entrepreneurs and small and medium-sized firms create jobs? A systematic review of the evidence for low and middle income countries," Labour Economics, Elsevier, vol. 32(C), pages 67-85.
    4. Gabriela Calderon & Jesse M. Cunha & Giacomo De Giorgi, 2020. "Business Literacy and Development: Evidence from a Randomized Controlled Trial in Rural Mexico," Economic Development and Cultural Change, University of Chicago Press, vol. 68(2), pages 507-540.
    5. Marshall Burke & Lauren Falcao Bergquist & Edward Miguel, 2019. "Sell Low and Buy High: Arbitrage and Local Price Effects in Kenyan Markets," The Quarterly Journal of Economics, Oxford University Press, vol. 134(2), pages 785-842.
    6. Reeg, Caroline, 2015. "Micro and small enterprises as drivers for job creation and decent work," IDOS Discussion Papers 10/2015, German Institute of Development and Sustainability (IDOS).
    7. Cho, Yoonyoung & Honorati, Maddalena, 2014. "Entrepreneurship programs in developing countries: A meta regression analysis," Labour Economics, Elsevier, vol. 28(C), pages 110-130.
    8. de Mel, Suresh & McKenzie, David & Woodruff, Christopher, 2014. "Business training and female enterprise start-up, growth, and dynamics: Experimental evidence from Sri Lanka," Journal of Development Economics, Elsevier, vol. 106(C), pages 199-210.
    9. Diether Beuermann & Maria Amelina, 2014. "Does Participatory Budgeting Improve Decentralized Public Service Delivery?," IDB Publications (Working Papers) 87095, Inter-American Development Bank.
    10. Túlio A. Cravo & Caio Piza, 2019. "The impact of business-support services on firm performance: a meta-analysis," Small Business Economics, Springer, vol. 53(3), pages 753-770, October.
    11. Diether W. Beuermann & Maria Amelina, 2018. "Does participatory budgeting improve decentralized public service delivery? Experimental evidence from rural Russia," Economics of Governance, Springer, vol. 19(4), pages 339-379, November.
    12. Karlan, Dean S. & Knight, Ryan & Udry, Christopher R., 2012. "Hoping to Win, Expected to Lose: Theory and Lessons on Micro Enterprise Development," Center Discussion Papers 133405, Yale University, Economic Growth Center.
    13. Lars Ivar Oppedal Berge & Kjetil Bjorvatn & Bertil Tungodden, 2015. "Human and Financial Capital for Microenterprise Development: Evidence from a Field and Lab Experiment," Management Science, INFORMS, vol. 61(4), pages 707-722, April.
    14. Islam, Asad & Lee, Wang-Sheng & Triyana, Margaret & Xia, Xing, 2023. "Improving Health and Safety in the Informal Sector: Evidence from a Randomized Trial in Bangladesh," IZA Discussion Papers 16150, Institute of Labor Economics (IZA).
    15. McKenzie, David, 2011. "How can we learn whether firm policies are working in africa ? challenges (and solutions?) for experiments and structural models," Policy Research Working Paper Series 5632, The World Bank.
    16. González-Uribe, Juanita & Reyes, Santiago, 2021. "Identifying and boosting “Gazelles”: Evidence from business accelerators," Journal of Financial Economics, Elsevier, vol. 139(1), pages 260-287.
    17. Antonia Grohmann & Lukas Menkhoff & Helke Seitz, 2022. "The Effect of Personalized Feedback on Small Enterprises’ Finances in Uganda," Economic Development and Cultural Change, University of Chicago Press, vol. 70(3), pages 1197-1227.
    18. David McKenzie, 2017. "Identifying and Spurring High-Growth Entrepreneurship: Experimental Evidence from a Business Plan Competition," American Economic Review, American Economic Association, vol. 107(8), pages 2278-2307, August.
    19. Michael Kremer & Jonathan Robinson & Olga Rostapshova, 2014. "Success in Entrepreneurship: Doing the Math," NBER Chapters, in: African Successes, Volume II: Human Capital, pages 281-303, National Bureau of Economic Research, Inc.
    20. Yuki Higuchi & Vu Hoang Nam & Tetsushi Sonobe, 2023. "Do Management Interventions Last? Evidence from Vietnamese SMEs," Working Papers DP-2022-42, Economic Research Institute for ASEAN and East Asia (ERIA).

    More about this item

    Keywords

    Microfinance; Banks&Banking Reform; Access to Finance; ICT Policy and Strategies; E-Business;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wbk:wbrwps:6296. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Roula I. Yazigi (email available below). General contact details of provider: https://edirc.repec.org/data/dvewbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.