Advanced Search
MyIDEAS: Login to save this paper or follow this series

Evaluating the Impact of Science, Technology and Innovation Programs: a Methodological Toolkit

Contents:

Author Info

  • Gustavo A. Crespi

    ()

  • Alessandro Maffioli

    ()

  • Pierre Mohnen

    ()

  • Gonzalo Vázquez

    ()

Abstract

The purpose of this guideline is to provide ideas and technical advice on how to measure the effectiveness of Science, Technology and Innovation Programs (STIP). The paper addresses the specific challenges of evaluating STIP, from the assessment of the intervention logic to the choice of the most appropriate method to solve the attribution problem. Much attention is devoted to the topic of data, discussing pros and cons of different data sources, data quality issues, and strategies for data collection. The paper analyzes in detail the potential application of experimental and quasi-experimental methods to STIP. For each method, the paper highlights characteristics and assumptions, practical issues related to the implementation, and strengths and weakness specifically related to the application to STIP. Other specific issues related to the evaluation of STIP are also considered: the timing of effects, intensity of treatment, multiple treatments, impact heterogeneity, externalities, and general equilibrium effects. Concrete examples of rigorous evaluations of STIP support the discussion of the various topics throughout the guideline.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL: http://www.iadb.org/document.cfm?id=36693990
Download Restriction: no

Bibliographic Info

Paper provided by Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD) in its series SPD Working Papers with number 1104.

as in new window
Length:
Date of creation: Dec 2011
Date of revision:
Handle: RePEc:idb:spdwps:1104

Contact details of provider:
Postal: 1300 New York Avenue, NW, Washington, DC 20577
Phone: 202-623-1000
Email:
Web page: http://www.iadb.org/spd
More information through EDIRC

Related research

Keywords: Impact Evaluation; Science; Technology Innovation; Development Effectiveness;

Find related papers by JEL classification:

This paper has been announced in the following NEP Reports:

References

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-93, September.
  2. Hall, Bronwyn & Van Reenen, John, 2000. "How effective are fiscal incentives for R&D? A review of the evidence," Research Policy, Elsevier, vol. 29(4-5), pages 449-469, April.
  3. Bronwyn Hall & Alessandro Maffioli, 2008. "Evaluating the impact of technology development funds in emerging economies: evidence from Latin America," European Journal of Development Research, Taylor and Francis Journals, vol. 20(2), pages 172-198.
  4. Soete, Luc & Verspagen, Bart & Weel, Bas ter, 2009. "Systems of Innovation," MERIT Working Papers 062, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
  5. Diego Ubfal & Alessandro Maffioli, 2010. "The Impact of Funding on Research Collaboration: Evidence from Argentina," IDB Publications 9395, Inter-American Development Bank.
  6. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of research grant funding on scientific productivity," Journal of Public Economics, Elsevier, vol. 95(9-10), pages 1168-1177, October.
  7. Nelson, Richard R, 1981. "Research on Productivity Growth and Productivity Differences: Dead Ends and New Departures," Journal of Economic Literature, American Economic Association, vol. 19(3), pages 1029-64, September.
  8. Guido Imbens & Karthik Kalyanaraman, 2009. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," NBER Working Papers 14726, National Bureau of Economic Research, Inc.
  9. Raffaello Bronzini & Eleonora Iachini, 2011. "Are incentives for R&D effective? Evidence from a regression discontinuity approach," Temi di discussione (Economic working papers) 791, Bank of Italy, Economic Research and International Relations Area.
  10. Blanes, J. Vicente & Busom, Isabel, 2004. "Who participates in R&D subsidy programs?: The case of Spanish manufacturing firms," Research Policy, Elsevier, vol. 33(10), pages 1459-1476, December.
  11. Guido Imbens & Jeffrey Wooldridge, 2008. "Recent developments in the econometrics of program evaluation," CeMMAP working papers CWP24/08, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  12. Josh Lerner, 2002. "When Bureaucrats Meet Entrepreneurs: The Design of Effective "Public Venture Capital" Programmes," Economic Journal, Royal Economic Society, vol. 112(477), pages F73-F84, February.
  13. Sebastian Galiani & Paul Gertler & Ernesto Schargrodsky, 2002. "Water for Life: The Impact of the Privatization of Water Services on Child Mortality," Working Papers 54, Universidad de San Andres, Departamento de Economia, revised Sep 2005.
  14. Czarnitzki, Dirk, 2002. "Research and development: financial constraints and the role of public funding for small and medium-sized enterprises," ZEW Discussion Papers 02-74, ZEW - Zentrum für Europäische Wirtschaftsforschung / Center for European Economic Research.
  15. Abhijit V. Banerjee & Esther Duflo, 2008. "The Experimental Approach to Development Economics," NBER Working Papers 14467, National Bureau of Economic Research, Inc.
  16. David S. Lee & Thomas Lemieux, 2009. "Regression Discontinuity Designs in Economics," NBER Working Papers 14723, National Bureau of Economic Research, Inc.
  17. Martin, Stephen & Scott, John T., 2000. "The nature of innovation market failure and the design of public support for private innovation," Research Policy, Elsevier, vol. 29(4-5), pages 437-447, April.
  18. Markus Frölich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 77-90, February.
  19. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, 02.
  20. O. Carboni, 2008. "The Effect of R&D Subsidies on Private R&D: Evidence from Italian Manufacturing Data," Working Paper CRENoS 200815, Centre for North South Economic Research, University of Cagliari and Sassari, Sardinia.
  21. Harding, Matthew & Lamarche, Carlos, 2009. "A quantile regression approach for estimating panel data models using instrumental variables," Economics Letters, Elsevier, vol. 104(3), pages 133-135, September.
  22. Sergio Firpo, 2004. "Efficient Semiparametric Estimation of Quantile Treatment Effects," Econometric Society 2004 North American Summer Meetings 605, Econometric Society.
  23. Mohnen, Pierre & Bérubé, Charles, 2007. "Are Firms That Received R&D Subsidies More Innovative?," MERIT Working Papers 015, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
  24. Victor Chernozhukov & Christian Hansen, 2005. "An IV Model of Quantile Treatment Effects," Econometrica, Econometric Society, vol. 73(1), pages 245-261, 01.
  25. Frölich, Markus & Melly, Blaise, 2008. "Unconditional Quantile Treatment Effects under Endogeneity," IZA Discussion Papers 3288, Institute for the Study of Labor (IZA).
  26. Steve Bond, 2002. "Dynamic panel data models: a guide to microdata methods and practice," CeMMAP working papers CWP09/02, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  27. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
  28. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2004. "Dealing with Limited Overlap in Estimation of Average Treatment Effects," Working Papers 0716, University of Miami, Department of Economics, revised 12 Jun 2007.
  29. Joshua Angrist & Alan Krueger, 1990. "Does Compulsory School Attendance Affect Schooling and Earnings?," Working Papers 653, Princeton University, Department of Economics, Industrial Relations Section..
  30. Heckman, James J & Smith, Jeffrey, 1997. "Making the Most Out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts," Review of Economic Studies, Wiley Blackwell, vol. 64(4), pages 487-535, October.
  31. Reiss, Peter C. & Wolak, Frank A., 2007. "Structural Econometric Modeling: Rationales and Examples from Industrial Organization," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 64 Elsevier.
  32. Heckman, James J & Ichimura, Hidehiko & Todd, Petra E, 1997. "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Wiley Blackwell, vol. 64(4), pages 605-54, October.
  33. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
  34. Koenker, Roger W & Bassett, Gilbert, Jr, 1978. "Regression Quantiles," Econometrica, Econometric Society, vol. 46(1), pages 33-50, January.
  35. Lokshin, Boris & Mohnen, Pierre, 2010. "How effective are level-based R&D tax credits? Evidence from the Netherlands," MERIT Working Papers 040, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
  36. Cohen, Wesley M & Levinthal, Daniel A, 1989. "Innovation and Learning: The Two Faces of R&D," Economic Journal, Royal Economic Society, vol. 99(397), pages 569-96, September.
  37. José Miguel Benavente & Gustavo Crespi & Alessandro Maffioli, 2007. "Public Support to Firm-Level Innovation: An Evaluation of the FONTEC Program," OVE Working Papers 0507, Inter-American Development Bank, Office of Evaluation and Oversight (OVE).
  38. Bhattacharya, Sudipto & Ritter, Jay R, 1983. "Innovation and Communication: Signalling with Partial Disclosure," Review of Economic Studies, Wiley Blackwell, vol. 50(2), pages 331-46, April.
  39. Lerner, Josh, 1999. "The Government as Venture Capitalist: The Long-Run Impact of the SBIR Program," The Journal of Business, University of Chicago Press, vol. 72(3), pages 285-318, July.
  40. Carolyn Heinrich & Alessandro Maffioli & Gonzalo Vázquez, 2010. "A Primer for Applying Propensity-Score Matching," SPD Working Papers 1005, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
  41. Stiglitz, Joseph E & Weiss, Andrew, 1981. "Credit Rationing in Markets with Imperfect Information," American Economic Review, American Economic Association, vol. 71(3), pages 393-410, June.
  42. Heshmati, Almas & Loof, Hans, 2005. "The Impact of Public Funds on Private R&D Investment: New Evidence from a Firm Level Innovation Study," Discussion Papers 11862, MTT Agrifood Research Finland.
  43. Manuela Angelucci & Vincenzo Di Maro, 2010. "Program Evaluation and Spillover Effects," SPD Working Papers 1003, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
  44. Bruno Van Pottelsberghe & Steve Nysten & Esmeralda Megally, 2003. "Evaluation of current fiscal incentives for business R&D in Belgium," Working Papers CEB 03-011.RS, ULB -- Universite Libre de Bruxelles.
Full references (including those not matched with items on IDEAS)

Citations

Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. Dezhina, I. & Simachev, Yu., 2013. "Matching Grants for Stimulating Partnerships between Companies and Universities in Innovation Area: Initial Effects in Russia," Journal of the New Economic Association, New Economic Association, vol. 19(3), pages 99-122.
  2. Mikhail Kuzyk & Yury Simachev, 2013. "Russia's Innovation Promotion Policies: Their Evolution, Achievements, Problems and Lessons," Published Papers 164, Gaidar Institute for Economic Policy, revised 2013.
  3. Giuliani, Elisa & Maffioli, Alessandro & Pacheco, Manuel & Pietrobelli, Carlo & Stucchi, Rodolfo, 2014. "Evaluating the Impact of Cluster Development Programs," CIRCLE Electronic Working Papers 2014/10, Lund University, CIRCLE - Center for Innovation, Research and Competences in the Learning Economy.

Lists

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

Statistics

Access and download statistics

Corrections

When requesting a correction, please mention this item's handle: RePEc:idb:spdwps:1104. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Monica Bazan).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.