IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v44y2015i8p1462-1472.html
   My bibliography  Save this article

Assessing an experimental approach to industrial policy evaluation: Applying RCT+ to the case of Creative Credits

Author

Listed:
  • Bakhshi, Hasan
  • Edwards, John S.
  • Roper, Stephen
  • Scully, Judy
  • Shaw, Duncan
  • Morley, Lorraine
  • Rathbone, Nicola

Abstract

Experimental methods of policy evaluation are well-established in social policy and development economics but are rare in industrial and innovation policy. In this paper, we consider the arguments for applying experimental methods to industrial policy measures, and propose an experimental policy evaluation approach (which we call RCT+). This approach combines the randomised assignment of firms to treatment and control groups with a longitudinal data collection strategy incorporating quantitative and qualitative data (so-called mixed methods). The RCT+ approach is designed to provide a causative rather than purely summative evaluation, i.e. to assess both ‘whether’ and ‘how’ programme outcomes are achieved. In this paper, we assess the RCT+ approach through an evaluation of Creative Credits – a UK business-to-business innovation voucher initiative intended to promote new innovation partnerships between SMEs and creative service providers. The results suggest the potential value of the RCT+ approach to industrial policy evaluation, and the benefits of mixed methods and longitudinal data collection.

Suggested Citation

  • Bakhshi, Hasan & Edwards, John S. & Roper, Stephen & Scully, Judy & Shaw, Duncan & Morley, Lorraine & Rathbone, Nicola, 2015. "Assessing an experimental approach to industrial policy evaluation: Applying RCT+ to the case of Creative Credits," Research Policy, Elsevier, vol. 44(8), pages 1462-1472.
  • Handle: RePEc:eee:respol:v:44:y:2015:i:8:p:1462-1472
    DOI: 10.1016/j.respol.2015.04.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733315000645
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.respol.2015.04.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    2. Nicholas Bloom & Benn Eifert & Aprajit Mahajan & David McKenzie & John Roberts, 2013. "Does Management Matter? Evidence from India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(1), pages 1-51.
    3. Hewitt-Dundas, Nola & Roper, Stephen, 2011. "Creating advantage in peripheral regions: The role of publicly funded R&D centres," Research Policy, Elsevier, vol. 40(6), pages 832-841, July.
    4. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    5. G. M.P. Swann, 2009. "The Economics of Innovation," Books, Edward Elgar Publishing, number 13211, December.
    6. Tuomas Takalo & Tanja Tanayama, 2010. "Adverse selection and financing of innovation: is there a need for R&D subsidies?," The Journal of Technology Transfer, Springer, vol. 35(1), pages 16-41, February.
    7. Paul Nightingale & Alex Coad, 2014. "Muppets and gazelles: political and methodological biases in entrepreneurship research," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 23(1), pages 113-143, February.
    8. Roper, Stephen & Arvanitis, Spyros, 2012. "From knowledge to added value: A comparative, panel-data analysis of the innovation value chain in Irish and Swiss manufacturing firms," Research Policy, Elsevier, vol. 41(6), pages 1093-1106.
    9. Garcia, Abraham & Mohnen, Pierre, 2010. "Impact of government support on R&D and innovation," MERIT Working Papers 2010-034, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    10. repec:pri:rpdevs:deaton_instruments_randomization_learning_all_04april_2010 is not listed on IDEAS
    11. Donaldson, Stewart I. & Gooler, Laura E., 2003. "Theory-driven evaluation in action: lessons from a $20 million statewide Work and Health Initiative," Evaluation and Program Planning, Elsevier, vol. 26(4), pages 355-366, November.
    12. Bratberg, Espen & Grasdal, Astrid & Risa, Alf Erling, 2002. " Evaluating Social Policy by Experimental and Nonexperimental Methods," Scandinavian Journal of Economics, Wiley Blackwell, vol. 104(1), pages 147-171.
    13. Maarten Cornet & Björn Vroomen & Marc van der Steeg, 2006. "Do innovation vouchers help SMEs to cross the bridge towards science?," CPB Discussion Paper 58, CPB Netherlands Bureau for Economic Policy Analysis.
    14. Nola Hewitt-Dundas & Stephen Roper, 2009. "Output Additionality of Public Support for Innovation: Evidence for Irish Manufacturing Plants," European Planning Studies, Taylor & Francis Journals, vol. 18(1), pages 107-122, September.
    15. Espen Bratberg & Astrid Grasdal & Alf Erling Risa, 2002. "Evaluating Social Policy by Experimental and Nonexperimental Methods," Scandinavian Journal of Economics, Wiley Blackwell, vol. 104(1), pages 147-171, March.
    16. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    17. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    18. Johan Lambrecht & Fabrice Pirnay, 2005. "An evaluation of public support measures for private external consultancies to SMEs in the Walloon Region of Belgium," Entrepreneurship & Regional Development, Taylor & Francis Journals, vol. 17(2), pages 89-108, March.
    19. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    20. Meuleman, Miguel & De Maeseneire, Wouter, 2012. "Do R&D subsidies affect SMEs’ access to external financing?," Research Policy, Elsevier, vol. 41(3), pages 580-591.
    21. Lerner, Josh, 1999. "The Government as Venture Capitalist: The Long-Run Impact of the SBIR Program," The Journal of Business, University of Chicago Press, vol. 72(3), pages 285-318, July.
    22. Rosenbusch, Nina & Brinckmann, Jan & Bausch, Andreas, 2011. "Is innovation always beneficial? A meta-analysis of the relationship between innovation and performance in SMEs," Journal of Business Venturing, Elsevier, vol. 26(4), pages 441-457, July.
    23. Trüby, Johannes & Rammer, Christian & Müller, Kathrin, 2008. "The Role of Creative Industries in Industrial Innovation," ZEW Discussion Papers 08-109, ZEW - Leibniz Centre for European Economic Research.
    24. Forsman, Helena, 2011. "Innovation capacity and innovation development in small enterprises. A comparison between the manufacturing and service sectors," Research Policy, Elsevier, vol. 40(5), pages 739-750, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Caloffi, Annalisa & Freo, Marzia & Ghinoi, Stefano & Mariani, Marco & Rossi, Federica, 2022. "Assessing the effects of a deliberate policy mix: The case of technology and innovation advisory services and innovation vouchers," Research Policy, Elsevier, vol. 51(6).
    2. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    3. Kleine, Marco & Heite, Jonas & Huber, Laura Rosendahl, 2022. "Subsidized R&D collaboration: The causal effect of innovation vouchers on innovation outcomes," Research Policy, Elsevier, vol. 51(6).
    4. Daunfeldt, Sven-Olov & Halvarsson, Daniel & Gustavsson Tingvall, Patrik & McKelvie, Alexander, 2021. "Do Targeted R&D Grants Towards Potential Highgrowth Firms Increase Employment and Demand for High Human Capital Workers?," HFI Working Papers 23, Institute of Retail Economics (Handelns Forskningsinstitut).
    5. Viktorie Klimova & Klaudia Glittova & Vladimir Zitek, 2023. "Innovation vouchers and cooperation: a different approach in two countries with a shared history," Eastern Journal of European Studies, Centre for European Studies, Alexandru Ioan Cuza University, vol. 14, pages 22-44, December.
    6. Chiara Dalle Nogare & Monika Murzyn-Kupisz, 2021. "Do museums foster innovation through engagement with the cultural and creative industries?," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 45(4), pages 671-704, December.
    7. Klaudia Bracio & Marek Szarucki, 2020. "Mixed Methods Utilisation in Innovation Management Research: A Systematic Literature Review and Meta-Summary," JRFM, MDPI, vol. 13(11), pages 1-27, October.
    8. Erik E. Lehmann & Matthias Menter, 2018. "Public cluster policy and performance," The Journal of Technology Transfer, Springer, vol. 43(3), pages 558-592, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hasan Bakhshi & John Edwards & Stephen Roper & Judy Scully & Duncan Shaw & Lorraine Morley & Nicola Rathbone, 2013. "An Experimental Approach to Industrial Policy Evaluation: The case of Creative Credits," Research Papers 0004, Enterprise Research Centre.
    2. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    3. Gani Aldashev & Georg Kirchsteiger & Alexander Sebald, 2017. "Assignment Procedure Biases in Randomised Policy Experiments," Economic Journal, Royal Economic Society, vol. 127(602), pages 873-895, June.
    4. Andrea Bellucci & Luca Pennacchio & Alberto Zazzaro, 2019. "R&D Subsidies and Firms’ Debt Financing," CSEF Working Papers 527, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy.
    5. Brautzsch, Hans-Ulrich & Günther, Jutta & Loose, Brigitte & Ludwig, Udo & Nulsch, Nicole, 2015. "Can R&D subsidies counteract the economic crisis? – Macroeconomic effects in Germany," Research Policy, Elsevier, vol. 44(3), pages 623-633.
    6. Wu, Aihua, 2017. "The signal effect of Government R&D Subsidies in China: Does ownership matter?," Technological Forecasting and Social Change, Elsevier, vol. 117(C), pages 339-345.
    7. Sarah Demeulemeester & Hanna Hottenrott, 2015. "R&D subsidies and firms' cost of debt," Working Papers of Department of Management, Strategy and Innovation, Leuven 516028, KU Leuven, Faculty of Economics and Business (FEB), Department of Management, Strategy and Innovation, Leuven.
    8. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    9. Isabel Busom & Beatriz Corchuelo & Ester Martinez Ros, 2012. "Tax incentives and direct support for R&D: What do firms use and why?," Working Papers wpdea1212, Department of Applied Economics at Universitat Autonoma of Barcelona.
    10. Albert Bravo-Biosca, 2020. "Experimental Innovation Policy," Innovation Policy and the Economy, University of Chicago Press, vol. 20(1), pages 191-232.
    11. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    12. Astrid Grasdal, 2001. "The performance of sample selection estimators to control for attrition bias," Health Economics, John Wiley & Sons, Ltd., vol. 10(5), pages 385-398, July.
    13. Kate Baldwin & Rikhil R. Bhavnani, 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series wp-2013-024, World Institute for Development Economic Research (UNU-WIDER).
    14. Stjepan Srhoj & Bruno Škrinjarić & Sonja Radas, 2021. "Bidding against the odds? The impact evaluation of grants for young micro and small firms during the recession," Small Business Economics, Springer, vol. 56(1), pages 83-103, January.
    15. Chiappini, Raphaël & Montmartin, Benjamin & Pommet, Sophie & Demaria, Samira, 2022. "Can direct innovation subsidies relax SMEs’ financial constraints?," Research Policy, Elsevier, vol. 51(5).
    16. Giebel, Marek & Kraft, Kornelius, 2021. "Subsidies and innovation in the recent financial crisis," ZEW Discussion Papers 21-097, ZEW - Leibniz Centre for European Economic Research.
    17. Ensthaler, Ludwig & Giebe, Thomas, 2014. "A dynamic auction for multi-object procurement under a hard budget constraint," Research Policy, Elsevier, vol. 43(1), pages 179-189.
    18. Yonatan Eyal, 2020. "Self-Assessment Variables as a Source of Information in the Evaluation of Intervention Programs: A Theoretical and Methodological Framework," SAGE Open, , vol. 10(1), pages 21582440198, January.
    19. Chen, Jin & Heng, Cheng Suang & Tan, Bernard C.Y. & Lin, Zhijie, 2018. "The distinct signaling effects of R&D subsidy and non-R&D subsidy on IPO performance of IT entrepreneurial firms in China," Research Policy, Elsevier, vol. 47(1), pages 108-120.
    20. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:44:y:2015:i:8:p:1462-1472. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.