IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v39y2010i1p67-78.html
   My bibliography  Save this article

Review and new evidence on composite innovation indicators for evaluating national performance

Author

Listed:
  • Grupp, Hariolf
  • Schubert, Torben

Abstract

The purpose of this contribution is to present a survey of the recent developments in constructing composite science and technology (S&T) indicators on a national level as well as new evidence of the variability of such S&T indicators which opens the gateway to "country-tuning". It has become standard practice to combine several indicators for science, technology, and innovation to form composite numbers. Especially in the light of this variability, two questions arise. Firstly, are the results (especially rankings) stable with respect to weights? Secondly, is there hope to define "economically" reasonable weights? In order to provide answers to these questions, we use data from the European Innovation Scoreboard 2005 (EIS 2005) to exemplify our reasoning. Concerning the first question, we give genuine evidence on the existence of immense variability, possibly invalidating the results. Further, we also show that even existing and well-accepted methods, like equal weighting, Benefit of the Doubt weighting (BoD) and principal component analysis weighting (PCA) may lead to drastically differing results. Concerning the second question we will demonstrate that by each composite indicator weighting a set of shadow prices is implied expressing one indicator in terms of another. Whether the weights are sensible should be evaluated on the basis of these shadow prices. It turns out that those implied by EIS 2005 contain strange peculiarities. After that we plead for more care in constructing composite indicators. Especially weights should be chosen on the basis of shadow prices, rather than, say, by equal weighting or other automatic methods. Lastly, we discuss the merit of composite indicators and argue that they have a valuable communication and competition function, but they should be accompanied by multidimensional representations, which provide the basis for the construction of policy measures.

Suggested Citation

  • Grupp, Hariolf & Schubert, Torben, 2010. "Review and new evidence on composite innovation indicators for evaluating national performance," Research Policy, Elsevier, vol. 39(1), pages 67-78, February.
  • Handle: RePEc:eee:respol:v:39:y:2010:i:1:p:67-78
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048-7333(09)00199-1
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Romer, Paul M, 1986. "Increasing Returns and Long-run Growth," Journal of Political Economy, University of Chicago Press, vol. 94(5), pages 1002-1037, October.
    2. Hye-Seon Moon & Jeong-Dong Lee, 2005. "A fuzzy set theory approach to national composite S&T indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 64(1), pages 67-83, July.
    3. Aghion, Philippe & Howitt, Peter, 1992. "A Model of Growth through Creative Destruction," Econometrica, Econometric Society, vol. 60(2), pages 323-351, March.
    4. Hülskamp, Nicola & Koppel, Oliver, 2005. "Deutschlands Position im Innovationswettbewerb: Ergebnisse des IW-Innovationsbenchmarkings," IW-Trends – Vierteljahresschrift zur empirischen Wirtschaftsforschung, Institut der deutschen Wirtschaft (IW) / German Economic Institute, vol. 32(3), pages 45-61.
    5. Michela Nardo & Michaela Saisana & Andrea Saltelli & Stefano Tarantola & Anders Hoffman & Enrico Giovannini, 2005. "Handbook on Constructing Composite Indicators: Methodology and User Guide," OECD Statistics Working Papers 2005/3, OECD Publishing.
    6. Archibugi, Daniele & Coco, Alberto, 2005. "Measuring technological capabilities at the country level: A survey and a menu for choice," Research Policy, Elsevier, vol. 34(2), pages 175-194, March.
    7. Grupp, Hariolf & Mogee, Mary Ellen, 2004. "Indicators for national science and technology policy: how robust are composite indicators?," Research Policy, Elsevier, vol. 33(9), pages 1373-1384, November.
    8. Archibugi, Daniele & Coco, Alberto, 2004. "A New Indicator of Technological Capabilities for Developed and Developing Countries (ArCo)," World Development, Elsevier, vol. 32(4), pages 629-654, April.
    9. Theodoros Papaioannou & Howard Rush & John Bessant, 2006. "Benchmarking as a policy-making tool: From the private to the public sector," Science and Public Policy, Oxford University Press, vol. 33(2), pages 91-102, March.
    10. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    11. Romer, Paul M, 1990. "Endogenous Technological Change," Journal of Political Economy, University of Chicago Press, vol. 98(5), pages 71-102, October.
    12. Cherchye, Laurens & Knox Lovell, C.A. & Moesen, Wim & Van Puyenbroeck, Tom, 2007. "One market, one number? A composite indicator assessment of EU internal market dynamics," European Economic Review, Elsevier, vol. 51(3), pages 749-779, April.
    13. Gene M. Grossman & Elhanan Helpman, 1993. "Innovation and Growth in the Global Economy," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262570971, December.
    14. Hariolf Grupp, 1998. "Foundations of the Economics of Innovation," Books, Edward Elgar Publishing, number 1390, December.
    15. L Cherchye & W Moesen & N Rogge & T Van Puyenbroeck & M Saisana & A Saltelli & R Liska & S Tarantola, 2008. "Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(2), pages 239-251, February.
    16. Romer, Paul M, 1987. "Growth Based on Increasing Returns Due to Specialization," American Economic Review, American Economic Association, vol. 77(2), pages 56-62, May.
    17. Laurens Cherchye & Wim Moesen & Tom Van Puyenbroeck, 2004. "Legitimately Diverse, yet Comparable: On Synthesizing Social Inclusion Performance in the EU," Journal of Common Market Studies, Wiley Blackwell, vol. 42(5), pages 919-955, December.
    18. Hollenstein, Heinz, 1996. "A composite indicator of a firm's innovativeness. An empirical analysis based on survey data for Swiss manufacturing," Research Policy, Elsevier, vol. 25(4), pages 633-645, June.
    19. Grupp, Hariolf, 1994. "The measurement of technical performance of innovations by technometrics and its impact on established technology indicators," Research Policy, Elsevier, vol. 23(2), pages 175-193, March.
    20. Robert J W Tijssen, 2003. "Scoreboards of research excellence," Research Evaluation, Oxford University Press, vol. 12(2), pages 91-103, August.
    21. Michael Freudenberg, 2003. "Composite Indicators of Country Performance: A Critical Assessment," OECD Science, Technology and Industry Working Papers 2003/16, OECD Publishing.
    22. Cazals, Catherine & Florens, Jean-Pierre & Simar, Leopold, 2002. "Nonparametric frontier estimation: a robust approach," Journal of Econometrics, Elsevier, vol. 106(1), pages 1-25, January.
    23. Frederik Booysen, 2002. "An Overview and Evaluation of Composite Indices of Development," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 59(2), pages 115-151, August.
    24. Jon Zabala-Iturriagagoitia & Peter Voigt & Antonio Gutierrez-Gracia & Fernando Jimenez-Saez, 2007. "Regional Innovation Systems: How to Assess Performance," Regional Studies, Taylor & Francis Journals, vol. 41(5), pages 661-672.
    25. Axel Werwatz & Heike Belitz & Tanja Kirn & Jens Schmidt-Ehmcke, 2006. "Innovationsindikator Deutschland 2006," DIW Berlin: Politikberatung kompakt, DIW Berlin, German Institute for Economic Research, volume 22, number pbk22.
    26. Marianne Paasi, 2005. "Collective benchmarking of policies: an instrument for policy learning in adaptive research and innovation policy," Science and Public Policy, Oxford University Press, vol. 32(1), pages 17-27, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Laurens CHERCHYE & Willem MOESEN & Nicky ROGGE & Tom VAN PUYENBROECK, 2009. "Constructing a knowledge economy composite indicator with imprecise data," Working Papers of Department of Economics, Leuven ces09.15, KU Leuven, Faculty of Economics and Business (FEB), Department of Economics, Leuven.
    2. L Cherchye & W Moesen & N Rogge & T Van Puyenbroeck & M Saisana & A Saltelli & R Liska & S Tarantola, 2008. "Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(2), pages 239-251, February.
    3. Van Puyenbroeck, Tom & Rogge, Nicky, 2017. "Geometric mean quantity index numbers with Benefit-of-the-Doubt weights," European Journal of Operational Research, Elsevier, vol. 256(3), pages 1004-1014.
    4. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    5. Laurens Cherchye & Willem Moesen & Nicky Rogge & Tom Puyenbroeck, 2007. "An Introduction to ‘Benefit of the Doubt’ Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 82(1), pages 111-145, May.
    6. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    7. Teemu Makkonen & Robert P. Have, 2013. "Benchmarking regional innovative performance: composite measures and direct innovation counts," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 247-262, January.
    8. Fabio Pammolli & Francesco Porcelli & Francesco Vidoli & Guido Borà, 2014. "La spesa sanitaria delle Regioni in Italia - Saniregio 3," Working Papers CERM 02-2014, Competitività, Regole, Mercati (CERM).
    9. Fabio Pammolli & Francesco Porcelli & Francesco Vidoli & Monica Auteri & Guido Borà, 2017. "La spesa sanitaria delle Regioni in Italia - Saniregio2017," Working Papers CERM 01-2017, Competitività, Regole, Mercati (CERM).
    10. Khatab Alqararah, 2023. "Assessing the robustness of composite indicators: the case of the Global Innovation Index," Journal of Innovation and Entrepreneurship, Springer, vol. 12(1), pages 1-22, December.
    11. Kristof Witte & Nicky Rogge, 2010. "To publish or not to publish? On the aggregation and drivers of research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 657-680, December.
    12. Fagerberg, Jan & Srholec, Martin & Knell, Mark, 2007. "The Competitiveness of Nations: Why Some Countries Prosper While Others Fall Behind," World Development, Elsevier, vol. 35(10), pages 1595-1620, October.
    13. Fagerberg, Jan & Srholec, Martin & Verspagen, Bart, 2010. "Innovation and Economic Development," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 2, chapter 0, pages 833-872, Elsevier.
    14. Daniele Archibugi & Francesco Crespi & Mario Denni & Andrea Filippetti, 2009. "The Technological Capabilities of Nations: A Survey of Composite Indicators," QA - Rivista dell'Associazione Rossi-Doria, Associazione Rossi Doria, issue 2, May.
    15. Arazmuradov, Annageldy & Martini, Gianmaria & Scotti, Davide, 2014. "Determinants of total factor productivity in former Soviet Union economies: A stochastic frontier approach," Economic Systems, Elsevier, vol. 38(1), pages 115-135.
    16. Francesco Vidoli & Elisa Fusco & Claudio Mazziotta, 2015. "Non-compensability in Composite Indicators: A Robust Directional Frontier Method," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 122(3), pages 635-652, July.
    17. Fabio Pammolli & Francesco Porcelli & Francesco Vidoli & Guido Borà, 2015. "La spesa sanitaria delle Regioni in Italia - Saniregio 2015," Working Papers CERM 01-2015, Competitività, Regole, Mercati (CERM), revised 04 Jan 2016.
    18. P. Zhou & B. Ang, 2009. "Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(1), pages 83-96, October.
    19. Andrea Bonaccorsi & Cinzia Daraio, 2013. "Knowledge spillover effects at the sub-regional level. Theory and estimation," DIAG Technical Reports 2013-13, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    20. Radosevic, Slavo & Yoruk, Esin, 2018. "Technology upgrading of middle income economies: A new approach and results," Technological Forecasting and Social Change, Elsevier, vol. 129(C), pages 56-75.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:39:y:2010:i:1:p:67-78. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.