IDEAS home Printed from https://ideas.repec.org/p/aeg/report/2017-04.html
   My bibliography  Save this paper

A framework for the Assessment of Research and its impacts

Author

Listed:
  • Cinzia Daraio

    () (Department of Computer, Control and Management Engineering Antonio Ruberti (DIAG), University of Rome La Sapienza, Rome, Italy)

Abstract

This paper proposes a framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions: theory, methodology and data, each of which is further characterized by three main building blocks: education, research and innovation (theory); efficiency, effectiveness and impact (methodology); and availability, interoperability and \unit free" property (data). The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as "quality". Three additional quality attributes are identified as implementation factors (tailorability, transparency and openness) and three "enabling" conditions (convergence, mixed methods and knowledge infrastructures) complete the framework.The paper illustrates the complexity of the evaluation describing the generalized "implementation problem" in research assessment, according to the proposed framework. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation. Three examples of application, as well as further directions for future research are provided.

Suggested Citation

  • Cinzia Daraio, 2017. "A framework for the Assessment of Research and its impacts," DIAG Technical Reports 2017-04, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
  • Handle: RePEc:aeg:report:2017-04
    as

    Download full text from publisher

    File URL: http://wwwold.dis.uniroma1.it/~bibdis/index.php?option=com_docman&task=doc_download&gid=116&Itemid=34
    File Function: First version, 2017
    Download Restriction: no

    References listed on IDEAS

    as
    1. Ki-Seok Kwon & Han Woo Park & Minho So & Loet Leydesdorff, 2012. "Has globalization strengthened South Korea’s national research system? National and international dynamics of the Triple Helix of scientific co-authorship relationships in South Korea," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(1), pages 163-176, January.
    2. Romer, Paul M, 1986. "Increasing Returns and Long-run Growth," Journal of Political Economy, University of Chicago Press, vol. 94(5), pages 1002-1037, October.
    3. Feldman, Maryann & Kenney, Martin & Lissoni, Francesco, 2015. "The New Data Frontier," Research Policy, Elsevier, vol. 44(9), pages 1629-1632.
    4. Jérôme Vandenbussche & Philippe Aghion & Costas Meghir, 2006. "Growth, distance to frontier and composition of human capital," Journal of Economic Growth, Springer, vol. 11(2), pages 97-127, June.
    5. Griliches, Zvi, 1986. "Economic data issues," Handbook of Econometrics, in: Z. Griliches† & M. D. Intriligator (ed.), Handbook of Econometrics, edition 1, volume 3, chapter 25, pages 1465-1514, Elsevier.
    6. Daraio, Cinzia & Simar, Léopold, 2014. "Directional distances and their robust versions: Computational and testing issues," European Journal of Operational Research, Elsevier, vol. 237(1), pages 358-369.
    7. Fredrik Niclas Piro & Gunnar Sivertsen, 2016. "How can differences in international university rankings be explained?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2263-2278, December.
    8. Mark A. Largent & Julia I. Lane, 2012. "STAR METRICS and the Science of Science Policy," Review of Policy Research, Policy Studies Organization, vol. 29(3), pages 431-438, May.
    9. Etzkowitz, Henry & Leydesdorff, Loet, 2000. "The dynamics of innovation: from National Systems and "Mode 2" to a Triple Helix of university-industry-government relations," Research Policy, Elsevier, vol. 29(2), pages 109-123, February.
    10. Cinzia Daraio & Maurizio Lenzerini & Claudio Leporelli & Paolo Naggar & Andrea Bonaccorsi & Alessandro Bartolucci, 2016. "The advantages of an Ontology-Based Data Management approach: openness, interoperability and data quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 441-455, July.
    11. Martin Kowarsch & Jennifer Garard & Pauline Riousset & Dominic Lenzi & Marcel J. Dorsch & Brigitte Knopf & Jan-Albrecht Harrs & Ottmar Edenhofer, 2016. "Scientific assessments to facilitate deliberative policy learning," Palgrave Communications, Palgrave Macmillan, vol. 2(1), pages 1-20, December.
    12. Christopher F. Parmeter & Alan T. K. Wan & Xinyu Zhang, 2019. "Model averaging estimators for the stochastic frontier model," Journal of Productivity Analysis, Springer, vol. 51(2), pages 91-103, June.
    13. Ludger Vessman & Jerik Hanushek, 2007. "The role of education quality in economic growth (Part I)," Educational Studies, Higher School of Economics, issue 2, pages 86-116.
    14. Jorge Galán & Helena Veiga & Michael Wiper, 2014. "Bayesian estimation of inefficiency heterogeneity in stochastic frontier models," Journal of Productivity Analysis, Springer, vol. 42(1), pages 85-101, August.
    15. Chien-Ming Chen & Magali A. Delmas & Marvin B. Lieberman, 2015. "Production frontier methodologies and efficiency as a performance measure in strategic management research," Strategic Management Journal, Wiley Blackwell, vol. 36(1), pages 19-36, January.
    16. Fagerberg, Jan & Martin, Ben R. & Andersen, Esben Sloth (ed.), 2013. "Innovation Studies: Evolution and Future Challenges," OUP Catalogue, Oxford University Press, number 9780199686353.
    17. Oecd, 2015. "Making Open Science a Reality," OECD Science, Technology and Industry Policy Papers 25, OECD Publishing.
    18. Tony Lawson, 2012. "Ontology and the study of social reality: emergence, organisation, community, power, social relations, corporations, artefacts and money," Cambridge Journal of Economics, Oxford University Press, vol. 36(2), pages 345-385.
    19. Hildebrandt, Steen, 1977. "Implementation of the operations research/management science process," European Journal of Operational Research, Elsevier, vol. 1(5), pages 289-294, September.
    20. Harvey Leibenstein, 1975. "Aspects of the X-Efficiency Theory of the Firm," Bell Journal of Economics, The RAND Corporation, vol. 6(2), pages 580-606, Autumn.
    21. Hanushek, Eric A. & Woessmann, Ludger, 2007. "The role of education quality for economic growth," Policy Research Working Paper Series 4122, The World Bank.
    22. Hanushek, Eric A. & Jamison, Dean T. & Jamison, Eliot A. & Wößmann, Ludger, 2008. "Education and economic growth: It’s not just going to school, but learning something while there that matters," Munich Reprints in Economics 20467, University of Munich, Department of Economics.
    23. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    24. Martin, Ben R., 2012. "The evolution of science policy and innovation studies," Research Policy, Elsevier, vol. 41(7), pages 1219-1239.
    25. Leibenstein, Harvey & Maital, Shlomo, 1992. "Empirical Estimation and Partitioning of X-Inefficiency: A Data-Envelopment Approach," American Economic Review, American Economic Association, vol. 82(2), pages 428-433, May.
    26. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    27. Lane, Julia I. & Owen-Smith, Jason & Rosen, Rebecca F. & Weinberg, Bruce A., 2015. "New linked data on research investments: Scientific workforce, productivity, and public value," Research Policy, Elsevier, vol. 44(9), pages 1659-1671.
    28. Cinzia Daraio & Maurizio Lenzerini & Claudio Leporelli & Henk F. Moed & Paolo Naggar & Andrea Bonaccorsi & Alessandro Bartolucci, 2016. "Data integration for research and innovation policy: an Ontology-Based Data Management approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 857-871, February.
    29. Bronwyn H. Hall & Nathan Rosenberg (ed.), 2010. "Handbook of the Economics of Innovation," Handbook of the Economics of Innovation, Elsevier, edition 1, volume 1, number 1, 00.
    30. Cinzia Daraio & Wolfgang Glänzel, 2016. "Grand challenges in data integration—state of the art and future perspectives: an introduction," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 391-400, July.
    31. Cinzia Daraio & Leopold Simar & Paul W. Wilson, 2016. "Nonparametric Estimation of Efficiency in the Presence of Environmental Variables," DIAG Technical Reports 2016-02, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    32. Leibenstein, Harvey, 1978. "On the Basic Proposition of X-Efficiency Theory," American Economic Review, American Economic Association, vol. 68(2), pages 328-332, May.
    33. Geraint Johnes & Jill Johnes (ed.), 2004. "International Handbook on the Economics of Education," Books, Edward Elgar Publishing, number 2847.
    34. Richard Owen & Phil Macnaghten & Jack Stilgoe, 2012. "Responsible research and innovation: From science in society to science for society, with society," Science and Public Policy, Oxford University Press, vol. 39(6), pages 751-760, December.
    35. Borrás, Susana & Edquist, Charles, 2013. "The Choice of Innovation Policy Instruments," Papers in Innovation Studies 2013/4, Lund University, CIRCLE - Center for Innovation, Research and Competences in the Learning Economy.
    36. Léopold Simar & Paul W. Wilson, 2015. "Statistical Approaches for Non-parametric Frontier Models: A Guided Tour," International Statistical Review, International Statistical Institute, vol. 83(1), pages 77-110, April.
    37. Peter Dahler-Larsen, 2014. "Constitutive Effects of Performance Indicators: Getting beyond unintended consequences," Public Management Review, Taylor & Francis Journals, vol. 16(7), pages 969-986, October.
    38. Hamid Ekbia & Michael Mattioli & Inna Kouper & G. Arave & Ali Ghazinejad & Timothy Bowman & Venkata Ratandeep Suri & Andrew Tsou & Scott Weingart & Cassidy R. Sugimoto, 2015. "Big data, bigger dilemmas: A critical review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(8), pages 1523-1545, August.
    39. Mansfield, Edwin, 1995. "Academic Research Underlying Industrial Innovations:," The Review of Economics and Statistics, MIT Press, vol. 77(1), pages 55-65, February.
    40. Stewart, Jenny, 1995. "Models of priority-setting for public sector research," Research Policy, Elsevier, vol. 24(1), pages 115-126, January.
    41. Salter, Ammon J. & Martin, Ben R., 2001. "The economic benefits of publicly funded basic research: a critical review," Research Policy, Elsevier, vol. 30(3), pages 509-532, March.
    42. Blaise Cronin, 2013. "Thinking about data," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(3), pages 435-436, March.
    43. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, . "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications, The World Bank, number 2693.
    44. Rosenberg, Nathan & Nelson, Richard R., 1994. "American universities and technical advance in industry," Research Policy, Elsevier, vol. 23(3), pages 323-348, May.
    45. Leibenstein, Harvey, 1978. "General X-Efficiency Theory and Economic Development," OUP Catalogue, Oxford University Press, number 9780195023800.
    46. Mansfield, Edwin, 1991. "Academic research and industrial innovation," Research Policy, Elsevier, vol. 20(1), pages 1-12, February.
    47. Borrás, Susana & Edquist, Charles, 2013. "The choice of innovation policy instruments," Technological Forecasting and Social Change, Elsevier, vol. 80(8), pages 1513-1522.
    48. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    49. Golan, Amos, 2008. "Information and Entropy Econometrics — A Review and Synthesis," Foundations and Trends(R) in Econometrics, now publishers, vol. 2(1–2), pages 1-145, February.
    50. Perkmann, Markus & Tartari, Valentina & McKelvey, Maureen & Autio, Erkko & Broström, Anders & D’Este, Pablo & Fini, Riccardo & Geuna, Aldo & Grimaldi, Rosa & Hughes, Alan & Krabel, Stefan & Kitson, Mi, 2013. "Academic engagement and commercialisation: A review of the literature on university–industry relations," Research Policy, Elsevier, vol. 42(2), pages 423-442.
    51. Paul M. Romer, 1994. "The Origins of Endogenous Growth," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 3-22, Winter.
    52. Herbert Simon, 2000. "Bounded rationality in social science: Today and tomorrow," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 1(1), pages 25-39, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cinzia Daraio & Alessio Vaccari, 2019. "Sorting out Guidelines for the Good Evaluation of Research Practices," DIAG Technical Reports 2019-02, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    2. Cinzia Daraio & Léopold Simar & Paul W. Wilson, 2019. "Quality and its Impact on Efficiency," DIAG Technical Reports 2019-01, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    3. Cinzia Daraio & Alessio Vaccari, 2019. "Sorting out Guidelines for a Good Evaluation of Research Practices.Towards the Assessment of Researcher’s Virtues," DIAG Technical Reports 2019-10, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    4. Marco Angelini & Cinzia Daraio & Maurizio Lenzerini & Francesco Leotta & Giuseppe Santucci, 2019. "Performance Model’s development: A Novel Approach encompassing Ontology-Based Data Access and Visual Analytics," DIAG Technical Reports 2019-11, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    5. Moriah B. Bostian & Cinzia Daraio & Rolf Fare & Shawna Grosskopf & Maria Grazia Izzo & Luca Leuzzi & Giancarlo Ruocco & William L. Weber, 2018. "Inference for Nonparametric Productivity Networks: A Pseudo-likelihood Approach," DIAG Technical Reports 2018-06, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".

    More about this item

    Keywords

    Evaluation of Research ; Efficiency ; Effectiveness ; Impacts ; Modelling ; Implementation Problem ; Responsible Metrics ; Ethics of Research Evaluation;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aeg:report:2017-04. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Antonietta Angelica Zucconi). General contact details of provider: http://edirc.repec.org/data/dirosit.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.