IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v12y2018i4p1263-1281.html
   My bibliography  Save this article

A ‘power law’ based method to reduce size-related bias in indicators of knowledge performance: An application to university research assessment

Author

Listed:
  • Calabrese, Armando
  • Capece, Guendalina
  • Costa, Roberta
  • Di Pillo, Francesca
  • Giuffrida, Stefania

Abstract

The knowledge production provided by universities is essential to sustaining a country’s long-term economic growth and international competitiveness. Many nations are thus driving to create sustainable and effective funding environments. The evaluation of university knowledge, productivity and research quality becomes critical, with ever increasing share of public funding allocated on the basis of research assessment exercises. Nevertheless, the existing methods to assess the universities’ knowledge production are often affected by limits and biases, extensively discussed in the scientific literature.

Suggested Citation

  • Calabrese, Armando & Capece, Guendalina & Costa, Roberta & Di Pillo, Francesca & Giuffrida, Stefania, 2018. "A ‘power law’ based method to reduce size-related bias in indicators of knowledge performance: An application to university research assessment," Journal of Informetrics, Elsevier, vol. 12(4), pages 1263-1281.
    • Handle: RePEc:eee:infome:v:12:y:2018:i:4:p:1263-1281
    DOI: 10.1016/j.joi.2018.10.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157717303401
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2018.10.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Cruz-Castro, Laura & Sanz-Menéndez, Luis, 2016. "The effects of the economic crisis on public research: Spanish budgetary policies and research organizations," Technological Forecasting and Social Change, Elsevier, vol. 113(PB), pages 157-167.
    2. Avkiran, Necmi K., 2001. "Investigating technical and scale efficiencies of Australian Universities through data envelopment analysis," Socio-Economic Planning Sciences, Elsevier, vol. 35(1), pages 57-80, March.
    3. Biggiero, Lucio & Angelini, Pier Paolo, 2015. "Hunting scale-free properties in R&D collaboration networks: Self-organization, power-law and policy issues in the European aerospace research area," Technological Forecasting and Social Change, Elsevier, vol. 94(C), pages 21-43.
    4. David Audretsch & Marcel Hülsbeck & Erik Lehmann, 2012. "Regional competitiveness, university spillovers, and entrepreneurial activity," Small Business Economics, Springer, vol. 39(3), pages 587-601, October.
    5. Anthony F.J. van Raan, 2006. "Statistical properties of bibliometric indicators: Research group indicator distributions and correlations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(3), pages 408-430, February.
    6. B. B. Mandelbrot, 2001. "Scaling in financial prices: III. Cartoon Brownian motions in multifractal time," Quantitative Finance, Taylor & Francis Journals, vol. 1(4), pages 427-440.
    7. J Sylvan Katz, 2000. "Scale-independent indicators and research evaluation," Science and Public Policy, Oxford University Press, vol. 27(1), pages 23-36, February.
    8. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2010. "Should you believe in the Shanghai ranking?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 237-263, July.
    9. Daniel Torres-Salinas & Jose G. Moreno-Torres & Emilio Delgado-López-Cózar & Francisco Herrera, 2011. "A methodology for Institution-Field ranking based on a bidimensional analysis: the IFQ 2 A index," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 771-786, September.
    10. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2010. "Should you believe in the Shanghai ranking?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 237-263, July.
    11. Youngki Lee & Luis A. N. Amaral & David Canning & Martin Meyer & H. Eugene Stanley, 1998. "Universal features in the growth dynamics of complex organizations," Papers cond-mat/9804100, arXiv.org.
    12. L. A. N. Amaral & S. V. Buldyrev & S. Havlin & H. Leschhorn & P. Maass & M. A. Salinger & H. E. Stanley & M. H. R. Stanley, 1997. "Scaling behavior in economics: I. Empirical results for company growth," Papers cond-mat/9702082, arXiv.org.
    13. Carmen López-Illescas & Félix Moya-Anegón & Henk F. Moed, 2011. "A ranking of universities should account for differences in their disciplinary specialization," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(2), pages 563-574, August.
    14. Frank Havemann & Michael Heinz & Roland Wagner‐Döbler, 2005. "Firm‐like behavior of journals? Scaling properties of their output and impact growth dynamics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(1), pages 3-12, January.
    15. Katz, J. Sylvan, 2006. "Indicators for complex innovation systems," Research Policy, Elsevier, vol. 35(7), pages 893-909, September.
    16. repec:dau:papers:123456789/2947 is not listed on IDEAS
    17. Koen Jonkers & Thomas Zacharewicz, 2016. "Research Performance Based Funding Systems: a Comparative Assessment," JRC Research Reports JRC101043, Joint Research Centre.
    18. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    19. Geoffrey B. West & James H. Brown & Brian J. Enquist, 1999. "The Fourth Dimension of Life: Fractal Geometry and Allometric Scaling of Organisms," Working Papers 99-07-047, Santa Fe Institute.
    20. Etzkowitz, Henry & Leydesdorff, Loet, 2000. "The dynamics of innovation: from National Systems and "Mode 2" to a Triple Helix of university-industry-government relations," Research Policy, Elsevier, vol. 29(2), pages 109-123, February.
    21. L. Egghe, 2005. "The power of power laws and an interpretation of Lotkaian informetric systems as self‐similar fractals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(7), pages 669-675, May.
    22. Ralph Kenna & Bertrand Berche, 2011. "Normalization of peer-evaluation measures of group research quality across academic disciplines," Research Evaluation, Oxford University Press, vol. 20(2), pages 107-116, June.
    23. Geoffrey B. West & James H. Brown & Brian J. Enquist, 1997. "A General Model for the Origin of Allometric Scaling Laws in Biology," Working Papers 97-03-019, Santa Fe Institute.
    24. Vasiliki Plerou & Luís A. Nunes Amaral & Parameswaran Gopikrishnan & Martin Meyer & H. Eugene Stanley, 1999. "Similarities between the growth dynamics of university research and of competitive economic activities," Nature, Nature, vol. 400(6743), pages 433-437, July.
    25. J Sylvan Katz & Viv Cothey, 2006. "Web indicators for complex innovation systems," Research Evaluation, Oxford University Press, vol. 15(2), pages 85-95, August.
    26. Xavier Gabaix, 1999. "Zipf's Law for Cities: An Explanation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(3), pages 739-767.
    27. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: the effects of changing the rules of the game during the game," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(1), pages 229-238, July.
    28. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "Revisiting the scaling of citations for research assessment," Journal of Informetrics, Elsevier, vol. 6(4), pages 470-479.
    29. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2011. "The dangers of performance-based research funding in non-competitive higher education systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 641-654, June.
    30. Pietronero, L., 1987. "The fractal structure of the universe: Correlations of galaxies and clusters and the average mass density," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 144(2), pages 257-284.
    31. Ben R Martin, 2011. "The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster?," Research Evaluation, Oxford University Press, vol. 20(3), pages 247-254, September.
    32. R. Kenna & B. Berche, 2011. "Critical mass and the dependency of research quality on group size," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 527-540, February.
    33. Osmo Kivinen & Juha Hedman & Päivi Kaipainen, 2013. "Productivity analysis of research in Natural Sciences, Technology and Clinical Medicine: an input–output model applied in comparison of Top 300 ranked universities of 4 North European and 4 East Asian," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 683-699, February.
    34. Anthony F. J. van Raan, 2006. "Performance‐related differences of bibliometric statistical properties of research groups: Cumulative advantages and hierarchically layered networks," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(14), pages 1919-1935, December.
    35. Loet Leydesdorff & Wilfred Dolfsma & Gerben Van der Panne, 2010. "Measuring the Knowledge Base of an Economy in Terms of Triple-Helix Relations," Chapters, in: Riccardo Viale & Henry Etzkowitz (ed.), The Capitalization of Knowledge, chapter 11, Edward Elgar Publishing.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Camil Demetrescu & Andrea Ribichini & Marco Schaerf, 2020. "Are Italian research assessment exercises size-biased?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 533-549, October.
    2. Katerina Guba & Mikhail Sokolov & Angelika Tsivinskaya, 2020. "Fictitious Efficiency: What the Russian Survey of Performance of Higher Education Institutions Actually Assessed," Voprosy obrazovaniya / Educational Studies Moscow, National Research University Higher School of Economics, issue 1, pages 97-125.
    3. Губа К. С. & Соколов М. М. & Цивинская А. О., 2020. "Фиктивная Эффективность: Что На Самом Деле Оценивал Мониторинг Эффективности Образовательных Организаций," Вопросы образования // Educational Studies Moscow, National Research University Higher School of Economics, issue 1, pages 97-125.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Evaluating university research: Same performance indicator, different rankings," Journal of Informetrics, Elsevier, vol. 9(3), pages 514-525.
    2. Vieira, Elizabeth S. & Lepori, Benedetto, 2016. "The growth process of higher education institutions and public policies," Journal of Informetrics, Elsevier, vol. 10(1), pages 286-298.
    3. J. Sylvan Katz & Guillermo Armando Ronda-Pupo, 2019. "Cooperation, scale-invariance and complex innovation systems: a generalization," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 1045-1065, November.
    4. Amaral, L.A.N. & Gopikrishnan, P. & Plerou, V. & Stanley, H.E., 2001. "A model for the growth dynamics of economic organizations," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 299(1), pages 127-136.
    5. Stanley, H.E. & Gabaix, Xavier & Gopikrishnan, Parameswaran & Plerou, Vasiliki, 2007. "Economic fluctuations and statistical physics: Quantifying extremely rare and less rare events in finance," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 382(1), pages 286-301.
    6. Hiran H. Lathabai & Abhirup Nandy & Vivek Kumar Singh, 2021. "x-index: Identifying core competency and thematic research strengths of institutions using an NLP and network based ranking framework," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9557-9583, December.
    7. Guillermo Armando Ronda-Pupo, 2017. "The citation-based impact of complex innovation systems scales with the size of the system," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 141-151, July.
    8. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & Daniel Torres-Salinas & Francisco Herrera, 2012. "Ranking of research output of universities on the basis of the multidimensional prestige of influential fields: Spanish universities as a case of study," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 1081-1099, December.
    9. Stanley, H. Eugene & Plerou, Vasiliki & Gabaix, Xavier, 2008. "A statistical physics view of financial fluctuations: Evidence for scaling and universality," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 387(15), pages 3967-3981.
    10. Guillermo Armando Ronda-Pupo & J. Sylvan Katz, 2018. "The power law relationship between citation impact and multi-authorship patterns in articles in Information Science & Library Science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 919-932, March.
    11. Osmo Kivinen & Juha Hedman & Kalle Artukka, 2017. "Scientific publishing and global university rankings. How well are top publishing universities recognized?," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 679-695, July.
    12. Giovanni Abramo & Corrado Costa & Ciriaco Andrea D’Angelo, 2015. "A multivariate stochastic model to assess research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1755-1772, February.
    13. Vicente Safón, 2013. "What do global university rankings really measure? The search for the X factor and the X entity," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 223-244, November.
    14. Misako Takayasu & Hayafumi Watanabe & Hideki Takayasu, 2013. "Generalised central limit theorems for growth rate distribution of complex systems," Papers 1301.2728, arXiv.org, revised Jan 2014.
    15. Hongguang Dong & Menghui Li & Ru Liu & Chensheng Wu & Jinshan Wu, 2017. "Allometric scaling in scientific fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 583-594, July.
    16. Hernan Mondani & Petter Holme & Fredrik Liljeros, 2014. "Fat-Tailed Fluctuations in the Size of Organizations: The Role of Social Influence," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-9, July.
    17. Maxim Kotsemir & Alexander Abroskin & Dirk Meissner, 2013. "Innovation concepts and typology – an evolutionary discussion," HSE Working papers WP BRP 05/STI/2013, National Research University Higher School of Economics.
    18. El Gibari, Samira & Gómez, Trinidad & Ruiz, Francisco, 2018. "Evaluating university performance using reference point based composite indicators," Journal of Informetrics, Elsevier, vol. 12(4), pages 1235-1250.
    19. Franceschini, Fiorenzo & Maisano, Domenico, 2011. "Structured evaluation of the scientific output of academic research groups by recent h-based indicators," Journal of Informetrics, Elsevier, vol. 5(1), pages 64-74.
    20. Degl’Innocenti, Marta & Matousek, Roman & Tzeremes, Nickolaos G., 2019. "The interconnections of academic research and universities’ “third mission”: Evidence from the UK," Research Policy, Elsevier, vol. 48(9), pages 1-1.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:12:y:2018:i:4:p:1263-1281. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.