IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v10y2016i4p1055-1078.html
   My bibliography  Save this article

A critical cluster analysis of 44 indicators of author-level performance

Author

Listed:
  • Wildgaard, Lorna

Abstract

This paper explores a 7-stage cluster methodology as a process to identify appropriate indicators for evaluation of individual researchers at a disciplinary and seniority level. Publication and citation data for 741 researchers from 4 disciplines was collected in Web of Science. Forty-four indicators of individual researcher performance were computed using the data. The clustering solution was supported by continued reference to the researcher’s curriculum vitae, an effect analysis and a risk analysis. Disciplinary appropriate indicators were identified and used to divide the researchers into four groups; low, middle, high and extremely high performers. Seniority-specific indicators were not identified. The practical importance of the recommended disciplinary appropriate indicators is concerning. Our study revealed several critical concerns that should be investigated in the application of statistics in research evaluation.

Suggested Citation

  • Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    • Handle: RePEc:eee:infome:v:10:y:2016:i:4:p:1055-1078
    DOI: 10.1016/j.joi.2016.09.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157716301171
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2016.09.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Schneider, Jesper W., 2013. "Caveats for using statistical significance tests in research assessments," Journal of Informetrics, Elsevier, vol. 7(1), pages 50-62.
    2. Rodrigo Costas & Thed N. van Leeuwen & María Bordons, 2010. "A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(8), pages 1564-1581, August.
    3. Jesper W. Schneider & Thed N. van Leeuwen, 2014. "Analysing robustness and uncertainty levels of bibliometric performance statistics supporting science policy. A case study evaluating Danish postdoctoral funding," Research Evaluation, Oxford University Press, vol. 23(4), pages 285-297.
    4. Wolfgang Glänzel & Henk F. Moed, 2013. "Opinion paper: thoughts and facts on bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 381-394, July.
    5. Lutz Bornmann & Rüdiger Mutz & Hans‐Dieter Daniel, 2008. "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 830-837, March.
    6. Jesper W. Schneider & Pia Borlund, 2007. "Matrix comparison, Part 1: Motivation and important issues for measuring the resemblance between proximity measures or ordination results," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(11), pages 1586-1595, September.
    7. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    8. Bruno S. Frey & Katja Rost, 2010. "Do rankings reflect research quality?," Journal of Applied Economics, Universidad del CEMA, vol. 13, pages 1-38, May.
    9. Aksnes, Dag W. & Rip, Arie, 2009. "Researchers' perceptions of citations," Research Policy, Elsevier, vol. 38(6), pages 895-905, July.
    10. Larry V. Hedges, 1981. "Distribution Theory for Glass's Estimator of Effect size and Related Estimators," Journal of Educational and Behavioral Statistics, , vol. 6(2), pages 107-128, June.
    11. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    12. Johann Bacher, 2000. "A Probabilistic Clustering Model for Variables of Mixed Type," Quality & Quantity: International Journal of Methodology, Springer, vol. 34(3), pages 223-235, August.
    13. Crespo, Juan A. & Li, Yunrong & Ruiz-Castillo, Javier, 2012. "Differences in citation impact across scientific fields," UC3M Working papers. Economics we1206, Universidad Carlos III de Madrid. Departamento de Economía.
    14. Seongkyoon Jeong & Jae Young Choi, 2012. "The taxonomy of research collaboration in science and technology: evidence from mechanical research through probabilistic clustering analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 719-735, June.
    15. Anne-Wil Harzing & Satu Alakangas, 2016. "Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 787-804, February.
    16. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    17. Rodrigo Costas & Thed N. van Leeuwen & María Bordons, 2010. "A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(8), pages 1564-1581, August.
    18. Jesper W. Schneider & Pia Borlund, 2007. "Matrix comparison, Part 2: Measuring the resemblance between proximity measures or ordination results by use of the mantel and procrustes statistics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(11), pages 1596-1609, September.
    19. Díaz-Faes, Adrián A. & Costas, Rodrigo & Galindo, M. Purificación & Bordons, María, 2015. "Unravelling the performance of individual scholars: Use of Canonical Biplot analysis to explore the performance of scientists by academic rank and scientific field," Journal of Informetrics, Elsevier, vol. 9(4), pages 722-733.
    20. Chun-Ting Zhang, 2009. "The e-Index, Complementing the h-Index for Excess Citations," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-4, May.
    21. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    22. Jean-François Bach & J.F. Bach & Denis Jérome & Brigitte d'Artemare, 2011. "On The Proper Use Of Bibliometrics To Evaluate Individual Researchers," Working Papers hal-00604136, HAL.
    23. Lorna Wildgaard, 2015. "A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 873-906, September.
    24. Leo Egghe & Ronald Rousseau, 2000. "Aging, obsolescence, impact, growth, and utilization: Definitions and relations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(11), pages 1004-1017.
    25. S. Alonso & F. J. Cabrerizo & E. Herrera-Viedma & F. Herrera, 2010. "hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 391-400, February.
    26. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2013. "Individual research performance: A proposal for comparing apples to oranges," Journal of Informetrics, Elsevier, vol. 7(2), pages 528-539.
    27. Waltman, Ludo & van Eck, Nees Jan & Noyons, Ed C.M., 2010. "A unified approach to mapping and clustering of bibliometric networks," Journal of Informetrics, Elsevier, vol. 4(4), pages 629-635.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tian Chen & Shiyao Li & Chun-Ming Yang & Wenting Deng, 2022. "Developing an Enterprise Diagnostic Index System Based on Interval-Valued Hesitant Fuzzy Clustering," Mathematics, MDPI, vol. 10(14), pages 1-22, July.
    2. Zharova, Alona & Härdle, Wolfgang Karl & Lessmann, Stefan, 2023. "Data-driven support for policy and decision-making in university research management: A case study from Germany," European Journal of Operational Research, Elsevier, vol. 308(1), pages 353-368.
    3. Gauffriau, Marianne, 2017. "A categorization of arguments for counting methods for publication and citation indicators," Journal of Informetrics, Elsevier, vol. 11(3), pages 672-684.
    4. Bart Thijs & Koenraad Debackere & Wolfgang Glänzel, 2017. "Improved author profiling through the use of citation classes," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 829-839, May.
    5. Qingsong Wang & Hongrui Tang & Xueliang Yuan & Mansen Wang & Hongkun Xiao & Zhi Ma, 2018. "An Early Warning System for Oil Security in China," Sustainability, MDPI, vol. 10(1), pages 1-17, January.
    6. Soleidy Rivero Amador & Maidelyn Díaz Pérez & María José López-Huertas & Reinaldo Javier Rodríguez Font, 2018. "Indicator system for managing science, technology and innovation in universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1575-1587, June.
    7. Fabio Zagonari, 2019. "Scientific Production and Productivity for Characterizing an Author’s Publication History: Simple and Nested Gini’s and Hirsch’s Indexes Combined," Publications, MDPI, vol. 7(2), pages 1-30, May.
    8. David A. Pendlebury, 2019. "Charting a path between the simple and the false and the complex and unusable: Review of Henk F. Moed, Applied Evaluative Informetrics [in the series Qualitative and Quantitative Analysis of Scientifi," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 549-560, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    2. Andersen, Jens Peter & Nielsen, Mathias Wullum, 2018. "Google Scholar and Web of Science: Examining gender differences in citation coverage across five scientific disciplines," Journal of Informetrics, Elsevier, vol. 12(3), pages 950-959.
    3. Andersen, Jens Peter, 2017. "An empirical and theoretical critique of the Euclidean index," Journal of Informetrics, Elsevier, vol. 11(2), pages 455-465.
    4. Lorna Wildgaard, 2015. "A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 873-906, September.
    5. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    6. Zhenbin Yan & Qiang Wu & Xingchen Li, 2016. "Do Hirsch-type indices behave the same in assessing single publications? An empirical study of 29 bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1815-1833, December.
    7. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    8. Yu Liu & Wei Zuo & Ying Gao & Yanhong Qiao, 2013. "Comprehensive geometrical interpretation of h-type indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 605-615, August.
    9. Madiha Ameer & Muhammad Tanvir Afzal, 2019. "Evaluation of h-index and its qualitative and quantitative variants in Neuroscience," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 653-673, November.
    10. Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2020. "On bibliometrics in academic promotions: a case study in computer science and engineering in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2207-2228, September.
    11. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    12. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    13. Alona Zharova & Wolfgang K. Härdle & Stefan Lessmann, 2017. "Is Scientific Performance a Function of Funds?," SFB 649 Discussion Papers SFB649DP2017-028, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    14. Brandão, Luana Carneiro & Soares de Mello, João Carlos Correia Baptista, 2019. "A multi-criteria approach to the h-index," European Journal of Operational Research, Elsevier, vol. 276(1), pages 357-363.
    15. Copiello, Sergio, 2019. "Peer and neighborhood effects: Citation analysis using a spatial autoregressive model and pseudo-spatial data," Journal of Informetrics, Elsevier, vol. 13(1), pages 238-254.
    16. Díaz-Faes, Adrián A. & Costas, Rodrigo & Galindo, M. Purificación & Bordons, María, 2015. "Unravelling the performance of individual scholars: Use of Canonical Biplot analysis to explore the performance of scientists by academic rank and scientific field," Journal of Informetrics, Elsevier, vol. 9(4), pages 722-733.
    17. Sidiropoulos, A. & Gogoglou, A. & Katsaros, D. & Manolopoulos, Y., 2016. "Gazing at the skyline for star scientists," Journal of Informetrics, Elsevier, vol. 10(3), pages 789-813.
    18. Qurat-ul Ain & Hira Riaz & Muhammad Tanvir Afzal, 2019. "Evaluation of h-index and its citation intensity based variants in the field of mathematics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 187-211, April.
    19. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.
    20. Ioana Alexandra HORODNIC, 2014. "Academic Performance: Measurement Methods Used In Socio - Economic Sciences," THE YEARBOOK OF THE "GH. ZANE" INSTITUTE OF ECONOMIC RESEARCHES, Gheorghe Zane Institute for Economic and Social Research ( from THE ROMANIAN ACADEMY, JASSY BRANCH), vol. 23(1), pages 5-17.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:10:y:2016:i:4:p:1055-1078. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.