IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v122y2020i3d10.1007_s11192-020-03364-1.html
   My bibliography  Save this article

Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors

Author

Listed:
  • Jingda Ding

    (Shanghai University)

  • Chao Liu

    (Shanghai University)

  • Goodluck Asobenie Kandonga

    (Shanghai University)

Abstract

With the introduction of an increasing number of evaluation indexes, researchers have begun to pay attention to the limitations of such indexes in research evaluation, understanding which to avoid misusing and making evaluation more scientific and reasonable. Analysing the principles of the h-index, g-index, AR-index, p-index, integrated impact indicator (I3), and academic trace, this paper explores their limitations in measuring the research performance of authors from the perspectives of consistency, the degree of discrimination, and the statistical relationship between the values of indicators and the number of publications and citations. There are some interesting findings. These six indicators are highly consistent, and they are all more susceptible to the number of publications than to the frequency of citations. Among them, the h-index has the lowest degree of discrimination, followed by the g-index, I3, AR-index, p-index, and academic trace. The g-index ignores papers and citations other than the g-core. Moreover, compared to the h-index, the accumulation of citations makes it easier for the g-index to be equal to the number of papers published by an author, and once its value equals the number of papers, subsequent citations received by these papers will no longer contribute to the growth of the g-index unless the author publishes a new paper. Additionally, the AR-index ignores the h-tail papers and citations, which underestimates the impact of many researchers. Moreover, the p-index is insensitive to highly cited papers. Furthermore, the I3 is very vulnerable to the influence of the extremums in a data set. Finally, we propose considerations and suggestions for the research performance evaluation of authors.

Suggested Citation

  • Jingda Ding & Chao Liu & Goodluck Asobenie Kandonga, 2020. "Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1303-1322, March.
    • Handle: RePEc:spr:scient:v:122:y:2020:i:3:d:10.1007_s11192-020-03364-1
    DOI: 10.1007/s11192-020-03364-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03364-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03364-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Caroline S. Wagner & Loet Leydesdorff, 2012. "An Integrated Impact Indicator: A new definition of 'Impact' with policy relevance," Research Evaluation, Oxford University Press, vol. 21(3), pages 183-188, July.
    2. Tol, Richard S.J., 2008. "A rational, successive g-index applied to economics departments in Ireland," Journal of Informetrics, Elsevier, vol. 2(2), pages 149-155.
    3. J. E. Hirsch, 2010. "An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 741-754, December.
    4. Lutz Bornmann & Rüdiger Mutz & Hans‐Dieter Daniel, 2008. "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 830-837, March.
    5. Isola Ajiferuke & Dietmar Wolfram, 2010. "Citer analysis as a measure of research impact: library and information science as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 623-638, June.
    6. Marcin Kozak & Lutz Bornmann, 2012. "A New Family of Cumulative Indexes for Measuring Scientific Performance," PLOS ONE, Public Library of Science, vol. 7(10), pages 1-4, October.
    7. Fred Y. Ye & Loet Leydesdorff, 2014. "The “academic trace” of the performance matrix: A mathematical synthesis of the h-index and the integrated impact indicator (I3)," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 742-750, April.
    8. Philip Ball, 2005. "Index aims for fair ranking of scientists," Nature, Nature, vol. 436(7053), pages 900-900, August.
    9. Juan E. Iglesias & Carlos Pecharromán, 2007. "Scaling the h-index for different scientific ISI fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 73(3), pages 303-320, December.
    10. Pablo D. Batista & Mônica G. Campiteli & Osame Kinouchi, 2006. "Is it possible to compare researchers with different scientific interests?," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(1), pages 179-189, July.
    11. Michael Schreiber, 2009. "Fractionalized counting of publications for the g‐Index," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(10), pages 2145-2150, October.
    12. Gangan Prathap, 2010. "Is there a place for a mock h-index?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 153-165, July.
    13. Lutz Bornmann, 2013. "How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(3), pages 587-595, March.
    14. Loet Leydesdorff & Lutz Bornmann, 2011. "Integrated impact indicators compared with impact factors: An alternative research design with policy implications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(11), pages 2133-2146, November.
    15. Chun-Ting Zhang, 2013. "The h’-Index, Effectively Improving the h-Index Based on the Citation Distribution," PLOS ONE, Public Library of Science, vol. 8(4), pages 1-8, April.
    16. Isola Ajiferuke & Kun Lu & Dietmar Wolfram, 2010. "A comparison of citer and citation-based measure outcomes for multiple disciplines," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(10), pages 2086-2096, October.
    17. S. Alonso & F. J. Cabrerizo & E. Herrera-Viedma & F. Herrera, 2010. "hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 391-400, February.
    18. Giovanni Abramo & Ciriaco Andrea D'Angelo & Fulvio Viel, 2013. "Assessing the accuracy of the h- and g-indexes for measuring researchers' productivity," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(6), pages 1224-1234, June.
    19. J. E. Hirsch, 2019. "hα: An index to quantify an individual’s scientific leadership," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 673-686, February.
    20. Michael Schreiber, 2008. "An empirical investigation of the g‐index for 26 physicists in comparison with the h‐index, the A‐index, and the R‐index," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(9), pages 1513-1522, July.
    21. Bornmann, Lutz & Mutz, Rüdiger & Hug, Sven E. & Daniel, Hans-Dieter, 2011. "A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants," Journal of Informetrics, Elsevier, vol. 5(3), pages 346-359.
    22. Isola Ajiferuke & Kun Lu & Dietmar Wolfram, 2010. "A comparison of citer and citation‐based measure outcomes for multiple disciplines," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(10), pages 2086-2096, October.
    23. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    24. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    25. Michael Schreiber, 2010. "Revisiting the g-index: The average number of citations in the g-core," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(1), pages 169-174, January.
    26. Burrell, Quentin L., 2007. "On the h-index, the size of the Hirsch core and Jin's A-index," Journal of Informetrics, Elsevier, vol. 1(2), pages 170-177.
    27. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fulvio Viel, 2013. "The suitability of h and g indexes for measuring the research performance of institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 555-570, December.
    28. Gerhard J. Woeginger, 2009. "Generalizations of Egghe's g‐index," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(6), pages 1267-1273, June.
    29. Chun-Ting Zhang, 2009. "The e-Index, Complementing the h-Index for Excess Citations," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-4, May.
    30. Rodrigo Costas & María Bordons, 2008. "Is g-index better than h-index? An exploratory study at the individual level," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(2), pages 267-288, November.
    31. Liming Liang, 2006. "h-index sequence and h-index matrix: Constructions and applications," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 153-159, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ioan Ianoş & Alexandru-Ionuţ Petrişor, 2020. "An Overview of the Dynamics of Relative Research Performance in Central-Eastern Europe Using a Ranking-Based Analysis Derived from SCImago Data," Publications, MDPI, vol. 8(3), pages 1-25, July.
    2. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    3. Wei, Shelia X. & Tong, Tong & Rousseau, Ronald & Wang, Wanru & Ye, Fred Y., 2022. "Relations among the h-, g-, ψ-, and p-index and offset-ability," Journal of Informetrics, Elsevier, vol. 16(4).
    4. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    5. Pantea Kamrani & Isabelle Dorsch & Wolfgang G. Stock, 2021. "Do researchers know what the h-index is? And how do they estimate its importance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5489-5508, July.
    6. Zoltán Krajcsák, 2021. "Researcher Performance in Scopus Articles ( RPSA ) as a New Scientometric Model of Scientific Output: Tested in Business Area of V4 Countries," Publications, MDPI, vol. 9(4), pages 1-23, October.
    7. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    8. Alexander Serenko & Mauricio Marrone & John Dumay, 2022. "Scientometric portraits of recognized scientists: a structured literature review," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4827-4846, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lathabai, Hiran H., 2020. "ψ-index: A new overall productivity index for actors of science and technology," Journal of Informetrics, Elsevier, vol. 14(4).
    2. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    3. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    4. Sidiropoulos, A. & Gogoglou, A. & Katsaros, D. & Manolopoulos, Y., 2016. "Gazing at the skyline for star scientists," Journal of Informetrics, Elsevier, vol. 10(3), pages 789-813.
    5. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.
    6. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    7. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    8. Bornmann, Lutz & Mutz, Rüdiger & Hug, Sven E. & Daniel, Hans-Dieter, 2011. "A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants," Journal of Informetrics, Elsevier, vol. 5(3), pages 346-359.
    9. Ana Paula dos Santos Rubem & Ariane Lima Moura & João Carlos Correia Baptista Soares de Mello, 2015. "Comparative analysis of some individual bibliometric indices when applied to groups of researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 1019-1035, January.
    10. Yves Fassin, 2020. "The HF-rating as a universal complement to the h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 965-990, November.
    11. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    12. Yu Liu & Wei Zuo & Ying Gao & Yanhong Qiao, 2013. "Comprehensive geometrical interpretation of h-type indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 605-615, August.
    13. Miguel A. García-Pérez, 2009. "A multidimensional extension to Hirsch’s h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 779-785, December.
    14. Wei, Shelia X. & Tong, Tong & Rousseau, Ronald & Wang, Wanru & Ye, Fred Y., 2022. "Relations among the h-, g-, ψ-, and p-index and offset-ability," Journal of Informetrics, Elsevier, vol. 16(4).
    15. S. Alonso & F. J. Cabrerizo & E. Herrera-Viedma & F. Herrera, 2010. "hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 391-400, February.
    16. van Eck, Nees Jan & Waltman, Ludo, 2008. "Generalizing the h- and g-indices," Journal of Informetrics, Elsevier, vol. 2(4), pages 263-271.
    17. Brandão, Luana Carneiro & Soares de Mello, João Carlos Correia Baptista, 2019. "A multi-criteria approach to the h-index," European Journal of Operational Research, Elsevier, vol. 276(1), pages 357-363.
    18. van Eck, N.J.P. & Waltman, L., 2008. "Generalizing the h- and g-indices," ERIM Report Series Research in Management ERS-2008-049-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    19. Christopher McCarty & James W. Jawitz & Allison Hopkins & Alex Goldman, 2013. "Predicting author h-index using characteristics of the co-author network," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 467-483, August.
    20. Chen, Dar-zen & Huang, Mu-hsuan & Ye, Fred Y., 2013. "A probe into dynamic measures for h-core and h-tail," Journal of Informetrics, Elsevier, vol. 7(1), pages 129-137.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:122:y:2020:i:3:d:10.1007_s11192-020-03364-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.