IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v104y2015i3d10.1007_s11192-015-1608-4.html
   My bibliography  Save this article

A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar

Author

Listed:
  • Lorna Wildgaard

    (Copenhagen University)

Abstract

Author-level bibliometric indicators are becoming a standard tool in research assessment. It is important to investigate what these indicators actually measure to assess their appropriateness in scholar ranking and benchmarking average individual levels of performance. 17 author-level indicators were calculated for 512 researchers in Astronomy, Environmental Science, Philosophy and Public Health. Indicator scores and scholar rankings calculated in Web of Science (WoS) and Google Scholar (GS) were analyzed. The indexing policies of WoS and GS were found to have a direct effect on the amount of available bibliometric data, thus indicator scores and rankings in WoS and GS were different, correlations between 0.24 and 0.99. High correlation could be caused by scholars in bottom rank positions with a low number of publications and citations in both databases. The hg indicator produced scholar rankings with the highest level of agreement between WoS and GS and rankings with the least amount of variance. Expected average performance benchmarks were influenced by how the mean indicator value was calculated. Empirical validation of the aggregate mean h-index values compared to previous studies resulted in a very poor fit of predicted average scores. Rankings based on author-level indicators are influenced by (1) the coverage of papers and citations in the database, (2) how the indicators are calculated and, (3) the assessed discipline and seniority. Indicator rankings display the visibility of the scholar in the database not their impact in the academic community compared to their peers. Extreme caution is advised when choosing indicators and benchmarks in scholar rankings.

Suggested Citation

  • Lorna Wildgaard, 2015. "A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 873-906, September.
    • Handle: RePEc:spr:scient:v:104:y:2015:i:3:d:10.1007_s11192-015-1608-4
    DOI: 10.1007/s11192-015-1608-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-015-1608-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-015-1608-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Thed van Leeuwen, 2008. "Testing the validity of the Hirsch-index for research assessment purposes," Research Evaluation, Oxford University Press, vol. 17(2), pages 157-160, June.
    2. repec:ebl:ecbull:v:3:y:2008:i:78:p:1-9 is not listed on IDEAS
    3. Farhadi, Hadi & Salehi, Hadi & Md Yunus, Melor & Arezoo, Aghaei Chadegani & Farhadi, Maryam & Fooladi, Masood & Ale Ebrahim, Nader, 2012. "Does it Matter Which Citation Tool is Used to Compare the H-Index of a Group of Highly Cited Researchers?," MPRA Paper 47414, University Library of Munich, Germany, revised Dec 2012.
    4. John Panaretos & Chrisovaladis Malesios, 2009. "Assessing scientific research performance and impact with single indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 635-670, December.
    5. Alastair G. Smith, 2008. "Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 309-316, February.
    6. Francesca De Battisti & Silvia Salini, 2013. "Robust analysis of bibliometric data," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 22(2), pages 269-283, June.
    7. Massimo Franceschet, 2010. "A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(1), pages 243-258, April.
    8. Judit Bar-Ilan, 2008. "Which h-index? — A comparison of WoS, Scopus and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 257-271, February.
    9. Anne-Wil Harzing, 2014. "A longitudinal study of Google Scholar coverage between 2012 and 2013," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 565-575, January.
    10. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    11. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    12. S. Alonso & F. J. Cabrerizo & E. Herrera-Viedma & F. Herrera, 2010. "hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 391-400, February.
    13. Richard S. J. Tol, 2009. "The h-index and its alternatives: An application to the 100 most prolific economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(2), pages 317-324, August.
    14. Schneider, Jesper W., 2013. "Caveats for using statistical significance tests in research assessments," Journal of Informetrics, Elsevier, vol. 7(1), pages 50-62.
    15. Fiorenzo Franceschini & Domenico Maisano, 2011. "Criticism on the hg-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 339-346, February.
    16. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    17. Schreiber, Michael, 2013. "How to derive an advantage from the arbitrariness of the g-index," Journal of Informetrics, Elsevier, vol. 7(2), pages 555-561.
    18. Schreiber, Michael, 2013. "A case study of the arbitrariness of the h-index and the highly-cited-publications indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 379-387.
    19. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    20. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    21. Jean-Michel Courtault & Naïla Hayek, 2008. "On the Robustness of the h-index: a mathematical approach," Economics Bulletin, AccessEcon, vol. 3(78), pages 1-9.
    22. Francesca DE BATTISTI & Silvia SALINI, 2011. "Robust analysis of bibliometric data," Departmental Working Papers 2011-36, Department of Economics, Management and Quantitative Methods at Università degli Studi di Milano.
    23. Jesper W. Schneider, 2015. "Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 411-432, January.
    24. Ludo Waltman & Nees Jan van Eck, 2012. "The inconsistency of the h-index," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(2), pages 406-415, February.
    25. Chun-Ting Zhang, 2009. "The e-Index, Complementing the h-Index for Excess Citations," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-4, May.
    26. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile-based bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    27. Costas, Rodrigo & Bordons, María, 2007. "The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level," Journal of Informetrics, Elsevier, vol. 1(3), pages 193-203.
    28. Schreiber, M. & Malesios, C.C. & Psarakis, S., 2012. "Exploratory factor analysis for the Hirsch index, 17 h-type variants, and some traditional bibliometric indicators," Journal of Informetrics, Elsevier, vol. 6(3), pages 347-358.
    29. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Isabelle Dorsch, 2017. "Relative visibility of authors’ publications in different information services," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 917-925, August.
    2. Antonia Gogoglou & Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2017. "The fractal dimension of a citation curve: quantifying an individual’s scientific output using the geometry of the entire curve," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1751-1774, June.
    3. Andersen, Jens Peter & Nielsen, Mathias Wullum, 2018. "Google Scholar and Web of Science: Examining gender differences in citation coverage across five scientific disciplines," Journal of Informetrics, Elsevier, vol. 12(3), pages 950-959.
    4. Moustafa, Khaled, 2018. "Aberration of the citation," arabixiv.org gn8zb, Center for Open Science.
    5. Marek Kwiek & Wojciech Roszka, 2022. "Academic vs. biological age in research on academic careers: a large-scale study with implications for scientifically developing systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3543-3575, June.
    6. Anne-Wil Harzing & Satu Alakangas, 2017. "Microsoft Academic: is the phoenix getting wings?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 371-383, January.
    7. Nataliya N. Matveeva & Oleg V. Poldin, 2017. "How Network Characteristics of Researchers Relate to Their Citation Indicators – a Co-Authorship Network Analysis Based on Google Scholar," HSE Working papers WP BRP 44/EDU/2017, National Research University Higher School of Economics.
    8. Fabio Zagonari, 2019. "Scientific Production and Productivity for Characterizing an Author’s Publication History: Simple and Nested Gini’s and Hirsch’s Indexes Combined," Publications, MDPI, vol. 7(2), pages 1-30, May.
    9. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    10. Maor Weinberger & Maayan Zhitomirsky-Geffet, 2021. "Diversity of success: measuring the scholarly performance diversity of tenured professors in the Israeli academia," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 2931-2970, April.
    11. Franceschini, Fiorenzo & Maisano, Domenico & Mastrogiacomo, Luca, 2016. "Empirical analysis and classification of database errors in Scopus and Web of Science," Journal of Informetrics, Elsevier, vol. 10(4), pages 933-953.
    12. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    13. Matveeva, Nataliya & Poldin, Oleg, 2016. "Citation of scholars in co-authorship network: Analysis of Google Scholar data," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 44, pages 100-118.
    14. Anne-Wil Harzing & Satu Alakangas, 2016. "Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 787-804, February.
    15. Halevi, Gali & Moed, Henk & Bar-Ilan, Judit, 2017. "Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the Literature," Journal of Informetrics, Elsevier, vol. 11(3), pages 823-834.
    16. Nane, Gabriela F. & Larivière, Vincent & Costas, Rodrigo, 2017. "Predicting the age of researchers using bibliometric data," Journal of Informetrics, Elsevier, vol. 11(3), pages 713-729.
    17. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    18. M. Ryan Haley & M. Kevin McGee, 2023. "A flexible functional method for jointly valuing journal visibility and author citation count," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3337-3346, June.
    19. David A. Pendlebury, 2019. "Charting a path between the simple and the false and the complex and unusable: Review of Henk F. Moed, Applied Evaluative Informetrics [in the series Qualitative and Quantitative Analysis of Scientifi," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 549-560, April.
    20. Wu, Dengsheng & Li, Jing & Lu, Xiaoli & Li, Jianping, 2018. "Journal editorship index for assessing the scholarly impact of academic institutions: An empirical analysis in the field of economics," Journal of Informetrics, Elsevier, vol. 12(2), pages 448-460.
    21. Anne-Wil Harzing, 2016. "Microsoft Academic (Search): a Phoenix arisen from the ashes?," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1637-1647, September.
    22. Peyman Akhavan & Nader Ale Ebrahim & Mahdieh A. Fetrati & Amir Pezeshkan, 2016. "Major trends in knowledge management research: a bibliometric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1249-1264, June.
    23. Moed, Henk F. & Bar-Ilan, Judit & Halevi, Gali, 2016. "A new methodology for comparing Google Scholar and Scopus," Journal of Informetrics, Elsevier, vol. 10(2), pages 533-551.
    24. Haley, M. Ryan & McGee, M. Kevin, 2020. "Jointly valuing journal visibility and author citation count: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 14(1).
    25. Belussi, Fiorenza & Orsi, Luigi & Savarese, Maria, 2019. "Mapping Business Model Research: A Document Bibliometric Analysis," Scandinavian Journal of Management, Elsevier, vol. 35(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    2. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    3. Muhammad Usman & Ghulam Mustafa & Muhammad Tanvir Afzal, 2021. "Ranking of author assessment parameters using Logistic Regression," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 335-353, January.
    4. Brandão, Luana Carneiro & Soares de Mello, João Carlos Correia Baptista, 2019. "A multi-criteria approach to the h-index," European Journal of Operational Research, Elsevier, vol. 276(1), pages 357-363.
    5. Sidiropoulos, A. & Gogoglou, A. & Katsaros, D. & Manolopoulos, Y., 2016. "Gazing at the skyline for star scientists," Journal of Informetrics, Elsevier, vol. 10(3), pages 789-813.
    6. Muhammad Raheel & Samreen Ayaz & Muhammad Tanvir Afzal, 2018. "Evaluation of h-index, its variants and extensions based on publication age & citation intensity in civil engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1107-1127, March.
    7. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.
    8. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    9. Madiha Ameer & Muhammad Tanvir Afzal, 2019. "Evaluation of h-index and its qualitative and quantitative variants in Neuroscience," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 653-673, November.
    10. Maor Weinberger & Maayan Zhitomirsky-Geffet, 2021. "Diversity of success: measuring the scholarly performance diversity of tenured professors in the Israeli academia," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 2931-2970, April.
    11. Bornmann, Lutz & Mutz, Rüdiger & Hug, Sven E. & Daniel, Hans-Dieter, 2011. "A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants," Journal of Informetrics, Elsevier, vol. 5(3), pages 346-359.
    12. Ana Paula dos Santos Rubem & Ariane Lima Moura & João Carlos Correia Baptista Soares de Mello, 2015. "Comparative analysis of some individual bibliometric indices when applied to groups of researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 1019-1035, January.
    13. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    14. Zhenbin Yan & Qiang Wu & Xingchen Li, 2016. "Do Hirsch-type indices behave the same in assessing single publications? An empirical study of 29 bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1815-1833, December.
    15. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    16. Yu Liu & Wei Zuo & Ying Gao & Yanhong Qiao, 2013. "Comprehensive geometrical interpretation of h-type indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 605-615, August.
    17. Roberto Todeschini, 2011. "The j-index: a new bibliometric index and multivariate comparisons between other common indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 621-639, June.
    18. Fiorenzo Franceschini & Domenico Maisano, 2011. "Bibliometric positioning of scientific manufacturing journals: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 463-485, February.
    19. Giovanni Anania & Annarosa Caruso, 2013. "Two simple new bibliometric indexes to better evaluate research in disciplines where publications typically receive less citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 617-631, August.
    20. Franceschini, Fiorenzo & Maisano, Domenico, 2010. "The citation triad: An overview of a scientist's publication output based on Ferrers diagrams," Journal of Informetrics, Elsevier, vol. 4(4), pages 503-511.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:104:y:2015:i:3:d:10.1007_s11192-015-1608-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.