IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v130y2025i3d10.1007_s11192-025-05253-x.html
   My bibliography  Save this article

Ranking the rankers. An analysis of science-wide author databases of standardised citation indicators

Author

Listed:
  • Marta Kuc-Czarnecka

    (Gdańsk University of Technology)

  • Andrea Saltelli

    (Pompeu Fabra University
    University of Bergen)

Abstract

In this paper, we investigate the properties of the science-wide author databases’ standardised citation indicators produced since 2015 by a team of researchers led by John P. A. Ioannidis. Based on data from Scopus, the researchers published close to 200,000 of the most-cited authors across all scientific fields and ranked them using a composite indicator that includes six citation metrics (total citations; Hirsch h-index; coauthorship-adjusted Schreiber hm-index; the number of citations to papers as a single author; the number of citations to papers as a single or first author; and the number of citations to papers as a single, first, or last author). We look at the quality of these new rankings to suggest improvements. Specifically, we try to correct for correlation among variables and the deviation between the purported weights of the variables, as declared by the developers, and the effective weights based on the theory of global sensitivity analysis. We aim to ascertain if our modified measure is an improvement over the existing one by addressing the issue of redundant data due to correlated variables. Lacking a ground truth based on a ‘true’ ranking possibly supported by independent data, we content ourselves for the purpose of the present analysis with benchmarking against a best practice. Given that most practitioners have reservations against linear aggregation approaches (Munda, Social multi-criteria evaluation for a sustainable economy, Springer, Berlin, 2008; Balinski & Laraki, Majority judgment: Measuring, ranking, and electing, Mass, 2011; Arrow, Social choice and individual values, Martino Fine Books, Eastford, 2012 [1951]), while those based on the method of Condorcet are considered by many as a good practice (Munda, Social multi-criteria evaluation for a sustainable economy, Springer, Berlin, 2008; Roy, Multicriteria methodology for decision aiding, Springer, Cham, 1996), we entrust of defence of our modified ranking on a pairwise comparison between linear aggregation and Condorcet.

Suggested Citation

  • Marta Kuc-Czarnecka & Andrea Saltelli, 2025. "Ranking the rankers. An analysis of science-wide author databases of standardised citation indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(3), pages 1497-1517, March.
    • Handle: RePEc:spr:scient:v:130:y:2025:i:3:d:10.1007_s11192-025-05253-x
    DOI: 10.1007/s11192-025-05253-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-025-05253-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-025-05253-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paolo Paruolo & Michaela Saisana & Andrea Saltelli, 2013. "Ratings and rankings: voodoo or science?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 609-634, June.
    2. Gangan Prathap, 2019. "Letter to the editor: Revisiting the h-index and the p-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(3), pages 1829-1833, December.
    3. Mario Pagliaro, 2021. "Purposeful Evaluation of Scholarship in the Open Science Era," Challenges, MDPI, vol. 12(1), pages 1-11, February.
    4. Daniel Torres-Salinas & Wenceslao Arroyo-Machado & Nicolas Robinson-Garcia, 2023. "Bibliometric denialism," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5357-5359, September.
    5. Fabrice Valognes & William V. Gehrlein, 2001. "Condorcet efficiency: A preference for indifference," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 18(1), pages 193-205.
    6. Kai Petersen & Nauman Bin Ali, 2021. "An analysis of top author citations in software engineering and a comparison with other fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9147-9183, November.
    7. Samreen Ayaz & Nayyer Masood, 2020. "Comparison of researchers’ impact indices," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-15, May.
    8. Saltelli A. & Tarantola S., 2002. "On the Relative Importance of Input Factors in Mathematical Models: Safety Assessment for Nuclear Waste Disposal," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 702-709, September.
    9. Sachin S. Gunthe & Ravindra Gettu, 2022. "A new index for assessing faculty research performance in higher educational institutions of emerging economies such as India," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4959-4976, August.
    10. Michel Balinski & Rida Laraki, 2011. "Majority Judgment: Measuring, Ranking, and Electing," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262015137, December.
    11. Schreiber, Michael, 2008. "A modification of the h-index: The hm-index accounts for multi-authored manuscripts," Journal of Informetrics, Elsevier, vol. 2(3), pages 211-216.
    12. William Gehrlein & Fabrice Valognes, 2001. "Condorcet efficiency: A preference for indifference," Post-Print halshs-02173169, HAL.
    13. Giuseppe Munda, 2008. "Social Multi-Criteria Evaluation for a Sustainable Economy," Springer Books, Springer, number 978-3-540-73703-2, December.
    14. repec:plo:pbio00:3000384 is not listed on IDEAS
    15. Vladlen Koltun & David Hafner, 2021. "The h-index is no longer an effective correlate of scientific reputation," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-16, June.
    16. Živan Živković & Marija Panić, 2020. "Development of science and education in the Western Balkan countries: competitiveness with the EU," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2319-2339, September.
    17. repec:plo:pbio00:1002501 is not listed on IDEAS
    18. Park, Hyoungjoo & Wolfram, Dietmar, 2019. "Research software citation in the Data Citation Index: Current practices and implications for research software sharing and reuse," Journal of Informetrics, Elsevier, vol. 13(2), pages 574-582.
    19. John P A Ioannidis & Kevin W Boyack & Jeroen Baas, 2020. "Updated science-wide author databases of standardized citation indicators," PLOS Biology, Public Library of Science, vol. 18(10), pages 1-3, October.
    20. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    21. Marek Kosmulski, 2020. "Nobel laureates are not hot," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 487-495, April.
    22. Wil M. P. Aalst & Oliver Hinz & Christof Weinhardt, 2023. "Ranking the Ranker: How to Evaluate Institutions, Researchers, Journals, and Conferences?," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(6), pages 615-621, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yang, Alex Jie & Wu, Linwei & Zhang, Qi & Wang, Hao & Deng, Sanhong, 2023. "The k-step h-index in citation networks at the paper, author, and institution levels," Journal of Informetrics, Elsevier, vol. 17(4).
    2. Nick Haslam & Naomi Baes, 2023. "Scientific eminence and scientific hierarchy: bibliometric prediction of fellowship in the Australian Academy of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(12), pages 6659-6674, December.
    3. Lauranne Chaignon & Domingo Docampo & Daniel Egret, 2023. "In search of a scientific elite: highly cited researchers (HCR) in France," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5801-5827, October.
    4. Mostapha Diss & Eric Kamwa & Issofa Moyouwou & Hatem Smaoui, 2021. "Condorcet Efficiency of General Weighted Scoring Rules Under IAC: Indifference and Abstention," Studies in Choice and Welfare, in: Mostapha Diss & Vincent Merlin (ed.), Evaluating Voting Systems with Probability Models, pages 55-73, Springer.
    5. Wil M. P. Aalst & Oliver Hinz & Christof Weinhardt, 2023. "Ranking the Ranker: How to Evaluate Institutions, Researchers, Journals, and Conferences?," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(6), pages 615-621, December.
    6. Becker William & Paruolo Paolo & Saltelli Andrea, 2021. "Variable Selection in Regression Models Using Global Sensitivity Analysis," Journal of Time Series Econometrics, De Gruyter, vol. 13(2), pages 187-233, July.
    7. Mostapha Diss & Eric Kamwa & Issofa Moyouwou & Hatem Smaoui, 2019. "Condorcet efficiency of general weighted scoring rules under IAC: indifference and abstention," Working Papers hal-02196387, HAL.
    8. Merlin, Vincent & Valognes, Fabrice, 2004. "The impact of indifferent voters on the likelihood of some voting paradoxes," Mathematical Social Sciences, Elsevier, vol. 48(3), pages 343-361, November.
    9. Ruiz, Francisco & El Gibari, Samira & Cabello, José M. & Gómez, Trinidad, 2020. "MRP-WSCI: Multiple reference point based weak and strong composite indicators," Omega, Elsevier, vol. 95(C).
    10. Enrique Orduña-Malea & Núria Bautista-Puig, 2024. "Research assessment under debate: disentangling the interest around the DORA declaration on Twitter," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(1), pages 537-559, January.
    11. John P A Ioannidis & Kevin Boyack & Paul F Wouters, 2016. "Citation Metrics: A Primer on How (Not) to Normalize," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-7, September.
    12. Mathieu Martin, 2002. "On the emptiness of the stability set of order d," Theory and Decision, Springer, vol. 52(4), pages 313-326, June.
    13. Torres-Salinas, Daniel & Orduña-Malea, Enrique & Delgado-Vázquez, Ángel & Gorraiz, Juan & Arroyo-Machado, Wenceslao, 2024. "Foundations of Narrative Bibliometrics," Journal of Informetrics, Elsevier, vol. 18(3).
    14. Kai Petersen & Nauman Bin Ali, 2021. "An analysis of top author citations in software engineering and a comparison with other fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9147-9183, November.
    15. Martin Schlossarek & Miroslav Syrovátka & Ondřej Vencálek, 2019. "The Importance of Variables in Composite Indices: A Contribution to the Methodology and Application to Development Indices," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 145(3), pages 1125-1160, October.
    16. Mostapha Diss & Vincent Merlin, 2010. "On the stability of a triplet of scoring rules," Theory and Decision, Springer, vol. 69(2), pages 289-316, August.
    17. Ali Nasiri Khiavi & Mehdi Vafakhah & Seyed Hamidreza Sadeghi, 2022. "Comparative prioritization of sub-watersheds based on Flood Generation potential using physical, hydrological and co-managerial approaches," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(6), pages 1897-1917, April.
    18. William V. Gehrlein & Dominique Lepelley, 2015. "The Condorcet Efficiency Advantage that Voter Indifference Gives to Approval Voting Over Some Other Voting Rules," Group Decision and Negotiation, Springer, vol. 24(2), pages 243-269, March.
    19. Tanmoy Konar, 2021. "Author-Suggested, Weighted Citation Index: A Novel Approach for Determining the Contribution of Individual Researchers," Publications, MDPI, vol. 9(3), pages 1-8, July.
    20. Erik Friese & William V. Gehrlein & Dominique Lepelley & Achill Schürmann, 2017. "The impact of dependence among voters’ preferences with partial indifference," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(6), pages 2793-2812, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:130:y:2025:i:3:d:10.1007_s11192-025-05253-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.