IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v130y2025i3d10.1007_s11192-025-05253-x.html
   My bibliography  Save this article

Ranking the rankers. An analysis of science-wide author databases of standardised citation indicators

Author

Listed:
  • Marta Kuc-Czarnecka

    (Gdańsk University of Technology)

  • Andrea Saltelli

    (Pompeu Fabra University
    University of Bergen)

Abstract

In this paper, we investigate the properties of the science-wide author databases’ standardised citation indicators produced since 2015 by a team of researchers led by John P. A. Ioannidis. Based on data from Scopus, the researchers published close to 200,000 of the most-cited authors across all scientific fields and ranked them using a composite indicator that includes six citation metrics (total citations; Hirsch h-index; coauthorship-adjusted Schreiber hm-index; the number of citations to papers as a single author; the number of citations to papers as a single or first author; and the number of citations to papers as a single, first, or last author). We look at the quality of these new rankings to suggest improvements. Specifically, we try to correct for correlation among variables and the deviation between the purported weights of the variables, as declared by the developers, and the effective weights based on the theory of global sensitivity analysis. We aim to ascertain if our modified measure is an improvement over the existing one by addressing the issue of redundant data due to correlated variables. Lacking a ground truth based on a ‘true’ ranking possibly supported by independent data, we content ourselves for the purpose of the present analysis with benchmarking against a best practice. Given that most practitioners have reservations against linear aggregation approaches (Munda, Social multi-criteria evaluation for a sustainable economy, Springer, Berlin, 2008; Balinski & Laraki, Majority judgment: Measuring, ranking, and electing, Mass, 2011; Arrow, Social choice and individual values, Martino Fine Books, Eastford, 2012 [1951]), while those based on the method of Condorcet are considered by many as a good practice (Munda, Social multi-criteria evaluation for a sustainable economy, Springer, Berlin, 2008; Roy, Multicriteria methodology for decision aiding, Springer, Cham, 1996), we entrust of defence of our modified ranking on a pairwise comparison between linear aggregation and Condorcet.

Suggested Citation

  • Marta Kuc-Czarnecka & Andrea Saltelli, 2025. "Ranking the rankers. An analysis of science-wide author databases of standardised citation indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(3), pages 1497-1517, March.
    • Handle: RePEc:spr:scient:v:130:y:2025:i:3:d:10.1007_s11192-025-05253-x
    DOI: 10.1007/s11192-025-05253-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-025-05253-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-025-05253-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Paolo Paruolo & Michaela Saisana & Andrea Saltelli, 2013. "Ratings and rankings: voodoo or science?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 609-634, June.
    2. repec:plo:pbio00:1002501 is not listed on IDEAS
    3. Park, Hyoungjoo & Wolfram, Dietmar, 2019. "Research software citation in the Data Citation Index: Current practices and implications for research software sharing and reuse," Journal of Informetrics, Elsevier, vol. 13(2), pages 574-582.
    4. Gangan Prathap, 2019. "Letter to the editor: Revisiting the h-index and the p-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(3), pages 1829-1833, December.
    5. John P A Ioannidis & Kevin W Boyack & Jeroen Baas, 2020. "Updated science-wide author databases of standardized citation indicators," PLOS Biology, Public Library of Science, vol. 18(10), pages 1-3, October.
    6. Mario Pagliaro, 2021. "Purposeful Evaluation of Scholarship in the Open Science Era," Challenges, MDPI, vol. 12(1), pages 1-11, February.
    7. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    8. Daniel Torres-Salinas & Wenceslao Arroyo-Machado & Nicolas Robinson-Garcia, 2023. "Bibliometric denialism," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5357-5359, September.
    9. Marek Kosmulski, 2020. "Nobel laureates are not hot," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 487-495, April.
    10. Michel Balinski & Rida Laraki, 2011. "Majority Judgment: Measuring, Ranking, and Electing," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262015137, December.
    11. Fabrice Valognes & William V. Gehrlein, 2001. "Condorcet efficiency: A preference for indifference," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 18(1), pages 193-205.
    12. Kai Petersen & Nauman Bin Ali, 2021. "An analysis of top author citations in software engineering and a comparison with other fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9147-9183, November.
    13. Samreen Ayaz & Nayyer Masood, 2020. "Comparison of researchers’ impact indices," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-15, May.
    14. Schreiber, Michael, 2008. "A modification of the h-index: The hm-index accounts for multi-authored manuscripts," Journal of Informetrics, Elsevier, vol. 2(3), pages 211-216.
    15. William Gehrlein & Fabrice Valognes, 2001. "Condorcet efficiency: A preference for indifference," Post-Print halshs-02173169, HAL.
    16. repec:plo:pbio00:3000384 is not listed on IDEAS
    17. Saltelli A. & Tarantola S., 2002. "On the Relative Importance of Input Factors in Mathematical Models: Safety Assessment for Nuclear Waste Disposal," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 702-709, September.
    18. Vladlen Koltun & David Hafner, 2021. "The h-index is no longer an effective correlate of scientific reputation," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-16, June.
    19. Giuseppe Munda, 2008. "Social Multi-Criteria Evaluation for a Sustainable Economy," Springer Books, Springer, number 978-3-540-73703-2, December.
    20. Živan Živković & Marija Panić, 2020. "Development of science and education in the Western Balkan countries: competitiveness with the EU," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2319-2339, September.
    21. Wil M. P. Aalst & Oliver Hinz & Christof Weinhardt, 2023. "Ranking the Ranker: How to Evaluate Institutions, Researchers, Journals, and Conferences?," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(6), pages 615-621, December.
    22. Sachin S. Gunthe & Ravindra Gettu, 2022. "A new index for assessing faculty research performance in higher educational institutions of emerging economies such as India," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4959-4976, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yang, Alex Jie & Wu, Linwei & Zhang, Qi & Wang, Hao & Deng, Sanhong, 2023. "The k-step h-index in citation networks at the paper, author, and institution levels," Journal of Informetrics, Elsevier, vol. 17(4).
    2. Nick Haslam & Naomi Baes, 2023. "Scientific eminence and scientific hierarchy: bibliometric prediction of fellowship in the Australian Academy of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(12), pages 6659-6674, December.
    3. Lauranne Chaignon & Domingo Docampo & Daniel Egret, 2023. "In search of a scientific elite: highly cited researchers (HCR) in France," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5801-5827, October.
    4. Nicola Scafetta, 2025. "Measuring scholarly performance using comprehensive standardized research-teaching (RT) score," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(5), pages 2595-2616, May.
    5. Regenwetter, Michel & Grofman, Bernard & Marley, A. A. J., 2002. "On the model dependence of majority preference relations reconstructed from ballot or survey data," Mathematical Social Sciences, Elsevier, vol. 43(3), pages 451-466, July.
    6. Mostapha Diss & Eric Kamwa & Issofa Moyouwou & Hatem Smaoui, 2021. "Condorcet Efficiency of General Weighted Scoring Rules Under IAC: Indifference and Abstention," Studies in Choice and Welfare, in: Mostapha Diss & Vincent Merlin (ed.), Evaluating Voting Systems with Probability Models, pages 55-73, Springer.
    7. Wil M. P. Aalst & Oliver Hinz & Christof Weinhardt, 2023. "Ranking the Ranker: How to Evaluate Institutions, Researchers, Journals, and Conferences?," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(6), pages 615-621, December.
    8. Merlin, Vincent & Valognes, Fabrice, 2004. "The impact of indifferent voters on the likelihood of some voting paradoxes," Mathematical Social Sciences, Elsevier, vol. 48(3), pages 343-361, November.
    9. Becker William & Paruolo Paolo & Saltelli Andrea, 2021. "Variable Selection in Regression Models Using Global Sensitivity Analysis," Journal of Time Series Econometrics, De Gruyter, vol. 13(2), pages 187-233, July.
    10. Tommaso Luzzati & Bruno Cheli & S. Arcuri, 2014. "Measuring the sustainability performances of the Italian regions," Discussion Papers 2014/187, Dipartimento di Economia e Management (DEM), University of Pisa, Pisa, Italy.
    11. Tommaso Luzzati & Bruno Cheli & Gianluca Gucciardi, 2017. "Communicating the uncertainty of synthetic indicators: a reassessment of the HDI ranking," Discussion Papers 2017/228, Dipartimento di Economia e Management (DEM), University of Pisa, Pisa, Italy.
    12. Mostapha Diss & Eric Kamwa & Issofa Moyouwou & Hatem Smaoui, 2019. "Condorcet efficiency of general weighted scoring rules under IAC: indifference and abstention," Working Papers hal-02196387, HAL.
    13. Ruiz, Francisco & El Gibari, Samira & Cabello, José M. & Gómez, Trinidad, 2020. "MRP-WSCI: Multiple reference point based weak and strong composite indicators," Omega, Elsevier, vol. 95(C).
    14. Enrique Orduña-Malea & Núria Bautista-Puig, 2024. "Research assessment under debate: disentangling the interest around the DORA declaration on Twitter," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(1), pages 537-559, January.
    15. William V. Gehrlein & Dominique Lepelley, 2015. "The Condorcet Efficiency Advantage that Voter Indifference Gives to Approval Voting Over Some Other Voting Rules," Group Decision and Negotiation, Springer, vol. 24(2), pages 243-269, March.
    16. John P A Ioannidis & Kevin Boyack & Paul F Wouters, 2016. "Citation Metrics: A Primer on How (Not) to Normalize," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-7, September.
    17. Lepelley, Dominique & Martin, Mathieu, 2001. "Condorcet's paradox for weak preference orderings," European Journal of Political Economy, Elsevier, vol. 17(1), pages 163-177, March.
    18. Mathieu Martin, 2002. "On the emptiness of the stability set of order d," Theory and Decision, Springer, vol. 52(4), pages 313-326, June.
    19. Torres-Salinas, Daniel & Orduña-Malea, Enrique & Delgado-Vázquez, Ángel & Gorraiz, Juan & Arroyo-Machado, Wenceslao, 2024. "Foundations of Narrative Bibliometrics," Journal of Informetrics, Elsevier, vol. 18(3).
    20. Kai Petersen & Nauman Bin Ali, 2021. "An analysis of top author citations in software engineering and a comparison with other fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9147-9183, November.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:130:y:2025:i:3:d:10.1007_s11192-025-05253-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.