IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v130y2025i7d10.1007_s11192-025-05338-7.html
   My bibliography  Save this article

How similar are field-normalized citation impact scores obtained from OpenAlex and three popular commercial databases? An empirical comparison based on large German universities

Author

Listed:
  • Thomas Scheidsteger

    (IVS-CPT, Max Planck Institute for Solid State Research)

  • Robin Haunschild

    (IVS-CPT, Max Planck Institute for Solid State Research)

  • Lutz Bornmann

    (IVS-CPT, Max Planck Institute for Solid State Research
    Administrative Headquarters of the Max Planck Society)

Abstract

OpenAlex is a freely available bibliographic database that can be used for bibliometric studies. In this study, we compared certain field-normalized citation scores (NCS) from OpenAlex with those from three commercial databases (Web of Science, Scopus, and Dimensions). We were interested in the question whether the NCS from OpenAlex are comparable to those from the commercial databases and can be alternatively used in evaluative bibliometrics. The NCS have been calculated for nearly 335,000 papers published by 48 German universities in four main subject areas between 2013 and 2017. We found varying but overall strong agreement between the scores according to Lin’s concordance correlation coefficient. Separating the publication set along the single universities and moreover along the four main subject areas involved revealed significant differences at the level of single papers but also gave indications on how to possibly mitigate outlier cases. We calculated mean normalized citation scores for the 48 universities and found that the agreements across different databases are low. On the one hand, the results suggest that comparisons of universities using NCS across different databases should be avoided. On the other hand, the difference of the concordance correlation coefficients at paper and university level is a good example for the problem of ecological fallacy in bibliometrics: The mean impact is not representative for the single papers’ impact in the set.

Suggested Citation

  • Thomas Scheidsteger & Robin Haunschild & Lutz Bornmann, 2025. "How similar are field-normalized citation impact scores obtained from OpenAlex and three popular commercial databases? An empirical comparison based on large German universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(7), pages 3537-3569, July.
    • Handle: RePEc:spr:scient:v:130:y:2025:i:7:d:10.1007_s11192-025-05338-7
    DOI: 10.1007/s11192-025-05338-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-025-05338-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-025-05338-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Ludo Waltman & Clara Calero‐Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan van Eck & Thed N. van Leeuwen & Anthony F.J. van Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    2. Stephan Stahlschmidt & Dimity Stephen, 2022. "From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2413-2431, May.
    3. Lokman I. Meho & Cassidy R. Sugimoto, 2009. "Assessing the scholarly impact of information studies: A tale of two citation databases—Scopus and Web of Science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(12), pages 2499-2508, December.
    4. Haunschild, Robin & Daniels, Angela D. & Bornmann, Lutz, 2022. "Scores of a specific field-normalized indicator calculated with different approaches of field-categorization: Are the scores different or similar?," Journal of Informetrics, Elsevier, vol. 16(1).
    5. Ludo Waltman & Clara Calero-Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan Eck & Thed N. Leeuwen & Anthony F.J. Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    6. Irina D. Turgel & Olga A. Chernova, 2024. "Open Science Alternatives to Scopus and the Web of Science: A Case Study in Regional Resilience," Publications, MDPI, vol. 12(4), pages 1-18, November.
    7. Wolfgang Haak & Iosif Lazaridis & Nick Patterson & Nadin Rohland & Swapan Mallick & Bastien Llamas & Guido Brandt & Susanne Nordenfelt & Eadaoin Harney & Kristin Stewardson & Qiaomei Fu & Alissa Mittn, 2015. "Massive migration from the steppe was a source for Indo-European languages in Europe," Nature, Nature, vol. 522(7555), pages 207-211, June.
    8. Robin Haunschild & Lutz Bornmann, 2022. "Relevance of document types in the scores’ calculation of a specific field-normalized indicator: Are the scores strongly dependent on or nearly independent of the document type handling?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4419-4438, August.
    9. Lundberg, Jonas, 2007. "Lifting the crown—citation z-score," Journal of Informetrics, Elsevier, vol. 1(2), pages 145-154.
    10. José Luis Ortega & Lorena Delgado-Quirós, 2024. "The indexation of retracted literature in seven principal scholarly databases: a coverage comparison of dimensions, OpenAlex, PubMed, Scilit, Scopus, The Lens and Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 3769-3785, July.
    11. Daniel Torres-Salinas & Emilio Delgado Lopez-Cózar & Evaristo Jiménez-Contreras, 2009. "Ranking of departments and researchers within a university using two different databases: Web of Science versus Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(3), pages 761-774, September.
    12. Lutz Bornmann & Richard Williams, 2020. "An evaluation of percentile measures of citation impact, and a proposal for making them better," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1457-1478, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Nicolas Robinson-Garcia & Benjamín Vargas-Quesada & Daniel Torres-Salinas & Zaida Chinchilla-Rodríguez & Juan Gorraiz, 2024. "Errors of measurement in scientometrics: classification schemes and document types in citation and publication rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(10), pages 6455-6475, October.
    3. Robin Haunschild & Lutz Bornmann, 2022. "Relevance of document types in the scores’ calculation of a specific field-normalized indicator: Are the scores strongly dependent on or nearly independent of the document type handling?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4419-4438, August.
    4. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    5. Lutz Bornmann & Rüdiger Mutz & Robin Haunschild & Felix Moya-Anegon & Mirko Almeida Madeira Clemente & Moritz Stefaner, 2021. "Mapping the impact of papers on various status groups in excellencemapping.net: a new release of the excellence mapping tool based on citation and reader scores," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9305-9331, November.
    6. Hoekman, Jarno & Rake, Bastian, 2024. "Geography of authorship: How geography shapes authorship attribution in big team science," Research Policy, Elsevier, vol. 53(2).
    7. Marco Cavallaro & Benedetto Lepori, 2021. "Institutional barriers to participation in EU framework programs: contrasting the Swiss and UK cases," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1311-1328, February.
    8. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    9. Jeffrey Demaine, 2022. "Fractionalization of research impact reveals global trends in university collaboration," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2235-2247, May.
    10. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2022. "Using the Leiden Rankings as a Heuristics: Evidence from Italian universities in the European landscape," LEM Papers Series 2022/08, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    11. Leporia, Benedetto & Geuna, Aldo & Mira, Antonietta, 2018. "Scientific Output of US and European Universities Scales Super-linearly with Resources," Department of Economics and Statistics Cognetti de Martiis LEI & BRICK - Laboratory of Economics of Innovation "Franco Momigliano", Bureau of Research in Innovation, Complexity and Knowledge, Collegio 201806, University of Turin.
    12. Pedro Albarrán & Antonio Perianes-Rodríguez & Javier Ruiz-Castillo, 2015. "Differences in citation impact across countries," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(3), pages 512-525, March.
    13. Lavinia Mustea, 2022. "An Overview of Public Sector Performance in Europe," Ovidius University Annals, Economic Sciences Series, Ovidius University of Constantza, Faculty of Economic Sciences, vol. 0(1), pages 339-345, September.
    14. Fabio S. V. Silva & Peter A. Schulz & Everard C. M. Noyons, 2019. "Co-authorship networks and research impact in large research facilities: benchmarking internal reports and bibliometric databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 93-108, January.
    15. Klaus Wohlrabe & Sabine Gralka & Lutz Bornmann, 2019. "Zur Effizienz deutscher Universitäten und deren Entwicklung zwischen 2004 und 2015," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 72(21), pages 15-21, November.
    16. Fredrik Niclas Piro, 2019. "The R&D composition of European countries: concentrated versus dispersed profiles," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1095-1119, May.
    17. Tomaz Bartol & Gordana Budimir & Doris Dekleva-Smrekar & Miro Pusnik & Primoz Juznic, 2014. "Assessment of research fields in Scopus and Web of Science in the view of national research evaluation in Slovenia," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1491-1504, February.
    18. Massucci, Francesco Alessandro & Docampo, Domingo, 2019. "Measuring the academic reputation through citation networks via PageRank," Journal of Informetrics, Elsevier, vol. 13(1), pages 185-201.
    19. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    20. Mike Thelwall, 2019. "The influence of highly cited papers on field normalised indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 519-537, February.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:130:y:2025:i:7:d:10.1007_s11192-025-05338-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.