IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v130y2025i3d10.1007_s11192-025-05244-y.html
   My bibliography  Save this article

A large-scale semi-automated approach for assessing document-type classification errors in bibliometric databases

Author

Listed:
  • D. A. Maisano

    (Politecnico Di Torino)

  • L. Mastrogiacomo

    (Politecnico Di Torino)

  • L. Ferrara

    (Politecnico Di Torino)

  • F. Franceschini

    (Politecnico Di Torino)

Abstract

The accuracy of bibliometric databases in classifying document types (DTs)—such as research articles, conference proceedings, reviews, short notes, letters, book chapters, etc.—is crucial for the academic community, as bibliometric indicators may significantly influence research funding, decision-making, and academic reputation. This study presents a semi-automated methodology to assess the accuracy of DT classification in bibliometric databases, such as Scopus and Web of Science (WoS). The methodology can handle large document volumes and adapt to different DT categories without predefined correspondences. The first phase of the methodology automatically identifies discrepancies in DT classifications between Scopus and WoS, in order to find potentially misclassified documents; the second phase involves manually analyzing these documents to confirm and attribute classification errors. The methodology is applied to a sample of several tens of thousands of papers from the teaching staff of two major universities in Turin (Italy). The results show overall error rates of approximately 2.7% for Scopus and 2.3% for WoS. The paper also analyzes the most common types of errors found in both databases, providing an interpretation of these inaccuracies and some insights for possible improvements in the quality of these databases.

Suggested Citation

  • D. A. Maisano & L. Mastrogiacomo & L. Ferrara & F. Franceschini, 2025. "A large-scale semi-automated approach for assessing document-type classification errors in bibliometric databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(3), pages 1901-1938, March.
    • Handle: RePEc:spr:scient:v:130:y:2025:i:3:d:10.1007_s11192-025-05244-y
    DOI: 10.1007/s11192-025-05244-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-025-05244-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-025-05244-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2015. "Errors in DOI indexing by bibliometric databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2181-2186, March.
    2. Marlies Olensky & Marion Schmidt & Nees Jan Eck, 2016. "Evaluation of the citation matching algorithms of CWTS and iFQ in comparison to the Web of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(10), pages 2550-2564, October.
    3. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    4. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2013. "A novel approach for estimating the omitted‐citation rate of bibliometric databases with an application to the field of bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(10), pages 2149-2156, October.
    5. Paul Donner, 2017. "Document type assignment accuracy in the journal citation index data of Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 219-236, October.
    6. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2013. "A novel approach for estimating the omitted-citation rate of bibliometric databases with an application to the field of bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(10), pages 2149-2156, October.
    7. Anne-Wil Harzing, 2013. "Document categories in the ISI Web of Knowledge: Misunderstanding the Social Sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 23-34, January.
    8. Valderrama-Zurián, Juan-Carlos & Aguilar-Moya, Remedios & Melero-Fuentes, David & Aleixandre-Benavent, Rafael, 2015. "A systematic analysis of duplicate records in Scopus," Journal of Informetrics, Elsevier, vol. 9(3), pages 570-576.
    9. Anne Sigogneau, 2000. "An Analysis of Document Types Published in Journals Related to Physics: Proceeding Papers Recorded in the Science Citation Index Database," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(3), pages 589-604, March.
    10. Franceschini, Fiorenzo & Maisano, Domenico & Mastrogiacomo, Luca, 2016. "Empirical analysis and classification of database errors in Scopus and Web of Science," Journal of Informetrics, Elsevier, vol. 10(4), pages 933-953.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Shirley Ainsworth & Jane M. Russell, 2018. "Has hosting on science direct improved the visibility of Latin American scholarly journals? A preliminary analysis of data quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1463-1484, June.
    3. Thelwall, Mike, 2018. "Microsoft Academic automatic document searches: Accuracy for journal articles and suitability for citation analysis," Journal of Informetrics, Elsevier, vol. 12(1), pages 1-9.
    4. Paul Donner, 2017. "Document type assignment accuracy in the journal citation index data of Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 219-236, October.
    5. Shuo Xu & Liyuan Hao & Xin An & Dongsheng Zhai & Hongshen Pang, 2019. "Types of DOI errors of cited references in Web of Science with a cleaning method," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1427-1437, September.
    6. Nicolas Robinson-Garcia & Benjamín Vargas-Quesada & Daniel Torres-Salinas & Zaida Chinchilla-Rodríguez & Juan Gorraiz, 2024. "Errors of measurement in scientometrics: classification schemes and document types in citation and publication rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(10), pages 6455-6475, October.
    7. Franceschini, Fiorenzo & Maisano, Domenico & Mastrogiacomo, Luca, 2016. "Empirical analysis and classification of database errors in Scopus and Web of Science," Journal of Informetrics, Elsevier, vol. 10(4), pages 933-953.
    8. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    9. Gabriel Alves Vieira & Jacqueline Leta, 2024. "biblioverlap: an R package for document matching across bibliographic datasets," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4513-4527, July.
    10. Gerson Pech & Catarina Delgado, 2020. "Assessing the publication impact using citation data from both Scopus and WoS databases: an approach validated in 15 research fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 909-924, November.
    11. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    12. Thed Leeuwen & Rodrigo Costas & Clara Calero-Medina & Martijn Visser, 2013. "The role of editorial material in bibliometric research performance assessments," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 817-828, May.
    13. Bakthavachalam Elango, 2025. "Assigning different document types by Scopus for similar contents: an exploratory analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(3), pages 1989-2003, March.
    14. Gerson Pech & Catarina Delgado, 2020. "Percentile and stochastic-based approach to the comparison of the number of citations of articles indexed in different bibliographic databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 223-252, April.
    15. Carolin Michels & Jun-Ying Fu, 2014. "Systematic analysis of coverage and usage of conference proceedings in web of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(2), pages 307-327, August.
    16. Alessia Cioffi & Sara Coppini & Arcangelo Massari & Arianna Moretti & Silvio Peroni & Cristian Santini & Nooshin Shahidzadeh Asadi, 2022. "Identifying and correcting invalid citations due to DOI errors in Crossref data," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3593-3612, June.
    17. Junwen Zhu & Fang Liu & Weishu Liu, 2019. "The secrets behind Web of Science’s DOI search," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1745-1753, June.
    18. Domenico A. Maisano & Luca Mastrogiacomo & Fiorenzo Franceschini, 2023. "Empirical evidence on the relationship between research and teaching in academia," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4475-4507, August.
    19. Alexander Schniedermann, 2021. "A comparison of systematic reviews and guideline-based systematic reviews in medical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9829-9846, December.
    20. Hui Li & Xingmei Zhang, 2024. "Dr. Anonymous is still there: a revisit of legal scholarly publishing," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(1), pages 681-692, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:130:y:2025:i:3:d:10.1007_s11192-025-05244-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.