IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v129y2024i6d10.1007_s11192-024-05039-7.html
   My bibliography  Save this article

The use of ChatGPT to find similar institutions for institutional benchmarking

Author

Listed:
  • Lutz Bornmann

    (Administrative Headquarters of the Max Planck Society)

  • Benedetto Lepori

    (Università Della Svizzera Italiana)

Abstract

In evaluative bibliometrics and higher education studies, one is frequently confronted with the task of comparing institutions with similar institutions. In this Letter to the Editor, a simple approach is discussed which applies ChatGPT. Although the approach seems to produce promising results (tested with an example at the level of research institute and of a university), it is necessary to investigate it systematically based on a sample including many institutions before it should be applied in research evaluation. The challenge in systematic investigations is that ChatGPT provides the user with different answers on the sane request (missing reliability).

Suggested Citation

  • Lutz Bornmann & Benedetto Lepori, 2024. "The use of ChatGPT to find similar institutions for institutional benchmarking," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(6), pages 3593-3598, June.
    • Handle: RePEc:spr:scient:v:129:y:2024:i:6:d:10.1007_s11192-024-05039-7
    DOI: 10.1007/s11192-024-05039-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-024-05039-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-024-05039-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. E.C.M. Noyons & H.F. Moed & M. Luwel, 1999. "Combining mapping and citation analysis for evaluative bibliometric purposes: A bibliometric study," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 50(2), pages 115-131.
    2. Lutz Bornmann & Felix Moya Anegón & Rüdiger Mutz, 2013. "Do universities or research institutions with a specific subject profile have an advantage or a disadvantage in institutional rankings?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(11), pages 2310-2316, November.
    3. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    4. Nicolas Carayol & Ghislaine Filliatreau & Agenor Lahatte, 2012. "Reference classes: a tool for benchmarking universities’ research," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(2), pages 351-371, November.
    5. Qi Wang & Tobias Jeppsson, 2022. "Identifying benchmark units for research management and evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7557-7574, December.
    6. Lutz Bornmann & Felix de Moya Anegón & Rüdiger Mutz, 2013. "Do Universities or Research Institutions With a Specific Subject Profile Have an Advantage or a Disadvantage in Institutional Rankings? A Latent Class Analysis With Data From the SCImago Ranking," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(11), pages 2310-2316, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giulio Giacomo Cantone, 2024. "How to measure interdisciplinary research? A systemic design for the model of measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4937-4982, August.
    2. Lutz Bornmann & Lingfei Wu & Christoph Ettl, 2024. "The use of ChatGPT for identifying disruptive papers in science: a first exploration," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 7161-7165, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Klaus Wohlrabe & Felix de Moya Anegon & Lutz Bornmann, 2018. "How efficiently produce elite US universities highly cited papers? A case study based on input and output data," ifo Working Paper Series 264, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    2. Klaus Wohlrabe & Félix de Moya Anegon & Lutz Bornmann, 2019. "How Efficiently Do Elite US Universities Produce Highly Cited Papers?," Publications, MDPI, vol. 7(1), pages 1-15, January.
    3. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    4. Manganote, Edmilson J.T. & Araujo, Mariana S. & Schulz, Peter A., 2014. "Visualization of ranking data: Geographical signatures in international collaboration, leadership and research impact," Journal of Informetrics, Elsevier, vol. 8(3), pages 642-649.
    5. Hamdi A. Al-Jamimi & Galal M. BinMakhashen & Lutz Bornmann & Yousif Ahmed Al Wajih, 2023. "Saudi Arabia research: academic insights and trend analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5595-5627, October.
    6. Bornmann, Lutz, 2019. "Does the normalized citation impact of universities profit from certain properties of their published documents – such as the number of authors and the impact factor of the publishing journals? A mult," Journal of Informetrics, Elsevier, vol. 13(1), pages 170-184.
    7. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    8. Bornmann, Lutz & Haunschild, Robin, 2016. "Normalization of Mendeley reader impact on the reader- and paper-side: A comparison of the mean discipline normalized reader score (MDNRS) with the mean normalized reader score (MNRS) and bare reader ," Journal of Informetrics, Elsevier, vol. 10(3), pages 776-788.
    9. Qi Wang & Tobias Jeppsson, 2022. "Identifying benchmark units for research management and evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7557-7574, December.
    10. Bornmann, Lutz & Marx, Werner, 2018. "Critical rationalism and the search for standard (field-normalized) indicators in bibliometrics," Journal of Informetrics, Elsevier, vol. 12(3), pages 598-604.
    11. Ruben Miranda & Esther Garcia-Carpintero, 2019. "Comparison of the share of documents and citations from different quartile journals in 25 research areas," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 479-501, October.
    12. Waltman, Ludo & van Eck, Nees Jan, 2015. "Field-normalized citation impact indicators and the choice of an appropriate counting method," Journal of Informetrics, Elsevier, vol. 9(4), pages 872-894.
    13. Piñeiro-Chousa, Juan & López-Cabarcos, M. Ángeles & Romero-Castro, Noelia María & Pérez-Pico, Ada María, 2020. "Innovation, entrepreneurship and knowledge in the business scientific field: Mapping the research front," Journal of Business Research, Elsevier, vol. 115(C), pages 475-485.
    14. Gaviria-Marin, Magaly & Merigó, José M. & Baier-Fuentes, Hugo, 2019. "Knowledge management: A global examination based on bibliometric analysis," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 194-220.
    15. Gallego-Losada, María-Jesús & Montero-Navarro, Antonio & García-Abajo, Elisa & Gallego-Losada, Rocío, 2023. "Digital financial inclusion. Visualizing the academic literature," Research in International Business and Finance, Elsevier, vol. 64(C).
    16. P. Ebin Babu & Riya Mary, 2022. "Exploring the Research Trends in Green Tax: Bibliometric Analysis," International Journal of Energy Economics and Policy, Econjournals, vol. 12(6), pages 157-162, November.
    17. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    18. Daraio, Cinzia & Simar, Leopold & Wilson, Paul, 2019. "Quality and its impact on efficiency," LIDAM Discussion Papers ISBA 2019004, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    19. Ming Tang & Huchang Liao & Zhengjun Wan & Enrique Herrera-Viedma & Marc A. Rosen, 2018. "Ten Years of Sustainability (2009 to 2018): A Bibliometric Overview," Sustainability, MDPI, vol. 10(5), pages 1-21, May.
    20. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2022. "Using the Leiden Rankings as a Heuristics: Evidence from Italian universities in the European landscape," LEM Papers Series 2022/08, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:129:y:2024:i:6:d:10.1007_s11192-024-05039-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.