IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v109y2016i2d10.1007_s11192-016-2111-2.html
   My bibliography  Save this article

Change of perspective: bibliometrics from the point of view of cited references—a literature overview on approaches to the evaluation of cited references in bibliometrics

Author

Listed:
  • Werner Marx

    (Max Planck Institute for Solid State Research)

  • Lutz Bornmann

    (Administrative Headquarters of the Max Planck Society)

Abstract

Citation analyses normally investigate the number of citations of publications (e.g. by people, institutions or journals) where the information on times cited from the bibliographic databases (such as Scopus or Web of Science) is evaluated. But in recent years, a series of works have also been published which have undertaken a change of perspective and are based on the evaluation of the cited references. The cited references are the works cited in the publications which are used to calculate the times cited. Since these evaluations have led to important insights into science and into scientometric indicators, this paper presents an overview of methods based on cited references, and examples of some empirical results from studies are presented. Thus, the investigation of references allows general statements to be made on the precision of citation analyses, and offers alternatives for the normalization of citation numbers in the framework of research evaluation using citation impact. Via the analysis of references, the historical roots of research areas or the works of decisive importance in an area can be determined. References allow quantitative statements on the interdisciplinarity of research units and the overall growth of science. The use of a selection for the analysis of references from the publications of specific research areas enables the possibility of measuring citation impact target-oriented (i.e. limited to these areas). As some empirical studies have shown, the identification of publications with a high creative content seems possible via the analysis of the cited references. The possibilities presented here for cited reference analysis indicate the great potential of the data source. We assume that there are additional possibilities for its application in scientometrics.

Suggested Citation

  • Werner Marx & Lutz Bornmann, 2016. "Change of perspective: bibliometrics from the point of view of cited references—a literature overview on approaches to the evaluation of cited references in bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1397-1415, November.
  • Handle: RePEc:spr:scient:v:109:y:2016:i:2:d:10.1007_s11192-016-2111-2
    DOI: 10.1007/s11192-016-2111-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-016-2111-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11192-016-2111-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Vincent Larivière & Yves Gingras, 2010. "On the relationship between interdisciplinarity and scientific impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(1), pages 126-131, January.
    2. Ludo Waltman & Nees Eck, 2013. "A smart local moving algorithm for large-scale modularity-based community detection," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 86(11), pages 1-14, November.
    3. Vincent Larivière & Yves Gingras, 2010. "On the relationship between interdisciplinarity and scientific impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(1), pages 126-131, January.
    4. Lin Zhang & Frizo Janssens & Liming Liang & Wolfgang Glänzel, 2010. "Journal cross-citation analysis for validation and improvement of journal-based subject classification in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(3), pages 687-706, March.
    5. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    6. Lutz Bornmann & Werner Marx, 2014. "The wisdom of citing scientists," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(6), pages 1288-1292, June.
    7. Werner Marx & Lutz Bornmann, 2014. "Tracing the origin of a scientific legend by reference publication year spectroscopy (RPYS): the legend of the Darwin finches," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(3), pages 839-844, June.
    8. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    9. Richard Van Noorden & Brendan Maher & Regina Nuzzo, 2014. "The top 100 papers," Nature, Nature, vol. 514(7524), pages 550-553, October.
    10. Werner Marx & Lutz Bornmann, 2015. "On the causes of subject-specific citation rates in Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1823-1827, February.
    11. Di Vaio, Gianfranco & Waldenström, Daniel & Weisdorf, Jacob, 2012. "Citation success: Evidence from economic history journal publications," Explorations in Economic History, Elsevier, vol. 49(1), pages 92-104.
    12. Ludo Waltman & Clara Calero-Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan Eck & Thed N. Leeuwen & Anthony F.J. Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    13. Thor, Andreas & Marx, Werner & Leydesdorff, Loet & Bornmann, Lutz, 2016. "Introducing CitedReferencesExplorer (CRExplorer): A program for reference publication year spectroscopy with cited references standardization," Journal of Informetrics, Elsevier, vol. 10(2), pages 503-515.
    14. Werner Marx, 2011. "Special features of historical papers from the viewpoint of bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(3), pages 433-439, March.
    15. Werner Marx & Lutz Bornmann & Andreas Barth & Loet Leydesdorff, 2014. "Detecting the historical roots of research fields by reference publication year spectroscopy (RPYS)," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 751-764, April.
    16. Anthony F. J. van Raan, 2000. "On Growth, Ageing, and Fractal Differentiation of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(2), pages 347-362, February.
    17. Bornmann, Lutz & Marx, Werner, 2015. "Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?," Journal of Informetrics, Elsevier, vol. 9(2), pages 408-418.
    18. Bornmann, Lutz & Marx, Werner, 2013. "The proposal of a broadening of perspective in evaluative bibliometrics by complementing the times cited with a cited reference analysis," Journal of Informetrics, Elsevier, vol. 7(1), pages 84-88.
    19. Werner Marx, 2011. "Special features of historical papers from the viewpoint of bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(3), pages 433-439, March.
    20. Franceschini, Fiorenzo & Maisano, Domenico & Mastrogiacomo, Luca, 2016. "The museum of errors/horrors in Scopus," Journal of Informetrics, Elsevier, vol. 10(1), pages 174-182.
    21. Thomas Heinze, 2013. "Creative accomplishments in science: definition, theoretical considerations, examples from science history, and bibliometric findings," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(3), pages 927-940, June.
    22. Comins, Jordan A. & Hussey, Thomas W., 2015. "Compressing multiple scales of impact detection by Reference Publication Year Spectroscopy," Journal of Informetrics, Elsevier, vol. 9(3), pages 449-454.
    23. Loet Leydesdorff & Lutz Bornmann, 2016. "The operationalization of “fields” as WoS subject categories (WCs) in evaluative bibliometrics: The cases of “library and information science” and “science & technology studies”," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(3), pages 707-714, March.
    24. Werner Marx & Lutz Bornmann, 2013. "The emergence of plate tectonics and the Kuhnian model of paradigm shift: a bibliometric case study based on the Anna Karenina principle," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 595-614, February.
    25. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    26. Bakthavachalam Elango & Lutz Bornmann & Govindaraju Kannan, 2016. "Detecting the historical roots of tribology research: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(1), pages 305-313, April.
    27. Philippe Gorry & Pascal Ragouet, 2016. "“Sleeping beauty” and her restless sleep: Charles Dotter and the birth of interventional radiology," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 773-784, May.
    28. Alan L. Porter & Ismael Rafols, 2009. "Is science becoming more interdisciplinary? Measuring and mapping six research fields over time," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 719-745, December.
    29. Waltman, Ludo & Tijssen, Robert J.W. & Eck, Nees Jan van, 2011. "Globalisation of science in kilometres," Journal of Informetrics, Elsevier, vol. 5(4), pages 574-582.
    30. Leydesdorff, Loet & Bornmann, Lutz & Marx, Werner & Milojević, Staša, 2014. "Referenced Publication Years Spectroscopy applied to iMetrics: Scientometrics, Journal of Informetrics, and a relevant subset of JASIST," Journal of Informetrics, Elsevier, vol. 8(1), pages 162-174.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ksenia Silchenko, 2018. "The other "meta" of meta-analysis: Qualitative and text-based approaches to "analysis of analyses" in marketing," MERCATI & COMPETITIVIT?, FrancoAngeli Editore, vol. 2018(4), pages 27-45.
    2. António Osório & Lutz Bornmann, 2021. "On the disruptive power of small-teams research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 117-133, January.
    3. Andreas Thor & Lutz Bornmann & Werner Marx & Rüdiger Mutz, 2018. "Identifying single influential publications in a research field: new analysis opportunities of the CRExplorer," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 591-608, July.
    4. Kameliya Deyanova & Nataliia Brehmer & Artur Lapidus & Victor Tiberius & Steve Walsh, 2022. "Hatching start-ups for sustainable growth: a bibliometric review on business incubators," Review of Managerial Science, Springer, vol. 16(7), pages 2083-2109, October.
    5. Robin Haunschild & Werner Marx, 2022. "On Health Effects of Resveratrol in Wine," IJERPH, MDPI, vol. 19(5), pages 1-13, March.
    6. Fatemeh Ghaffari & Mark C. Wilson, 2023. "A model for reference list length of scholarly articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5335-5350, September.
    7. Lutz Bornmann & Robin Haunschild & Rüdiger Mutz, 2021. "Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-15, December.
    8. Sharma, Anuj & Nunkoo, Robin & Rana, Nripendra P. & Dwivedi, Yogesh K., 2021. "On the intellectual structure and influence of tourism social science research," Annals of Tourism Research, Elsevier, vol. 91(C).
    9. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    10. Matthew S. Bickley & Kayvan Kousha & Michael Thelwall, 2022. "A systematic method for identifying references to academic research in grey literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6913-6933, December.
    11. Lina Xu & Steven Dellaportas & Jin Wang, 2022. "A study of interdisciplinary accounting research: analysing the diversity of cited references," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 62(2), pages 2131-2162, June.
    12. Yonis Gulzar & Nisa Eksili & Pınar Celik Caylak & Mohammad Shuaib Mir, 2023. "Sustainability Consciousness Research Trends: A Bibliometric Analysis," Sustainability, MDPI, vol. 15(24), pages 1-26, December.
    13. Matthieu Ballandonne & Igor Cersosimo, 2021. "A note on reference publication year spectroscopy with incomplete information," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4927-4939, June.
    14. Anton Gruber & Alexander Tekles & Lutz Bornmann, 2023. "John Mearsheimer’s academic roots: a reference publication year spectroscopy of a political scientist’s oeuvre," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(7), pages 3867-3877, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andreas Thor & Lutz Bornmann & Werner Marx & Rüdiger Mutz, 2018. "Identifying single influential publications in a research field: new analysis opportunities of the CRExplorer," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 591-608, July.
    2. Werner Marx & Robin Haunschild & Andreas Thor & Lutz Bornmann, 2017. "Which early works are cited most frequently in climate change research literature? A bibliometric approach based on Reference Publication Year Spectroscopy," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 335-353, January.
    3. McLevey, John & McIlroy-Young, Reid, 2017. "Introducing metaknowledge: Software for computational research in information science, network analysis, and science of science," Journal of Informetrics, Elsevier, vol. 11(1), pages 176-197.
    4. Jianhua Hou, 2017. "Exploration into the evolution and historical roots of citation analysis by referenced publication year spectroscopy," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1437-1452, March.
    5. Jordan A. Comins & Loet Leydesdorff, 2017. "Citation algorithms for identifying research milestones driving biomedical innovation," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1495-1504, March.
    6. Comins, Jordan A. & Carmack, Stephanie A. & Leydesdorff, Loet, 2018. "Patent citation spectroscopy (PCS): Online retrieval of landmark patents based on an algorithmic approach," Journal of Informetrics, Elsevier, vol. 12(4), pages 1223-1231.
    7. Matthieu Ballandonne & Igor Cersosimo, 2021. "A note on reference publication year spectroscopy with incomplete information," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4927-4939, June.
    8. Werner Marx & Robin Haunschild & Bernie French & Lutz Bornmann, 2017. "Slow reception and under-citedness in climate change research: A case study of Charles David Keeling, discoverer of the risk of global warming," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 1079-1092, August.
    9. K. Brad Wray & Lutz Bornmann, 2015. "Philosophy of science viewed through the lense of “Referenced Publication Years Spectroscopy” (RPYS)," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 1987-1996, March.
    10. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    11. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    12. Loet Leydesdorff & Paul Wouters & Lutz Bornmann, 2016. "Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2129-2150, December.
    13. Thor, Andreas & Marx, Werner & Leydesdorff, Loet & Bornmann, Lutz, 2016. "Introducing CitedReferencesExplorer (CRExplorer): A program for reference publication year spectroscopy with cited references standardization," Journal of Informetrics, Elsevier, vol. 10(2), pages 503-515.
    14. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    15. Xin Li & Qiang Yao & Xuli Tang & Qian Li & Mengjia Wu, 2020. "How to investigate the historical roots and evolution of research fields in China? A case study on iMetrics using RootCite," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1253-1274, November.
    16. Juste Raimbault, 2019. "Exploration of an interdisciplinary scientific landscape," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 617-641, May.
    17. Feng Shi & James Evans, 2023. "Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    18. Liu, Weishu & Hu, Guangyuan & Tang, Li, 2018. "Missing author address information in Web of Science—An explorative study," Journal of Informetrics, Elsevier, vol. 12(3), pages 985-997.
    19. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    20. Fei Shu & Jesse David Dinneen & Shiji Chen, 2022. "Measuring the disparity among scientific disciplines using Library of Congress Subject Headings," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3613-3628, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:109:y:2016:i:2:d:10.1007_s11192-016-2111-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.