IDEAS home Printed from https://ideas.repec.org/a/plo/pbio00/1002542.html
   My bibliography  Save this article

Citation Metrics: A Primer on How (Not) to Normalize

Author

Listed:
  • John P A Ioannidis
  • Kevin Boyack
  • Paul F Wouters

Abstract

Citation metrics are increasingly used to appraise published research. One challenge is whether and how to normalize these metrics to account for differences across scientific fields, age (year of publication), type of document, database coverage, and other factors. We discuss the pros and cons for normalizations using different approaches. Additional challenges emerge when citation metrics need to be combined across multiple papers to appraise the corpus of scientists, institutions, journals, or countries, as well as when trying to attribute credit in multiauthored papers. Different citation metrics may offer complementary insights, but one should carefully consider the assumptions that underlie their calculation.Citation metrics are very influential and their normalization is a contentious issue. Each normalization approach has advantages and disadvantages that need to be understood for proper use of these metrics.

Suggested Citation

  • John P A Ioannidis & Kevin Boyack & Paul F Wouters, 2016. "Citation Metrics: A Primer on How (Not) to Normalize," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-7, September.
    • Handle: RePEc:plo:pbio00:1002542
    DOI: 10.1371/journal.pbio.1002542
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002542
    Download Restriction: no
    File URL: https://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.1002542&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pbio.1002542?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ding, Ying & Liu, Xiaozhong & Guo, Chun & Cronin, Blaise, 2013. "The distribution of references across texts: Some implications for citation analysis," Journal of Informetrics, Elsevier, vol. 7(3), pages 583-592.
    2. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    3. Hu, Zhigang & Chen, Chaomei & Liu, Zeyuan, 2013. "Where are citations located in the body of scientific articles? A study of the distributions of citation locations," Journal of Informetrics, Elsevier, vol. 7(4), pages 887-896.
    4. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.
    5. Richard S. J. Tol, 2011. "Credit where credit’s due: accounting for co-authorship in citation counts," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 291-299, October.
    6. Anne-Wil Harzing, 2014. "A longitudinal study of Google Scholar coverage between 2012 and 2013," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 565-575, January.
    7. Kaur, Jasleen & Radicchi, Filippo & Menczer, Filippo, 2013. "Universality of scholarly impact metrics," Journal of Informetrics, Elsevier, vol. 7(4), pages 924-932.
    8. B Ian Hutchins & Xin Yuan & James M Anderson & George M Santangelo, 2016. "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-25, September.
    9. Thed Leeuwen & Rodrigo Costas & Clara Calero-Medina & Martijn Visser, 2013. "The role of editorial material in bibliometric research performance assessments," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 817-828, May.
    10. Schreiber, Michael, 2008. "A modification of the h-index: The hm-index accounts for multi-authored manuscripts," Journal of Informetrics, Elsevier, vol. 2(3), pages 211-216.
    11. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    12. Ruiz-Castillo, Javier & Waltman, Ludo, 2015. "Field-normalized citation impact indicators using algorithmically constructed classification systems of science," Journal of Informetrics, Elsevier, vol. 9(1), pages 102-117.
    13. Waltman, Ludo, 2012. "An empirical analysis of the use of alphabetical authorship in scientific publishing," Journal of Informetrics, Elsevier, vol. 6(4), pages 700-711.
    14. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    2. Wang, Jiang-Pan & Guo, Qiang & Zhou, Lei & Liu, Jian-Guo, 2019. "Dynamic credit allocation for researchers," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 520(C), pages 208-216.
    3. Cano-Marin, Enrique & Mora-Cantallops, Marçal & Sánchez-Alonso, Salvador, 2023. "Twitter as a predictive system: A systematic literature review," Journal of Business Research, Elsevier, vol. 157(C).
    4. Antonin Mac'e, 2017. "The Limits of Citation Counts," Papers 1711.02695, arXiv.org, revised Sep 2023.
    5. Rodríguez-Navarro, Alonso & Brito, Ricardo, 2018. "Double rank analysis for research assessment," Journal of Informetrics, Elsevier, vol. 12(1), pages 31-41.
    6. Maziar Montazerian & Edgar Dutra Zanotto & Hellmut Eckert, 2020. "Prolificacy and visibility versus reputation in the hard sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 207-221, April.
    7. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    8. Juan Miguel Campanario, 2018. "Are leaders really leading? Journals that are first in Web of Science subject categories in the context of their groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 111-130, April.
    9. Robin Haunschild & Lutz Bornmann, 2022. "Relevance of document types in the scores’ calculation of a specific field-normalized indicator: Are the scores strongly dependent on or nearly independent of the document type handling?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4419-4438, August.
    10. Maziar Montazerian & Edgar Dutra Zanotto & Hellmut Eckert, 2019. "A new parameter for (normalized) evaluation of H-index: countries as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1065-1078, March.
    11. Antonin Macé, 2023. "The Limits of Citation Counts," Working Papers halshs-01630095, HAL.
    12. Haunschild, Robin & Daniels, Angela D. & Bornmann, Lutz, 2022. "Scores of a specific field-normalized indicator calculated with different approaches of field-categorization: Are the scores different or similar?," Journal of Informetrics, Elsevier, vol. 16(1).
    13. Loet Leydesdorff & Paul Wouters & Lutz Bornmann, 2016. "Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2129-2150, December.
    14. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    15. Edgar D. Zanotto & Vinicius Carvalho, 2021. "Article age- and field-normalized tools to evaluate scientific impact and momentum," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 2865-2883, April.
    16. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    17. Lutz Bornmann & Richard Williams, 2020. "An evaluation of percentile measures of citation impact, and a proposal for making them better," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1457-1478, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    3. Bornmann, Lutz & Marx, Werner, 2018. "Critical rationalism and the search for standard (field-normalized) indicators in bibliometrics," Journal of Informetrics, Elsevier, vol. 12(3), pages 598-604.
    4. Loet Leydesdorff & Paul Wouters & Lutz Bornmann, 2016. "Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2129-2150, December.
    5. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    6. Peter Sjögårde & Fereshteh Didegah, 2022. "The association between topic growth and citation impact of research publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1903-1921, April.
    7. James C. Ryan, 2016. "A validation of the individual annual h-index (hIa): application of the hIa to a qualitatively and quantitatively different sample," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(1), pages 577-590, October.
    8. Corrêa Jr., Edilson A. & Silva, Filipi N. & da F. Costa, Luciano & Amancio, Diego R., 2017. "Patterns of authors contribution in scientific manuscripts," Journal of Informetrics, Elsevier, vol. 11(2), pages 498-510.
    9. Torres-Salinas, Daniel & Valderrama-Baca, Pilar & Arroyo-Machado, Wenceslao, 2022. "Is there a need for a new journal metric? Correlations between JCR Impact Factor metrics and the Journal Citation Indicator—JCI," Journal of Informetrics, Elsevier, vol. 16(3).
    10. B Ian Hutchins & Xin Yuan & James M Anderson & George M Santangelo, 2016. "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-25, September.
    11. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    12. Hu, Zhigang & Tian, Wencan & Xu, Shenmeng & Zhang, Chunbo & Wang, Xianwen, 2018. "Four pitfalls in normalizing citation indicators: An investigation of ESI’s selection of highly cited papers," Journal of Informetrics, Elsevier, vol. 12(4), pages 1133-1145.
    13. Anne-Wil Harzing & Satu Alakangas & David Adams, 2014. "hIa: an individual annual h-index to accommodate disciplinary and career length differences," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(3), pages 811-821, June.
    14. CholMyong Pak & Guang Yu & Weibin Wang, 2018. "A study on the citation situation within the citing paper: citation distribution of references according to mention frequency," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 905-918, March.
    15. Tolga Yuret, 2018. "Author-weighted impact factor and reference return ratio: can we attain more equality among fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2097-2111, September.
    16. Weibin Wang & Zheng Wang & Tian Yu & CholMyong Pak & Guang Yu, 2020. "Research on citation mention times and contributions using a neural network," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2383-2400, December.
    17. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    18. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    19. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    20. Potter, Ross W.K. & Szomszor, Martin & Adams, Jonathan, 2020. "Interpreting CNCIs on a country-scale: The effect of domestic and international collaboration type," Journal of Informetrics, Elsevier, vol. 14(4).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pbio00:1002542. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosbiology (email available below). General contact details of provider: https://journals.plos.org/plosbiology/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.