IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i2d10.1007_s11192-020-03812-y.html
   My bibliography  Save this article

Citation inequality and the Journal Impact Factor: median, mean, (does it) matter?

Author

Listed:
  • Tobias Kiesslich

    (Paracelsus Medical University
    Paracelsus Medical University/Salzburger Landeskliniken)

  • Marlena Beyreis

    (Paracelsus Medical University)

  • Georg Zimmermann

    (Paracelsus Medical University
    Paracelsus Medical University
    Paris-Lodron University of Salzburg)

  • Andreas Traweger

    (Paracelsus Medical University
    Austrian Cluster for Tissue Regeneration)

Abstract

Skewed citation distribution is a major limitation of the Journal Impact Factor (JIF) representing an outlier-sensitive mean citation value per journal The present study focuses primarily on this phenomenon in the medical literature by investigating a total of n = 982 journals from two medical categories of the Journal Citation Report (JCR). In addition, the three highest-ranking journals from each JCR category were included in order to extend the analyses to non-medical journals. For the journals in these cohorts, the citation data (2018) of articles published in 2016 and 2017 classified as citable items (CI) were analysed using various descriptive approaches including e.g. the skewness, the Gini coefficient, and, the percentage of CI contributing 50% or 90% of the journal’s citations. All of these measures clearly indicated an unequal, skewed distribution with highly-cited articles as outliers. The %CI contributing 50% or 90% of the journal’s citations was in agreement with previously published studies with median values of 13–18% CI or 44–60% CI generating 50 or 90% of the journal’s citations, respectively. Replacing the mean citation values (corresponding to the JIF) with the median to represent the central tendency of the citation distributions resulted in markedly lower numerical values ranging from − 30 to − 50%. Up to 39% of journals showed a median citation number of zero in one medical journal category. For the two medical cohorts, median-based journal ranking was similar to mean-based ranking although the number of possible rank positions was reduced to 13. Correlation of mean citations with the measures of citation inequality indicated that the unequal distribution of citations per journal is more prominent and, thus, relevant for journals with lower citation rates. By using various indicators in parallel and the hitherto probably largest journal sample, the present study provides comprehensive up-to-date results on the prevalence, extent and consequences of citation inequality across medical and all-category journals listed in the JCR.

Suggested Citation

  • Tobias Kiesslich & Marlena Beyreis & Georg Zimmermann & Andreas Traweger, 2021. "Citation inequality and the Journal Impact Factor: median, mean, (does it) matter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1249-1269, February.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:2:d:10.1007_s11192-020-03812-y
    DOI: 10.1007/s11192-020-03812-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03812-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03812-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    2. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    3. David Colquhoun, 2003. "Challenging the tyranny of impact factors," Nature, Nature, vol. 423(6939), pages 479-479, May.
    4. Pedro Albarrán & Juan A. Crespo & Ignacio Ortuño & Javier Ruiz-Castillo, 2011. "The skewness of science in 219 sub-fields and a number of aggregates," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(2), pages 385-397, August.
    5. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    6. Per O. Seglen, 1992. "The skewness of science," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 43(9), pages 628-638, October.
    7. Bornmann, Lutz & Leydesdorff, Loet, 2017. "Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data," Journal of Informetrics, Elsevier, vol. 11(1), pages 164-175.
    8. Dorothea Koelblinger & Georg Zimmermann & Silke B Weineck & Tobias Kiesslich, 2019. "Size matters! Association between journal size and longitudinal variability of the Journal Impact Factor," PLOS ONE, Public Library of Science, vol. 14(11), pages 1-9, November.
    9. Corrado Gini, 2005. "On the measurement of concentration and variability of characters," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(1), pages 1-38.
    10. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    11. Lin Zhang & Ronald Rousseau & Gunnar Sivertsen, 2017. "Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen's work on journal impact and research evaluation," PLOS ONE, Public Library of Science, vol. 12(3), pages 1-18, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2023. "Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1877-1894, March.
    2. Frode Eika Sandnes, 2021. "Everyone onboard? Participation ratios as a metric for research activity assessments within young universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6105-6113, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2023. "Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1877-1894, March.
    2. Lanu Kim & Jason H. Portenoy & Jevin D. West & Katherine W. Stovel, 2020. "Scientific journals still matter in the era of academic search engines and preprint archives," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(10), pages 1218-1226, October.
    3. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    4. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).
    5. Peter Sjögårde & Fereshteh Didegah, 2022. "The association between topic growth and citation impact of research publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1903-1921, April.
    6. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    7. Bornmann, Lutz & Leydesdorff, Loet, 2017. "Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data," Journal of Informetrics, Elsevier, vol. 11(1), pages 164-175.
    8. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    9. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    10. Bouyssou, Denis & Marchant, Thierry, 2011. "Bibliometric rankings of journals based on Impact Factors: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 5(1), pages 75-86.
    11. Siler, Kyle & Larivière, Vincent, 2022. "Who games metrics and rankings? Institutional niches and journal impact factor inflation," Research Policy, Elsevier, vol. 51(10).
    12. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Soldatenkova, Anastasiia, 2017. "An investigation on the skewness patterns and fractal nature of research productivity distributions at field and discipline level," Journal of Informetrics, Elsevier, vol. 11(1), pages 324-335.
    13. Xiomara S. Q. Chacon & Thiago C. Silva & Diego R. Amancio, 2020. "Comparing the impact of subfields in scientific journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 625-639, October.
    14. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    15. Katchanov, Yurij L. & Markova, Yulia V. & Shmatko, Natalia A., 2023. "Uncited papers in the structure of scientific communication," Journal of Informetrics, Elsevier, vol. 17(2).
    16. T. S. Evans & N. Hopkins & B. S. Kaube, 2012. "Universality of performance indicators based on citation and reference counts," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(2), pages 473-495, November.
    17. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    18. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    19. Marcel Clermont & Johanna Krolak & Dirk Tunger, 2021. "Does the citation period have any effect on the informative value of selected citation indicators in research evaluations?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1019-1047, February.
    20. Juan A Crespo & Ignacio Ortuño-Ortín & Javier Ruiz-Castillo, 2012. "The Citation Merit of Scientific Publications," PLOS ONE, Public Library of Science, vol. 7(11), pages 1-9, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:2:d:10.1007_s11192-020-03812-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.