IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v6y2012i2p169-176.html
   My bibliography  Save this article

Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor

Author

Listed:
  • Mutz, Rüdiger
  • Daniel, Hans-Dieter

Abstract

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most prominent and common measures of the prestige, position, and importance of a scientific journal. The JIF may profit from its comprehensibility, robustness, methodological reproducibility, simplicity, and rapid availability, but it is at the expense of serious technical and methodological flaws. The paper discusses two core problems with the JIF: first, citations of documents are generally not normally distributed, and, furthermore, the distribution is affected by outliers, which has serious consequences for the use of the mean value in the JIF calculation. Second, the JIF is affected by bias factors that have nothing to do with the prestige or quality of a journal (e.g., document type). For solving these two problems, we suggest using McCall's area transformation and the Rubin Causal Model. Citation data for documents of all journals in the ISI Subject Category “Psychology, Mathematical” (Journal Citation Report) are used to illustrate the proposal.

Suggested Citation

  • Mutz, Rüdiger & Daniel, Hans-Dieter, 2012. "Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor," Journal of Informetrics, Elsevier, vol. 6(2), pages 169-176.
    • Handle: RePEc:eee:infome:v:6:y:2012:i:2:p:169-176
    DOI: 10.1016/j.joi.2011.12.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157711001180
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2011.12.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Loet Leydesdorff & Tobias Opthof, 2010. "Scopus's source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(11), pages 2365-2369, November.
    2. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    3. Loet Leydesdorff & Lutz Bornmann, 2011. "How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 217-229, February.
    4. Christoph Neuhaus & Werner Marx & Hans‐Dieter Daniel, 2009. "The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the J," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 176-183, January.
    5. Loet Leydesdorff & Lutz Bornmann, 2011. "Integrated impact indicators compared with impact factors: An alternative research design with policy implications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(11), pages 2133-2146, November.
    6. Loet Leydesdorff & Jung C. Shin, 2011. "How to evaluate universities in terms of their relative citation impacts: Fractional counting of citations and the normalization of differences among disciplines," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(6), pages 1146-1155, June.
    7. Beirlant, Jan & Glänzel, Wolfgang & Carbonez, An & Leemans, Herlinde, 2007. "Scoring research output using statistical quantile plotting," Journal of Informetrics, Elsevier, vol. 1(3), pages 185-192.
    8. Lutz Bornmann & Rüdiger Mutz & Werner Marx & Hermann Schier & Hans‐Dieter Daniel, 2011. "A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(4), pages 857-879, October.
    9. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    10. Wolfgang Glänzel & András Schubert & Bart Thijs & Koenraad Debackere, 2011. "A priori vs. a posteriori normalisation of citation indicators. The case of journal ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(2), pages 415-424, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tian-Yuan Huang & Liying Yang, 2022. "Superior identification index: Quantifying the capability of academic journals to recognize good research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4023-4043, July.
    2. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    3. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    4. Zhesi Shen & Liying Yang & Zengru Di & Jinshan Wu, 2019. "Large enough sample size to rank two groups of data reliably according to their means," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 653-671, February.
    5. Tobias Kiesslich & Silke B Weineck & Dorothea Koelblinger, 2016. "Reasons for Journal Impact Factor Changes: Influence of Changing Source Items," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-12, April.
    6. Pérez-Hornero, Patricia & Arias-Nicolás, José Pablo & Pulgarín, Antonio A. & Pulgarín, Antonio, 2013. "An annual JCR impact factor calculation based on Bayesian credibility formulas," Journal of Informetrics, Elsevier, vol. 7(1), pages 1-9.
    7. Rüdiger Mutz & Hans-Dieter Daniel, 2012. "The generalized propensity score methodology for estimating unbiased journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 377-390, August.
    8. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    9. Horenberg, Frank & Lungu, Daniel Adrian & Nuti, Sabina, 2020. "Measuring research in the big data era: The evolution of performance measurement systems in the Italian teaching hospitals," Health Policy, Elsevier, vol. 124(12), pages 1387-1394.
    10. Blake M. Louscher & Veerasathpurush Allareddy & Satheesh Elangovan, 2019. "Predictors of Citations of Systematic Reviews in Oral Implantology: A Cross-Sectional Bibliometric Analysis," SAGE Open, , vol. 9(1), pages 21582440198, March.
    11. Finardi, Ugo, 2013. "Correlation between Journal Impact Factor and Citation Performance: An experimental study," Journal of Informetrics, Elsevier, vol. 7(2), pages 357-370.
    12. Pilar Valderrama & Manuel Escabias & Evaristo Jiménez-Contreras & Alberto Rodríguez-Archilla & Mariano J. Valderrama, 2018. "Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1087-1095, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Wolfgang Glänzel & Henk F. Moed, 2013. "Opinion paper: thoughts and facts on bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 381-394, July.
    3. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    4. Loet Leydesdorff & Ping Zhou & Lutz Bornmann, 2013. "How can journal impact factors be normalized across fields of science? An assessment in terms of percentile ranks and fractional counts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 96-107, January.
    5. Bornmann, Lutz & Leydesdorff, Loet, 2012. "Which are the best performing regions in information science in terms of highly cited papers? Some improvements of our previous mapping approaches," Journal of Informetrics, Elsevier, vol. 6(2), pages 336-345.
    6. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    7. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    8. M. Zitt, 2011. "Behind citing-side normalization of citations: some properties of the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 329-344, October.
    9. Ludo Waltman & Nees Jan Eck, 2013. "Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 699-716, September.
    10. Henk F. Moed, 2016. "Comprehensive indicator comparisons intelligible to non-experts: the case of two SNIP versions," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 51-65, January.
    11. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    12. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.
    13. Fiorenzo Franceschini & Maurizio Galetto & Domenico Maisano & Luca Mastrogiacomo, 2012. "The success-index: an alternative approach to the h-index for evaluating an individual’s research output," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(3), pages 621-641, September.
    14. Ludo Waltman & Erjia Yan & Nees Jan Eck, 2011. "A recursive field-normalized bibliometric performance indicator: an application to the field of library and information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 301-314, October.
    15. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    16. Loet Leydesdorff, 2012. "Alternatives to the journal impact factor: I3 and the top-10% (or top-25%?) of the most-highly cited papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 355-365, August.
    17. P. Dorta-González & M. I. Dorta-González, 2013. "Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 645-672, May.
    18. Tolga Yuret, 2018. "Author-weighted impact factor and reference return ratio: can we attain more equality among fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2097-2111, September.
    19. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    20. Pilar Valderrama & Manuel Escabias & Evaristo Jiménez-Contreras & Alberto Rodríguez-Archilla & Mariano J. Valderrama, 2018. "Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1087-1095, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:6:y:2012:i:2:p:169-176. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.