IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v122y2020i3d10.1007_s11192-019-03338-y.html
   My bibliography  Save this article

Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor

Author

Listed:
  • Lei Lei

    (Huazhong University of Science and Technology)

  • Yunmei Sun

    (Huazhong University of Science and Technology)

Abstract

The use of the journal impact factor for evaluation purposes has been widely challenged because of the simplistic algorithm involved in impact factor calculation. As a result, a journal's impact factor may be affected by many variables, such as highly cited review articles. In the present study, we aimed to investigate whether and to what extent review articles affect the impact factors of journals. We examined the citations of original research articles and review articles in the field of physics, and compared the impact factors of the journals Science, Nature, and Cell with and without citation counts of review articles included. The results showed that review articles consistently received higher numbers of citations than did original research articles. In addition, review articles accounted for 3–10% of the impact factors of these top journals. Based on our findings, we suggest that sophisticated methods, in addition to the impact factor, be employed for research evaluation purposes.

Suggested Citation

  • Lei Lei & Yunmei Sun, 2020. "Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1697-1706, March.
    • Handle: RePEc:spr:scient:v:122:y:2020:i:3:d:10.1007_s11192-019-03338-y
    DOI: 10.1007/s11192-019-03338-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-019-03338-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-019-03338-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Erwin Krauskopf, 2013. "Deceiving the research community through manipulation of the impact factor," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(11), pages 2403-2403, November.
    2. Erwin Krauskopf, 2013. "Deceiving the research community through manipulation of the impact factor," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(11), pages 2403-2403, November.
    3. Paula Stephan & Reinhilde Veugelers & Jian Wang, 2017. "Reviewers are blinkered by bibliometrics," Nature, Nature, vol. 544(7651), pages 411-412, April.
    4. Dong-Hui Yang & Xin Li & Xiaoxia Sun & Jie Wan, 2016. "Detecting impact factor manipulation with data mining techniques," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1989-2005, December.
    5. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    6. Bornmann, Lutz & Marx, Werner, 2018. "Critical rationalism and the search for standard (field-normalized) indicators in bibliometrics," Journal of Informetrics, Elsevier, vol. 12(3), pages 598-604.
    7. Guang Yu & Dong-Hui Yang & Wang Liang, 2010. "Reliability-based citation impact factor and the manipulation of impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(1), pages 259-270, April.
    8. Dongbo Shi & Ronald Rousseau & Liu Yang & Jiang Li, 2017. "A journal's impact factor is influenced by changes in publication delays of citing journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(3), pages 780-789, March.
    9. Lei Lei & Sheng Yan, 2016. "Readability and citations in information science: evidence from abstracts and articles of four journals (2003–2012)," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1155-1169, September.
    10. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    11. Weishu Liu & Guangyuan Hu & Mengdi Gu, 2016. "The probability of publishing in first-quartile journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 1273-1276, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tian-Yuan Huang & Liying Yang, 2022. "Superior identification index: Quantifying the capability of academic journals to recognize good research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4023-4043, July.
    2. Yves Fassin, 2021. "The impact of review articles in management and economics journal rankings and metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9623-9632, December.
    3. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    4. Salim Moussa, 2021. "Are FT50 journals really leading? A comment on Fassin," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9613-9622, December.
    5. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    6. Guofeng Wang & Rui Shi & Wei Cheng & Lihua Gao & Xiankai Huang, 2023. "Bibliometric Analysis for Carbon Neutrality with Hotspots, Frontiers, and Emerging Trends between 1991 and 2022," IJERPH, MDPI, vol. 20(2), pages 1-23, January.
    7. Sepideh Fahimifar & Khadijeh Mousavi & Fatemeh Mozaffari & Marcel Ausloos, 2023. "Identification of the most important external features of highly cited scholarly papers through 3 (i.e., Ridge, Lasso, and Boruta) feature selection data mining methods," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3685-3712, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Julian N. Marewski, 2019. "Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 419-459, August.
    2. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    3. Gregorio González-Alcaide, 2021. "Bibliometric studies outside the information science and library science field: uncontainable or uncontrollable?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6837-6870, August.
    4. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    5. Juan Miguel Campanario, 2018. "Are leaders really leading? Journals that are first in Web of Science subject categories in the context of their groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 111-130, April.
    6. Daniela De Filippo & Fernanda Morillo & Borja González-Albo, 2023. "Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences," Publications, MDPI, vol. 11(2), pages 1-17, June.
    7. Рубинштейн Александр Яковлевич, "undated". "Ранжирование Российских Экономических Журналов: Научный Метод Или «Игра В Цыфирь»? [Ran Ranking of Russian Economic Journals: The Scientific Method or “Numbers Game”?]," Working papers a:pru175:ye:2016:1, Institute of Economics.
    8. Schmal, W. Benedikt & Haucap, Justus & Knoke, Leon, 2023. "The role of gender and coauthors in academic publication behavior," Research Policy, Elsevier, vol. 52(10).
    9. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2023. "Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1877-1894, March.
    10. Eugenio Petrovich, 2022. "Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2195-2233, May.
    11. William Cabos & Juan Miguel Campanario, 2018. "Exploring the Hjif-Index, an Analogue to the H-Like Index for Journal Impact Factors," Publications, MDPI, vol. 6(2), pages 1-11, April.
    12. Lanu Kim & Jason H. Portenoy & Jevin D. West & Katherine W. Stovel, 2020. "Scientific journals still matter in the era of academic search engines and preprint archives," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(10), pages 1218-1226, October.
    13. Sergio Copiello, 2019. "Research Interest: another undisclosed (and redundant) algorithm by ResearchGate," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 351-360, July.
    14. Hu, Guangyuan & Ni, Rong & Tang, Li, 2022. "Do international nonstop flights foster influential research? Evidence from Sino-US scientific collaboration," Journal of Informetrics, Elsevier, vol. 16(4).
    15. Khosrowjerdi, Mahmood & Bornmann, Lutz, 2021. "Is culture related to strong science? An empirical investigation," Journal of Informetrics, Elsevier, vol. 15(4).
    16. Torres-Salinas, Daniel & Valderrama-Baca, Pilar & Arroyo-Machado, Wenceslao, 2022. "Is there a need for a new journal metric? Correlations between JCR Impact Factor metrics and the Journal Citation Indicator—JCI," Journal of Informetrics, Elsevier, vol. 16(3).
    17. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    18. Juan Miguel Campanario, 2018. "Journals that Rise from the Fourth Quartile to the First Quartile in Six Years or Less: Mechanisms of Change and the Role of Journal Self-Citations," Publications, MDPI, vol. 6(4), pages 1-15, November.
    19. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    20. Xu, Shuqi & Mariani, Manuel Sebastian & Lü, Linyuan & Medo, Matúš, 2020. "Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data," Journal of Informetrics, Elsevier, vol. 14(1).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:122:y:2020:i:3:d:10.1007_s11192-019-03338-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.