IDEAS home Printed from https://ideas.repec.org/a/pal/palcom/v12y2025i1d10.1057_s41599-025-04681-7.html
   My bibliography  Save this article

Addressing the challenges in journal evaluation during the “covidization” of scientific research era : insights from the CAS journal ranking

Author

Listed:
  • Yahui Liu

    (General Administration of Customs of the People’s Republic of China
    Chinese Academy of Sciences)

  • Liying Yang

    (Chinese Academy of Sciences
    University of Chinese Academy of Sciences)

  • Jiandong Zhang

    (Chinese Academy of Sciences
    University of Chinese Academy of Sciences)

  • Zhesi Shen

    (Chinese Academy of Sciences
    University of Chinese Academy of Sciences)

Abstract

Journal impact measures are expected to demonstrate temporal stability, as journal quality typically remains consistent year-over-year. However, the COVID-19 pandemic has disrupted journal evaluation systems through an unprecedented surge in highly-cited publications. While conventional metrics like the Journal Impact Factor have shown high sensitivity to these citation patterns, the CAS Journal Ranking (CJR) system has demonstrated notable stability. This study compares three metrics—JIF, Category Normalized Citation Impact, and Field Normalized Citation Success Index (FNCSI, indicator used in CJR)—under different classification systems. We find that combining FNCSI with the CWTS paper-level classification system provides superior ranking stability. This robustness stems from FNCSI’s ability to moderate the influence of highly cited papers and paper-level system’s effective clustering of COVID-19 research.

Suggested Citation

  • Yahui Liu & Liying Yang & Jiandong Zhang & Zhesi Shen, 2025. "Addressing the challenges in journal evaluation during the “covidization” of scientific research era : insights from the CAS journal ranking," Palgrave Communications, Palgrave Macmillan, vol. 12(1), pages 1-11, December.
    • Handle: RePEc:pal:palcom:v:12:y:2025:i:1:d:10.1057_s41599-025-04681-7
    DOI: 10.1057/s41599-025-04681-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41599-025-04681-7
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1057/s41599-025-04681-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    2. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    3. Eoghan Cunningham & Barry Smyth & Derek Greene, 2021. "Collaboration in the time of COVID: a scientometric analysis of multidisciplinary SARS-CoV-2 research," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-8, December.
    4. Constantin Bürgi & Klaus Wohlrabe, 2022. "The influence of Covid-19 on publications in economics: bibliometric evidence from five working paper series," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5175-5189, September.
    5. Eoghan Cunningham & Barry Smyth & Derek Greene, 2021. "Correction: Collaboration in the time of COVID: a scientometric analysis of multidisciplinary SARS-CoV-2 research," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-1, December.
    6. Yves Fassin, 2021. "Research on Covid-19: a disruptive phenomenon for bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5305-5319, June.
    7. Emanuel Kulczycki & Ying Huang & Alesia A. Zuccala & Tim C. E. Engels & Antonio Ferrara & Raf Guns & Janne Pölönen & Gunnar Sivertsen & Zehra Taşkın & Lin Zhang, 2022. "Uses of the Journal Impact Factor in national journal rankings in China and Europe," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(12), pages 1741-1754, December.
    8. X. Cai & C. V. Fry & C. S. Wagner, 2021. "International collaboration during the COVID-19 crisis: autumn 2020 developments," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3683-3692, April.
    9. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2022. "How the Covid-19 crisis shaped research collaboration behaviour," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 5053-5071, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rubini, Lauretta & Pollio, Chiara & Barbieri, Elisa & Cattaruzzo, Sebastiano, 2024. "Changing structures in transnational research networks: An analysis of the impact of COVID-19 on China's scientific collaborations," Structural Change and Economic Dynamics, Elsevier, vol. 69(C), pages 281-297.
    2. Danilo Silva Carvalho & Lucas Lopes Felipe & Priscila Costa Albuquerque & Fabio Zicker & Bruna de Paula Fonseca, 2023. "Leadership and international collaboration on COVID-19 research: reducing the North–South divide?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4689-4705, August.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2022. "How the Covid-19 crisis shaped research collaboration behaviour," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 5053-5071, August.
    4. Rousseau, Ronald & Garcia-Zorita, Carlos & Sanz-Casado, Elías, 2023. "Publications during COVID-19 times: An unexpected overall increase," Journal of Informetrics, Elsevier, vol. 17(4).
    5. Chen, Guo & Hong, Siqi & Du, Chenxin & Wang, Panting & Yang, Zeyu & Xiao, Lu, 2024. "Comparing semantic representation methods for keyword analysis in bibliometric research," Journal of Informetrics, Elsevier, vol. 18(3).
    6. Breno Santana Santos & Ivanovitch Silva & Luciana Lima & Patricia Takako Endo & Gisliany Alves & Marcel da Câmara Ribeiro-Dantas, 2022. "Discovering temporal scientometric knowledge in COVID-19 scholarly production," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1609-1642, March.
    7. Török, Ádám & Konka, Boglárka & Nagy, Andrea Magda, 2023. "A koronavírus-járvány a közgazdasági szakirodalomban. Egy új határterület tudománymetriai elemzése [The coronavirus pandemic in the economics literature. The scientometric analysis of a new discipl," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(3), pages 284-304.
    8. Tong Li & Yanfen Wang & Lizhen Cui & Ranjay K. Singh & Hongdou Liu & Xiufang Song & Zhihong Xu & Xiaoyong Cui, 2023. "Exploring the evolving landscape of COVID-19 interfaced with livelihoods," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-12, December.
    9. Yujie Zhang & Hongzhen Li & Jingyi Mao & Guoxiu He & Yunhan Yang & Zhuoren Jiang & Yufeng Duan, 2023. "COVID-19: a disruptive impact on the knowledge support of references," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4791-4823, August.
    10. Keith Head & Yao Amber Li & Asier Minondo, 2019. "Geography, Ties, and Knowledge Flows: Evidence from Citations in Mathematics," The Review of Economics and Statistics, MIT Press, vol. 101(4), pages 713-727, October.
    11. Ana María Gómez-Aguayo & Joaquín M. Azagra-Caro & Carlos Benito-Amat, 2024. "The steady effect of knowledge co-creation with universities on business scientific impact throughout the economic cycle," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(5), pages 2771-2799, May.
    12. Shahd Al-Janabi & Lee Wei Lim & Luca Aquili, 2021. "Development of a tool to accurately predict UK REF funding allocation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8049-8062, September.
    13. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    14. Dorta-González, P. & Dorta-González, M.I., 2013. "Impact maturity times and citation time windows: The 2-year maximum journal impact factor," Journal of Informetrics, Elsevier, vol. 7(3), pages 593-602.
    15. Constantin Bürgi & Klaus Wohlrabe, 2022. "The influence of Covid-19 on publications in economics: bibliometric evidence from five working paper series," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5175-5189, September.
    16. Huang, Mu-Hsuan & Huang, Wei-Tzu & Chen, Dar-Zen, 2014. "Technological impact factor: An indicator to measure the impact of academic publications on practical innovation," Journal of Informetrics, Elsevier, vol. 8(1), pages 241-251.
    17. Walters, William H., 2014. "Do Article Influence scores overestimate the citation impact of social science journals in subfields that are related to higher-impact natural science disciplines?," Journal of Informetrics, Elsevier, vol. 8(2), pages 421-430.
    18. Tolga Yuret, 2023. "The citation performance of the references in the standard graduate-level microeconomics textbook: Mas-Collel et al. (1995)," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1473-1484, March.
    19. Tobias Kiesslich & Silke B Weineck & Dorothea Koelblinger, 2016. "Reasons for Journal Impact Factor Changes: Influence of Changing Source Items," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-12, April.
    20. Xiaojun Hu & Ronald Rousseau & Jin Chen, 2012. "Structural indicators in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 451-460, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:palcom:v:12:y:2025:i:1:d:10.1057_s41599-025-04681-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: https://www.nature.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.