IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v9y2015i4p990-1006.html
   My bibliography  Save this article

On the stability of citation-based journal rankings

Author

Listed:
  • Pajić, Dejan

Abstract

The article presents a large-scale comparison of journal rankings based on seven impact measures: Impact Factor (2- and 5-year), SJR, IPP, SNIP, H index, and Article Influence Score. Three aspects of ranking stability in the 2007–2014 period were analyzed: temporal, cross-discipline, and cross-indicator. Impact measures based on five-year citation windows enable more stable journal rankings over time. Journal rankings based on the source-normalized indicator (SNIP) have the largest cross-discipline stability. Journals in the fields of social sciences and humanities have lower temporal and cross-discipline ranking stability compared to those in “hard” sciences. Although correlation coefficients indicate relatively high agreement among the rankings based on different indicators, variations in quartile and percentile ranks suggest different conclusions. WoS journals almost linearly improve their ranking positions in Scopus lists, while many high-impact journals covered by Scopus are not available in WoS. An important element of the ranking stability is the discriminability of impact measures. Beyond the segregation between the top and bottom ranked journals, our assessment of “quality” relies in most cases on a rather arguable assumption that a couple of citations more or less is making a big difference.

Suggested Citation

  • Pajić, Dejan, 2015. "On the stability of citation-based journal rankings," Journal of Informetrics, Elsevier, vol. 9(4), pages 990-1006.
    • Handle: RePEc:eee:infome:v:9:y:2015:i:4:p:990-1006
    DOI: 10.1016/j.joi.2015.08.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S175115771530033X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2015.08.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rowlinson, Michael & Harvey, Charles & Kelly, Aidan & Morris, Huw & Todeva, Emanuela, 2015. "Accounting for research quality: Research audits and the journal rankings debate," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 26(C), pages 2-22.
    2. Ju-fang Shao & Hui-yun Shen, 2012. "Research assessment and monetary rewards: the overemphasized impact factor in China," Research Evaluation, Oxford University Press, vol. 21(3), pages 199-203, June.
    3. Vincent Larivière & Éric Archambault & Yves Gingras & Étienne Vignola‐Gagné, 2006. "The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(8), pages 997-1004, June.
    4. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    5. Anton J. Nederhof, 2006. "Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review," Scientometrics, Springer;Akadémiai Kiadó, vol. 66(1), pages 81-100, January.
    6. Dalibor Fiala, 2013. "Science Evaluation in the Czech Republic: The Case of Universities," Societies, MDPI, vol. 3(3), pages 1-14, June.
    7. Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2012. "A further step forward in measuring journals’ scientific prestige: The SJR2 indicator," Journal of Informetrics, Elsevier, vol. 6(4), pages 674-688.
    8. Kevin W. Boyack & Richard Klavans, 2010. "Co‐citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(12), pages 2389-2404, December.
    9. Sutter, Matthias & Kocher, Martin G., 2001. "Tools for evaluating research output: Are citation-base rankings of economics journals stable?," Munich Reprints in Economics 18222, University of Munich, Department of Economics.
    10. Philip M. Davis, 2008. "Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(13), pages 2186-2188, November.
    11. Kevin W. Boyack & Richard Klavans, 2010. "Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(12), pages 2389-2404, December.
    12. Vanclay, Jerome K., 2011. "An evaluation of the Australian Research Council's journal ranking," Journal of Informetrics, Elsevier, vol. 5(2), pages 265-274.
    13. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    14. Juan Miguel Campanario, 2011. "Large increases and decreases in journal impact factors in only one year: The effect of journal self-citations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(2), pages 230-235, February.
    15. David Pontille & Didier Torny, 2010. "The controversial policies of journal ratings: evaluating social sciences and humanities," Research Evaluation, Oxford University Press, vol. 19(5), pages 347-360, December.
    16. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    17. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    18. Christopher Barrett & Aliakbar Olia & Dee Von Bailey, 2000. "Subdiscipline-specific journal rankings: whither Applied Economics?," Applied Economics, Taylor & Francis Journals, vol. 32(2), pages 239-252.
    19. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    20. Vivien Beattie & Alan Goodacre, 2006. "A new method for ranking academic journals in accounting and finance," Accounting and Business Research, Taylor & Francis Journals, vol. 36(2), pages 65-91.
    21. Matthias Sutter & Martin G. Kocher, 2001. "Tools for Evaluating Research Output," Evaluation Review, , vol. 25(5), pages 555-566, October.
    22. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    23. Franceschet, Massimo, 2010. "Journal influence factors," Journal of Informetrics, Elsevier, vol. 4(3), pages 239-248.
    24. Juan Miguel Campanario, 2011. "Large increases and decreases in journal impact factors in only one year: The effect of journal self‐citations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 230-235, February.
    25. Juan Miguel Campanario, 2011. "Empirical study of journal impact factors obtained using the classical two-year citation window versus a five-year citation window," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(1), pages 189-204, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Péter Sasvári & András Nemeslaki & László Duma, 2019. "Exploring the influence of scientific journal ranking on publication performance in the Hungarian social sciences: the case of law and economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 595-616, May.
    2. Zhang, Fang & Wu, Shengli, 2020. "Predicting future influence of papers, researchers, and venues in a dynamic academic network," Journal of Informetrics, Elsevier, vol. 14(2).
    3. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    4. Aslıhan Sezgin & Keziban Orbay & Metin Orbay, 2022. "Educational Research Review From Diverse Perspectives: A Bibliometric Analysis of Web of Science (2011–2020)," SAGE Open, , vol. 12(4), pages 21582440221, December.
    5. Fang Zhang & Shengli Wu, 2021. "Measuring academic entities’ impact by content-based citation analysis in a heterogeneous academic network," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7197-7222, August.
    6. Lin Feng & Jian Zhou & Sheng-Lan Liu & Ning Cai & Jie Yang, 2020. "Analysis of journal evaluation indicators: an experimental study based on unsupervised Laplacian score," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 233-254, July.
    7. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    8. Dejian Yu & Wanru Wang & Shuai Zhang & Wenyu Zhang & Rongyu Liu, 2017. "A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 521-542, April.
    9. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    10. Juan Miguel Campanario, 2018. "Are leaders really leading? Journals that are first in Web of Science subject categories in the context of their groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 111-130, April.
    11. Shannon Mason & Lenandlar Singh, 2022. "When a journal is both at the ‘top’ and the ‘bottom’: the illogicality of conflating citation-based metrics with quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3683-3694, June.
    12. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    13. Gabriel-Alexandru Vȋiu & Mihai Păunescu, 2021. "The lack of meaningful boundary differences between journal impact factor quartiles undermines their independent use in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1495-1525, February.
    14. Liu, Xuan Zhen & Fang, Hui, 2020. "A comparison among citation-based journal indicators and their relative changes with time," Journal of Informetrics, Elsevier, vol. 14(1).
    15. Lina Xu & Steven Dellaportas & Zhiqiang Yang & Jin Wang, 2023. "More on the relationship between interdisciplinary accounting research and citation impact," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 63(4), pages 4779-4803, December.
    16. A. Ferrer-Sapena & J. M. Calabuig & L. M. García Raffi & E. A. Sánchez Pérez, 2020. "Where Should I Submit My Work for Publication? An Asymmetrical Classification Model to Optimize Choice," Journal of Classification, Springer;The Classification Society, vol. 37(2), pages 490-508, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    2. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    3. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    4. Yi-Ching Liaw & Te-Yi Chan & Chin-Yuan Fan & Cheng-Hsin Chiang, 2014. "Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 17-37, October.
    5. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    6. Walters, William H., 2014. "Do Article Influence scores overestimate the citation impact of social science journals in subfields that are related to higher-impact natural science disciplines?," Journal of Informetrics, Elsevier, vol. 8(2), pages 421-430.
    7. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    8. Saarela, Mirka & Kärkkäinen, Tommi & Lahtonen, Tommi & Rossi, Tuomo, 2016. "Expert-based versus citation-based ranking of scholarly and scientific publication channels," Journal of Informetrics, Elsevier, vol. 10(3), pages 693-718.
    9. Loet Leydesdorff & Paul Wouters & Lutz Bornmann, 2016. "Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2129-2150, December.
    10. Dejian Yu & Wanru Wang & Shuai Zhang & Wenyu Zhang & Rongyu Liu, 2017. "A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 521-542, April.
    11. Zhang, Fang & Wu, Shengli, 2020. "Predicting future influence of papers, researchers, and venues in a dynamic academic network," Journal of Informetrics, Elsevier, vol. 14(2).
    12. Cristina López-Duarte & Marta M. Vidal-Suárez & Belén González-Díaz & Nuno Rosa Reis, 2016. "Understanding the relevance of national culture in international business research: a quantitative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1553-1590, September.
    13. Dzieżyc, Maciej & Kazienko, Przemysław, 2022. "Effectiveness of research grants funded by European Research Council and Polish National Science Centre," Journal of Informetrics, Elsevier, vol. 16(1).
    14. Fang Zhang & Shengli Wu, 2021. "Measuring academic entities’ impact by content-based citation analysis in a heterogeneous academic network," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7197-7222, August.
    15. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    16. Itay Mayrose & Shiri Freilich, 2015. "The Interplay between Scientific Overlap and Cooperation and the Resulting Gain in Co-Authorship Interactions," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-10, September.
    17. Aslıhan Sezgin & Keziban Orbay & Metin Orbay, 2022. "Educational Research Review From Diverse Perspectives: A Bibliometric Analysis of Web of Science (2011–2020)," SAGE Open, , vol. 12(4), pages 21582440221, December.
    18. Henk F. Moed, 2016. "Comprehensive indicator comparisons intelligible to non-experts: the case of two SNIP versions," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 51-65, January.
    19. Peter Sjögårde & Fereshteh Didegah, 2022. "The association between topic growth and citation impact of research publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1903-1921, April.
    20. Sándor Soós & Zsófia Vida & András Schubert, 2018. "Long-term trends in the multidisciplinarity of some typical natural and social sciences, and its implications on the SSH versus STM distinction," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 795-822, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:9:y:2015:i:4:p:990-1006. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.