IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v129y2024i3d10.1007_s11192-023-04888-y.html
   My bibliography  Save this article

Ranking journals by voting with feet: a new method for journal evaluation

Author

Listed:
  • Kun Chen

    (Xinjiang University
    Xinjiang University)

  • Xu Liu

    (Xinjiang University
    Xinjiang University)

  • Abduhalik Wupur

    (Xinjiang University
    Xinjiang University)

  • Guo-liang Yang

    (Chinese Academy of Sciences
    University of Chinese Academy of Sciences
    Beijing Technology and Business University)

Abstract

This paper introduces the Voting with Feet Index (VFI), a novel method for evaluating the quality of academic journals. Unlike previous approaches that rely on citation analysis or expert surveys, which have limitations such as not indicating paper quality or being time-consuming and expensive, the VFI argues that a journal's quality should be determined by the quality of the papers it publishes. This is measured through proxies such as awards, author reputation, and institutional affiliation, as assessing each paper individually is challenging. The study also examines the relative importance of these proxies and their potential combination. To rank journals, the VFI analyzes the distribution of papers that have received important awards, papers authored by notable scholars, and papers affiliated with distinguished institutions. A score is then calculated for each journal. The VFI was applied to 40 MS & OR (Management Science & Operations Research) journals and compared with impact factor and expert survey rankings. The results demonstrate a strong correlation between the VFI and these rankings, particularly for high-level journals. Furthermore, the rankings of selected journals in physics and chemistry were determined using Nobel Prize data. This analysis not only confirmed the accuracy of the VFI in identifying top journals but also highlighted important considerations for the indicator's construction. Overall, the VFI presents a promising new approach to assess journal quality, addressing the limitations of existing methods.

Suggested Citation

  • Kun Chen & Xu Liu & Abduhalik Wupur & Guo-liang Yang, 2024. "Ranking journals by voting with feet: a new method for journal evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(3), pages 1567-1588, March.
    • Handle: RePEc:spr:scient:v:129:y:2024:i:3:d:10.1007_s11192-023-04888-y
    DOI: 10.1007/s11192-023-04888-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-023-04888-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-023-04888-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    2. Morris G. Danielson & Jean L. Heck, 2014. "Voting with Their Feet: In Which Journals Do the Most Prolific Finance Researchers Publish?," Financial Management, Financial Management Association International, vol. 43(1), pages 1-27, March.
    3. Juan A. Aledo & Jose A. Gámez & David Molina & Alejandro Rosete, 2018. "Consensus†based journal rankings: A complementary tool for bibliometric evaluation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(7), pages 936-948, July.
    4. Matías Ginieis & Xiaoni Li, 2020. "Ranking of Sustainability Journals Using the Author Affiliation Index and Comparison to Other Journal Metrics," Sustainability, MDPI, vol. 12(3), pages 1-18, February.
    5. Dorta-González, Pablo & Dorta-González, María Isabel & Santos-Peñate, Dolores Rosa & Suárez-Vega, Rafael, 2014. "Journal topic citation potential and between-field comparisons: The topic normalized impact factor," Journal of Informetrics, Elsevier, vol. 8(2), pages 406-418.
    6. Timothy D. Fry & Joan M. Donohue, 2014. "Exploring the Author Affiliation Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1647-1667, March.
    7. Blaise Cronin & Lokman I. Meho, 2008. "Applying the author affiliation index to library and information science journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(11), pages 1861-1865, September.
    8. Stewart Manley, 2022. "Letters to the editor and the race for publication metrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(5), pages 702-707, May.
    9. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    10. Chen, Carl R. & Huang, Ying, 2007. "Author Affiliation Index, finance journal ranking, and the pattern of authorship," Journal of Corporate Finance, Elsevier, vol. 13(5), pages 1008-1026, December.
    11. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    12. Wang, Xing & Zhang, Zhihui, 2020. "Improving the reliability of short-term citation impact indicators by taking into account the correlation between short- and long-term citation impact," Journal of Informetrics, Elsevier, vol. 14(2).
    13. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    14. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    15. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    16. Ehsan Mohammadi & Mike Thelwall & Stefanie Haustein & Vincent Larivière, 2015. "Who reads research articles? An altmetrics analysis of Mendeley user categories," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(9), pages 1832-1846, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Currie, Russell R. & Pandher, Gurupdesh S., 2020. "Finance journal rankings: Active scholar assessment revisited," Journal of Banking & Finance, Elsevier, vol. 111(C).
    2. Yan-Hong Yang & Ying-Hui Shao, 2024. "Education journal rankings: a diversity-based Author Affiliation Index assessment methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(5), pages 2677-2700, May.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    5. Chen, Meiqian & Guo, Zhaoxia & Dong, Yucheng & Chiclana, Francisco & Herrera-Viedma, Enrique, 2021. "Citations optimal growth path: A tool to analyze sensitivity to citations of h-like indexes," Journal of Informetrics, Elsevier, vol. 15(4).
    6. Ziqiang Zeng & Lantian Shi, 2021. "A two-dimensional journal classification method based on output and input factors: perspectives from citation and authorship related indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 3929-3964, May.
    7. Matías Ginieis & Xiaoni Li, 2020. "Ranking of Sustainability Journals Using the Author Affiliation Index and Comparison to Other Journal Metrics," Sustainability, MDPI, vol. 12(3), pages 1-18, February.
    8. Yves Fassin, 2021. "Does the Financial Times FT50 journal list select the best management and economics journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5911-5943, July.
    9. Thor, Andreas & Marx, Werner & Leydesdorff, Loet & Bornmann, Lutz, 2016. "Introducing CitedReferencesExplorer (CRExplorer): A program for reference publication year spectroscopy with cited references standardization," Journal of Informetrics, Elsevier, vol. 10(2), pages 503-515.
    10. Mingyang Wang & Zhenyu Wang & Guangsheng Chen, 2019. "Which can better predict the future success of articles? Bibliometric indices or alternative metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1575-1595, June.
    11. Christopher Zou & Jordan B. Peterson, 2016. "Quantifying the scientific output of new researchers using the zp-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 901-916, March.
    12. Wang, Jiang-Pan & Guo, Qiang & Zhou, Lei & Liu, Jian-Guo, 2019. "Dynamic credit allocation for researchers," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 520(C), pages 208-216.
    13. Ortega, José Luis, 2018. "The life cycle of altmetric impact: A longitudinal study of six metrics from PlumX," Journal of Informetrics, Elsevier, vol. 12(3), pages 579-589.
    14. Yurij L. Katchanov & Yulia V. Markova, 2017. "The “space of physics journals”: topological structure and the Journal Impact Factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 313-333, October.
    15. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    16. Liwei Zhang & Jue Wang, 2021. "What affects publications’ popularity on Twitter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9185-9198, November.
    17. Zhang, Fang & Wu, Shengli, 2020. "Predicting future influence of papers, researchers, and venues in a dynamic academic network," Journal of Informetrics, Elsevier, vol. 14(2).
    18. John Mingers & Jesse R. O’Hanley & Musbaudeen Okunola, 2017. "Using Google Scholar institutional level data to evaluate the quality of university research," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1627-1643, December.
    19. Laura Vana & Ronald Hochreiter & Kurt Hornik, 2016. "Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 229-251, January.
    20. Alhenawi, Yasser & Hassan, M. Kabir & Hasan, Rashedul, 2022. "Evolution of research in finance over the last two decades – A topographical view," Research in International Business and Finance, Elsevier, vol. 59(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:129:y:2024:i:3:d:10.1007_s11192-023-04888-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.