IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v11y2017i3p730-744.html
   My bibliography  Save this article

Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?

Author

Listed:
  • Walters, William H.

Abstract

This study uses journal ratings in criminology and criminal justice, library and information science, public administration, and social work to investigate two research questions: (1) Are stated preference (subjective) journal ratings more closely related to size-dependent citation metrics (eigenfactor and total citations, which represent the impact of the journal as a whole) or to size-independent citation metrics (article influence and CiteScore, which represent the impact of a typical article)? (2) Are stated preference ratings more closely related to unweighted citation metrics (five-year impact factor and source normalized impact per publication, which do not account for the impact of each citing journal) or to weighted citation metrics (article influence and SCImago journal rank, which do)? Within the disciplines evaluated here, respondents’ subjective ratings of journals are more closely related to size-independent metrics and weighted metrics. The relative strength of the relationship between subjective ratings and size-independent metrics is moderated by subject area and other factors, while the relative strength of the relationship between subjective ratings and weighted metrics is consistent across all four disciplines. These results are discussed with regard to popularity and prestige, which are sometimes associated with unweighted and weighted citation metrics, respectively.

Suggested Citation

  • Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    • Handle: RePEc:eee:infome:v:11:y:2017:i:3:p:730-744
    DOI: 10.1016/j.joi.2017.05.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157717300925
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2017.05.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Massimo Franceschet, 2009. "A cluster analysis of scholar and journal bibliometric indicators," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(10), pages 1950-1964, October.
    2. Gunther Maier, 2006. "Impact factors and peer judgment: The case of regional science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 651-667, December.
    3. Christian Schloegl & Wolfgang G. Stock, 2004. "Impact and relevance of LIS journals: A scientometric analysis of international and German‐language LIS journals—Citation analysis versus reader survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 55(13), pages 1155-1168, November.
    4. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    5. Frederic S. Lee & Bruce C. Cronin & Scott McConnell & Erik Dean, 2010. "Research Quality Rankings of Heterodox Economic Journals in a Contested Discipline," American Journal of Economics and Sociology, Wiley Blackwell, vol. 69(5), pages 1409-1452, November.
    6. A. Abrizah & A. N. Zainab & K. Kiran & R. G. Raj, 2013. "LIS journals scientific impact and subject categorization: a comparison between Web of Science and Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 721-740, February.
    7. Chaoqun Ni & Cassidy R. Sugimoto & Blaise Cronin, 2013. "Visualizing and comparing four facets of scholarly communication: producers, artifacts, concepts, and gatekeepers," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1161-1173, March.
    8. Birk Diedenhofen & Jochen Musch, 2015. "cocor: A Comprehensive Solution for the Statistical Comparison of Correlations," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-12, April.
    9. E. Susanna Cahn, 2014. "Journal Rankings: Comparing Reputation, Citation and Acceptance Rates," International Journal of Information Systems in the Service Sector (IJISSS), IGI Global, vol. 6(4), pages 92-103, October.
    10. Anthony J. Nederhof & Marc Luwel & Henk F. Moed, 2001. "Assessing the quality of scholarly journals in Linguistics:An alternative to citation-based journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 241-265, April.
    11. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    12. Anton J. Nederhof, 2006. "Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review," Scientometrics, Springer;Akadémiai Kiadó, vol. 66(1), pages 81-100, January.
    13. Erjia Yan & Ying Ding, 2010. "Weighted citation: An indicator of an article's prestige," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(8), pages 1635-1643, August.
    14. Cheng Zhang & Xu Liu & Yunjie (Calvin) Xu & Youwei Wang, 2011. "Quality-structure index: A new metric to measure scientific journal influence," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(4), pages 643-653, April.
    15. Johan Bollen & Herbert Van de Sompel, 2008. "Usage impact factor: The effects of sample characteristics on usage‐based impact metrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(1), pages 136-149, January.
    16. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    17. Rosenstreich, Daniela & Wooliscroft, Ben, 2009. "Measuring the impact of accounting journals using Google Scholar and the g-index," The British Accounting Review, Elsevier, vol. 41(4), pages 227-239.
    18. Serenko, Alexander & Bontis, Nick, 2011. "What's familiar is excellent: The impact of exposure effect on perceived journal quality," Journal of Informetrics, Elsevier, vol. 5(1), pages 219-223.
    19. Josephine E. Olson, 2005. "Top-25-Business-School Professors Rate Journals in Operations Management and Related Fields," Interfaces, INFORMS, vol. 35(4), pages 323-338, August.
    20. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Ryan Chute, 2009. "A Principal Component Analysis of 39 Scientific Impact Measures," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-11, June.
    21. Michael F. Gorman & John J. Kanet, 2005. "Evaluating Operations Management--Related Journals via the Author Affiliation Index," Manufacturing & Service Operations Management, INFORMS, vol. 7(1), pages 3-19, February.
    22. Christian Schloegl & Juan Gorraiz, 2010. "Comparison of citation and usage indicators: the case of oncology journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(3), pages 567-580, March.
    23. Tsai, Chih-Fong, 2014. "Citation impact analysis of top ranked computer science journals and their rankings," Journal of Informetrics, Elsevier, vol. 8(2), pages 318-328.
    24. Franceschet, Massimo, 2010. "The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis," Journal of Informetrics, Elsevier, vol. 4(1), pages 55-63.
    25. Shilbury, David & Rentschler, Ruth, 2007. "Assessing Sport Management Journals: A Multi-Dimensional Examination," Sport Management Review, Elsevier, vol. 10(1), pages 31-44, May.
    26. Donohue, Joan M. & Fox, Jeremy B., 2000. "A multi-method evaluation of journals in the decision and management sciences by US academics," Omega, Elsevier, vol. 28(1), pages 17-36, February.
    27. Erjia Yan & Ying Ding & Cassidy R. Sugimoto, 2011. "P-Rank: An indicator measuring prestige in heterogeneous scholarly networks," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(3), pages 467-477, March.
    28. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    29. Ellis, Larry V. & Durden, Garey C., 1991. "Why economists rank their journals the way they do," Journal of Economics and Business, Elsevier, vol. 43(3), pages 265-270, August.
    30. Ludo Waltman & Nees Jan van Eck, 2012. "The inconsistency of the h-index," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(2), pages 406-415, February.
    31. Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2012. "A further step forward in measuring journals’ scientific prestige: The SJR2 indicator," Journal of Informetrics, Elsevier, vol. 6(4), pages 674-688.
    32. Sorensen, Jon R., 2009. "An assessment of the relative impact of criminal justice and criminology journals," Journal of Criminal Justice, Elsevier, vol. 37(5), pages 505-511, September.
    33. Vanclay, Jerome K., 2008. "Ranking forestry journals using the h-index," Journal of Informetrics, Elsevier, vol. 2(4), pages 326-334.
    34. Loet Leydesdorff, 2009. "How are new citation‐based journal indicators adding to the bibliometric toolbox?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(7), pages 1327-1336, July.
    35. Cheng Zhang & Xu Liu & Yunjie (Calvin) Xu & Youwei Wang, 2011. "Quality‐structure index: A new metric to measure scientific journal influence," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(4), pages 643-653, April.
    36. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    37. Johan Bollen & Marko A. Rodriquez & Herbert Van de Sompel, 2006. "Journal status," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 669-687, December.
    38. Thomas Lange, 2006. "The Imprecise Science of Evaluating Scholarly Performance," Evaluation Review, , vol. 30(4), pages 505-532, August.
    39. Michael C. Calver & J. Stuart Bradley, 2009. "Should we use the mean citations per paper to summarise a journal’s impact or to rank journals in the same field?," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 611-615, December.
    40. Saarela, Mirka & Kärkkäinen, Tommi & Lahtonen, Tommi & Rossi, Tuomo, 2016. "Expert-based versus citation-based ranking of scholarly and scientific publication channels," Journal of Informetrics, Elsevier, vol. 10(3), pages 693-718.
    41. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    42. David Shilbury & Ruth Rentschler, 2007. "Assessing Sport Management Journals: A Multi-Dimensional Examination," Sport Management Review, Taylor & Francis Journals, vol. 10(1), pages 31-44, January.
    43. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Judit Dobránszki & Jaime A. Teixeira da Silva, 2019. "Corrective factors for author- and journal-based metrics impacted by citations to accommodate for retractions," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 387-398, October.
    2. Pooyan Makvandi & Anahita Nodehi & Franklin R. Tay, 2021. "Conference Accreditation and Need of a Bibliometric Measure to Distinguish Predatory Conferences," Publications, MDPI, vol. 9(2), pages 1-5, April.
    3. William H. Walters & Susanne Markgren, 2019. "Do faculty journal selections correspond to objective indicators of citation impact? Results for 20 academic departments at Manhattan College," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 321-337, January.
    4. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    5. Saarela, Mirka & Kärkkäinen, Tommi, 2020. "Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator," Journal of Informetrics, Elsevier, vol. 14(2).
    6. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    7. Yundong Xie & Qiang Wu & Xingchen Li, 2019. "Editorial team scholarly index (ETSI): an alternative indicator for evaluating academic journal reputation," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1333-1349, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    3. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    4. Saarela, Mirka & Kärkkäinen, Tommi & Lahtonen, Tommi & Rossi, Tuomo, 2016. "Expert-based versus citation-based ranking of scholarly and scientific publication channels," Journal of Informetrics, Elsevier, vol. 10(3), pages 693-718.
    5. Dejian Yu & Wanru Wang & Shuai Zhang & Wenyu Zhang & Rongyu Liu, 2017. "A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 521-542, April.
    6. Zhang, Fang & Wu, Shengli, 2020. "Predicting future influence of papers, researchers, and venues in a dynamic academic network," Journal of Informetrics, Elsevier, vol. 14(2).
    7. Currie, Russell R. & Pandher, Gurupdesh S., 2020. "Finance journal rankings: Active scholar assessment revisited," Journal of Banking & Finance, Elsevier, vol. 111(C).
    8. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    9. Zhang, Baolong & Wang, Hao & Deng, Sanhong & Su, Xinning, 2020. "Measurement and analysis of Chinese journal discriminative capacity," Journal of Informetrics, Elsevier, vol. 14(1).
    10. Fang Zhang & Shengli Wu, 2021. "Measuring academic entities’ impact by content-based citation analysis in a heterogeneous academic network," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7197-7222, August.
    11. Lior Rokach, 2012. "Applying the Publication Power Approach to Artificial Intelligence Journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(6), pages 1270-1277, June.
    12. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.
    13. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    14. Saarela, Mirka & Kärkkäinen, Tommi, 2020. "Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator," Journal of Informetrics, Elsevier, vol. 14(2).
    15. Johan Lyhagen & Per Ahlgren, 2020. "Uncertainty and the ranking of economics journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2545-2560, December.
    16. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    17. Vanclay, Jerome K., 2011. "An evaluation of the Australian Research Council's journal ranking," Journal of Informetrics, Elsevier, vol. 5(2), pages 265-274.
    18. Pajić, Dejan, 2015. "On the stability of citation-based journal rankings," Journal of Informetrics, Elsevier, vol. 9(4), pages 990-1006.
    19. Yu Zhang & Min Wang & Morteza Saberi & Elizabeth Chang, 2022. "Analysing academic paper ranking algorithms using test data and benchmarks: an investigation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4045-4074, July.
    20. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:11:y:2017:i:3:p:730-744. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.