IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v51y2022i10s0048733322001317.html
   My bibliography  Save this article

Who games metrics and rankings? Institutional niches and journal impact factor inflation

Author

Listed:
  • Siler, Kyle
  • Larivière, Vincent

Abstract

Ratings and rankings are omnipresent and influential in contemporary society. Individuals and organizations strategically respond to incentives set by rating systems. We use academic publishing as a case study to examine organizational variation in responses to influential metrics. The Journal Impact Factor (JIF) is a prominent metric linked to the value of academic journals, as well as career prospects of researchers. Since scholars, institutions, and publishers alike all have strong interests in affiliating with high JIF journals, strategic behaviors to ‘game’ the JIF metric are prevalent. Strategic self-citation is a common tactic employed to inflate JIF values. Based on empirical analyses of academic journals indexed in the Web of Science, we examine institutional characteristics conducive to strategic self-citation for JIF inflation. Journals disseminated by for-profit publishers, with lower JIFs, published in academically peripheral countries and with more recent founding dates were more likely to exhibit JIF-inflating self-citation patterns. Findings reveal the importance of status and institutional logics in influencing metrics gaming behaviors, as well as how metrics can affect work outcomes in different types of institutions. While quantitative rating systems affect many who are being evaluated, certain types of people and organizations are more prone to being influenced by rating systems than others.

Suggested Citation

  • Siler, Kyle & Larivière, Vincent, 2022. "Who games metrics and rankings? Institutional niches and journal impact factor inflation," Research Policy, Elsevier, vol. 51(10).
    • Handle: RePEc:eee:respol:v:51:y:2022:i:10:s0048733322001317
    DOI: 10.1016/j.respol.2022.104608
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733322001317
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2022.104608?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hall, Jeremy & Martin, Ben R., 2019. "Towards a taxonomy of research misconduct: The case of business school research," Research Policy, Elsevier, vol. 48(2), pages 414-427.
    2. Dina Mayzlin & Yaniv Dover & Judith Chevalier, 2014. "Promotional Reviews: An Empirical Investigation of Online Review Manipulation," American Economic Review, American Economic Association, vol. 104(8), pages 2421-2455, August.
    3. Theodore C. Bergstrom, 2001. "Free Labour for Costly Journals?," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 183-198, Fall.
    4. Seeber, Marco & Cattaneo, Mattia & Meoli, Michele & Malighetti, Paolo, 2019. "Self-citations as strategic response to the use of metrics for career decisions," Research Policy, Elsevier, vol. 48(2), pages 478-491.
    5. Shubhanshu Mishra & Brent D Fegley & Jana Diesner & Vetle I Torvik, 2018. "Self-citation is the hallmark of productive authors, of any gender," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-21, September.
    6. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    7. Vincent Larivière & Stefanie Haustein & Philippe Mongeon, 2015. "The Oligopoly of Academic Publishers in the Digital Era," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    8. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    9. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    10. Mario Biagioli, 2016. "Watch out for cheats in citation game," Nature, Nature, vol. 535(7611), pages 201-201, July.
    11. Kyle Siler, 2020. "Demarcating spectrums of predatory publishing: Economic and institutional sources of academic legitimacy," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(11), pages 1386-1401, November.
    12. Berggren, Christian & Karabag, Solmaz Filiz, 2019. "Scientific misconduct at an elite medical institute: The role of competing institutional logics and fragmented control," Research Policy, Elsevier, vol. 48(2), pages 428-443.
    13. Julien Jacqmin, 2021. "Do ads influence rankings? Evidence from the higher education sector," Education Economics, Taylor & Francis Journals, vol. 29(5), pages 509-526, September.
    14. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    15. Ruben Miranda & Esther Garcia-Carpintero, 2019. "Comparison of the share of documents and citations from different quartile journals in 25 research areas," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 479-501, October.
    16. Howard D. White, 2001. "Authors as citers over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 52(2), pages 87-108.
    17. Marion Fourcade & Kieran Healy, 2013. "Classification situations: Life-chances in the neoliberal era," Post-Print hal-03470535, HAL.
    18. Petersen, Alexander M., 2019. "Megajournal mismanagement: Manuscript decision bias and anomalous editor activity at PLOS ONE," Journal of Informetrics, Elsevier, vol. 13(4).
    19. repec:hal:spmain:info:hdl:2441/6cbt691h0h8o9q5rf0apko0pda is not listed on IDEAS
    20. Wilhite, Allen & Fong, Eric A. & Wilhite, Seth, 2019. "The influence of editorial decisions and the academic network on self-citations and journal impact factors," Research Policy, Elsevier, vol. 48(6), pages 1513-1522.
    21. Li Tang & Guangyuan Hu, 2018. "Evaluation woes: Metrics can help beat bias," Nature, Nature, vol. 559(7714), pages 331-331, July.
    22. Matthew S. Bothner & Joel M. Podolny & Edward Bishop Smith, 2011. "Organizing Contests for Status: The Matthew Effect vs. the Mark Effect," Management Science, INFORMS, vol. 57(3), pages 439-457, March.
    23. Jingfeng Xia & Jennifer L. Harmon & Kevin G. Connolly & Ryan M. Donnelly & Mary R. Anderson & Heather A. Howard, 2015. "Who publishes in “predatory” journals?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(7), pages 1406-1417, July.
    24. David A. King, 2004. "The scientific impact of nations," Nature, Nature, vol. 430(6997), pages 311-316, July.
    25. Fourcade, Marion & Healy, Kieran, 2013. "Classification situations: Life-chances in the neoliberal era," Accounting, Organizations and Society, Elsevier, vol. 38(8), pages 559-572.
    26. Badi H. Baltagi, 2021. "Econometric Analysis of Panel Data," Springer Texts in Business and Economics, Springer, edition 6, number 978-3-030-53953-5, August.
    27. Chirikov, Igor, 2021. "Does Conflict of Interest Distort Global University Rankings?," University of California at Berkeley, Center for Studies in Higher Education qt8hk672nh, Center for Studies in Higher Education, UC Berkeley.
    28. Kyle Siler & Philippe Vincent-Lamarre & Cassidy R. Sugimoto & Vincent Larivière, 2021. "Predatory publishers’ latest scam: bootlegged and rebranded papers," Nature, Nature, vol. 598(7882), pages 563-565, October.
    29. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    30. Jeremy Hall & Ben R. Martin, 2019. "Towards a Taxonomy of Academic Misconduct: The Case of Business School Research," SPRU Working Paper Series 2019-02, SPRU - Science Policy Research Unit, University of Sussex Business School.
    31. Amandine Ody‐Brasier & Amanda Sharkey, 2019. "Under pressure: Reputation, ratings, and inaccurate self‐reporting in the nursing home industry," Strategic Management Journal, Wiley Blackwell, vol. 40(10), pages 1517-1544, October.
    32. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    33. Charles F Hickman & Eric A Fong & Allen W Wilhite & Yeolan Lee, 2019. "Academic misconduct and criminal liability: Manipulating academic journal impact factors," Science and Public Policy, Oxford University Press, vol. 46(5), pages 661-667.
    34. Henry Small, 2004. "On the shoulders of Robert Merton: Towards a normative theory of citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(1), pages 71-79, May.
    35. Campbell, Donald T., 1979. "Assessing the impact of planned social change," Evaluation and Program Planning, Elsevier, vol. 2(1), pages 67-90, January.
    36. Michael Luca & Georgios Zervas, 2016. "Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud," Management Science, INFORMS, vol. 62(12), pages 3412-3427, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yangping Zhou, 2021. "Self-citation and citation of top journal publishers and their interpretation in the journal-discipline context," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6013-6040, July.
    2. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    3. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).
    4. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    5. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    6. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    7. Hui Li & Weishu Liu, 2020. "Same same but different: self-citations identified through Scopus and Web of Science Core Collection," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2723-2732, September.
    8. Peter Sjögårde & Fereshteh Didegah, 2022. "The association between topic growth and citation impact of research publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1903-1921, April.
    9. Schmal, W. Benedikt & Haucap, Justus & Knoke, Leon, 2023. "The role of gender and coauthors in academic publication behavior," Research Policy, Elsevier, vol. 52(10).
    10. Jesús de Frutos-Belizón & Fernando Martín-Alcázar & Gonzalo Sánchez-Gardey, 2021. "The research–practice gap in the field of HRM: a qualitative study from the academic side of the gap," Review of Managerial Science, Springer, vol. 15(6), pages 1465-1515, August.
    11. Lanu Kim & Jason H. Portenoy & Jevin D. West & Katherine W. Stovel, 2020. "Scientific journals still matter in the era of academic search engines and preprint archives," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(10), pages 1218-1226, October.
    12. Apostolos Filippas & John J. Horton & Joseph M. Golden, 2022. "Reputation Inflation," Marketing Science, INFORMS, vol. 41(4), pages 733-745, July.
    13. Hendrik P. Dalen, 2021. "How the publish-or-perish principle divides a science: the case of economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1675-1694, February.
    14. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    15. Sánchez-Gil, Susana & Gorraiz, Juan & Melero-Fuentes, David, 2018. "Reference density trends in the major disciplines," Journal of Informetrics, Elsevier, vol. 12(1), pages 42-58.
    16. Tobias Kiesslich & Marlena Beyreis & Georg Zimmermann & Andreas Traweger, 2021. "Citation inequality and the Journal Impact Factor: median, mean, (does it) matter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1249-1269, February.
    17. van Dalen, Hendrik Peter, 2020. "How the Publish-or-Perish Principle Divides a Science : The Case of Academic Economists," Other publications TiSEM 6fbb6b92-0e06-4271-b6e7-3, Tilburg University, School of Economics and Management.
    18. van Dalen, Hendrik Peter, 2021. "How the publish-or-perish principle divides a science: The case of economists," Other publications TiSEM a6a5a855-bb5a-4d52-a841-3, Tilburg University, School of Economics and Management.
    19. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    20. Ameni Kacem & Justin W. Flatt & Philipp Mayr, 2020. "Tracking self-citations in academic publishing," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1157-1165, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:51:y:2022:i:10:s0048733322001317. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.