IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v49y2020i4s0048733320300317.html
   My bibliography  Save this article

The Matthew effect of a journal's ranking

Author

Listed:
  • Drivas, Kyriakos
  • Kremmydas, Dimitris

Abstract

We examine the impact of a journal's ranking on the citations its papers garner. Our testbed is journals included in the early annual publications of the Academic Journal Guide, published by the Association of Business Schools from 2007 to 2010. By focusing on a small subset of these journals, we provide causal evidence that an increase in a journal's ranking will increase citations to its papers. We argue that this increase can be attributed to (i) more authors learning about and viewing these outlets and their publications and (ii) researchers signalling their paper's own impact by citing highly ranked journals. We find some evidence for the former, though not robust, and substantial evidence for the latter. Given that signalling is deliberate and associated with incentives to publish in highly ranked journals, we decompose this channel by citing researchers’ characteristics. Except for senior academics, all types of researchers are consistent with signalling. The policy implications of our results relate to the pervasive use of journal rankings and recent initiatives to evaluate research.

Suggested Citation

  • Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    • Handle: RePEc:eee:respol:v:49:y:2020:i:4:s0048733320300317
    DOI: 10.1016/j.respol.2020.103951
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733320300317
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2020.103951?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    2. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    3. Daniel Sgroi & Andrew J. Oswald, 2013. "How Should Peer‐review Panels Behave?," Economic Journal, Royal Economic Society, vol. 0, pages 255-278, August.
    4. Goldfarb, Brent, 2008. "The effect of government contracting on academic research: Does the source of funding affect scientific output," Research Policy, Elsevier, vol. 37(1), pages 41-58, February.
    5. Gourieroux, Christian & Monfort, Alain & Trognon, Alain, 1984. "Pseudo Maximum Likelihood Methods: Applications to Poisson Models," Econometrica, Econometric Society, vol. 52(3), pages 701-720, May.
    6. Brooks, Chris & Fenton, Evelyn & Schopohl, Lisa & Walker, James, 2019. "Why does research in finance have so little impact?," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 58(C), pages 24-52.
    7. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    8. Philippe Mongeon & Adèle Paul-Hus, 2016. "The journal coverage of Web of Science and Scopus: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 213-228, January.
    9. James J. Heckman & Sidharth Moktan, 2020. "Publishing and promotion in economics - The tyranny of the Top Five," Vox eBook Chapters, in: Sebastian Galliani & Ugo Panizza (ed.), Publishing and Measuring Success in Economics, edition 1, volume 1, chapter 1, pages 23-32, Centre for Economic Policy Research.
    10. Heidi L. Williams, 2013. "Intellectual Property Rights and Innovation: Evidence from the Human Genome," Journal of Political Economy, University of Chicago Press, vol. 121(1), pages 1-27.
    11. Hussain, Simon, 2010. "Accounting journals and the ABS quality ratings," The British Accounting Review, Elsevier, vol. 42(1), pages 1-16.
    12. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    13. Nikolaos A. Kazakis & Anastasios D. Diamantidis & Leonidas L. Fragidis & Miltos K. Lazarides, 2014. "Evaluating the research performance of the Greek medical schools using bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1367-1384, February.
    14. Christos Kolympiris & Sebastian Hoenen & Peter G. Klein, 2019. "Learning by Seconding: Evidence from National Science Foundation Rotators," Organization Science, INFORMS, vol. 30(3), pages 528-551, May.
    15. Salterio, Steven E., 2018. "Signaling effects of scholarly profiles–the editorial teams of North American accounting association journals: A personal perspective," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 78-83.
    16. Pierre Azoulay & Toby Stuart & Yanbo Wang, 2014. "Matthew: Effect or Fable?," Management Science, INFORMS, vol. 60(1), pages 92-109, January.
    17. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    18. David N. Laband, 2013. "On the Use and Abuse of Economics Journal Rankings," Economic Journal, Royal Economic Society, vol. 0, pages 223-254, August.
    19. Gerhard Knothe, 2006. "Comparative citation analysis of duplicate or highly related publications," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(13), pages 1830-1839, November.
    20. repec:bla:germec:v:9:y:2008:i::p:402-430 is not listed on IDEAS
    21. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    22. Iacus, Stefano & King, Gary & Porro, Giuseppe, 2009. "cem: Software for Coarsened Exact Matching," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 30(i09).
    23. William H. Starbuck, 2015. "Issues and Trends in Publishing Behavioral Science: A Quarrelsome Crew Struggling with a Disintegrating Boat on a Stormy Sea," Springer Books, in: Isabell M. Welpe & Jutta Wollersheim & Stefanie Ringelhan & Margit Osterloh (ed.), Incentives and Performance, edition 127, pages 3-18, Springer.
    24. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    25. Éric Archambault & David Campbell & Yves Gingras & Vincent Larivière, 2009. "Comparing bibliometric statistics obtained from the Web of Science and Scopus," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(7), pages 1320-1326, July.
    26. Matthew Blackwell & Stefano Iacus & Gary King & Giuseppe Porro, 2009. "cem: Coarsened exact matching in Stata," Stata Journal, StataCorp LP, vol. 9(4), pages 524-546, December.
    27. Daniel Feenberg & Ina Ganguli & Patrick Gaulé & Jonathan Gruber, 2017. "It’s Good to Be First: Order Bias in Reading and Citing NBER Working Papers," The Review of Economics and Statistics, MIT Press, vol. 99(1), pages 32-39, March.
    28. Endenich, Christoph & Trapp, Rouven, 2018. "Signaling effects of scholarly profiles – The editorial teams of North American accounting association journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 4-23.
    29. George A. Lozano & Vincent Larivière & Yves Gingras, 2012. "The weakening relationship between the impact factor and papers' citations in the digital age," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(11), pages 2140-2145, November.
    30. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    31. Julia Lane, 2010. "Let's make science metrics more scientific," Nature, Nature, vol. 464(7288), pages 488-489, March.
    32. Jeffrey L. Furman & Scott Stern, 2011. "Climbing atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research," American Economic Review, American Economic Association, vol. 101(5), pages 1933-1963, August.
    33. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    34. Aghaei Chadegani, Arezoo & Salehi, Hadi & Md Yunus, Melor & Farhadi, Hadi & Fooladi, Masood & Farhadi, Maryam & Ale Ebrahim, Nader, 2013. "A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases," MPRA Paper 46898, University Library of Munich, Germany, revised 18 Mar 2013.
    35. George A. Lozano & Vincent Larivière & Yves Gingras, 2012. "The weakening relationship between the impact factor and papers' citations in the digital age," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(11), pages 2140-2145, November.
    36. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    37. Stephan, Paula E., 2010. "The Economics of Science," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 1, chapter 0, pages 217-273, Elsevier.
    38. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    39. Macdonald, Stuart & Kam, Jacqueline, 2010. "Counting footnotes: Citability in management studies," Scandinavian Journal of Management, Elsevier, vol. 26(2), pages 189-203, June.
    40. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    41. Pierre Azoulay & Joshua S. Graff Zivin & Jialan Wang, 2010. "Superstar Extinction," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(2), pages 549-589.
    42. Brooks, Chris & Schopohl, Lisa, 2018. "Topics and trends in finance research: What is published, who publishes it and what gets cited?," The British Accounting Review, Elsevier, vol. 50(6), pages 615-637.
    43. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    44. Spangenberg, John F. A. & Starmans, Richard & Bally, Yvonne W. & Breemhaar, Bert & Nijhuis, Frans J. N., 1990. "Prediction of scientific performance in clinical medicine," Research Policy, Elsevier, vol. 19(3), pages 239-255, June.
    45. Iacus, Stefano M. & King, Gary & Porro, Giuseppe, 2012. "Causal Inference without Balance Checking: Coarsened Exact Matching," Political Analysis, Cambridge University Press, vol. 20(1), pages 1-24, January.
    46. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    47. Salter, Ammon & Salandra, Rossella & Walker, James, 2017. "Exploring preferences for impact versus publications among UK business and management academics," Research Policy, Elsevier, vol. 46(10), pages 1769-1782.
    48. Ritzberger Klaus, 2008. "A Ranking of Journals in Economics and Related Fields," German Economic Review, De Gruyter, vol. 9(4), pages 402-430, December.
    49. Endenich, Christoph & Trapp, Rouven, 2018. "Rejoinder: Toward an overarching signaling framework – The editorial teams of North American accounting association journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 84-86.
    50. Michael Roach & Wesley M. Cohen, 2013. "Lens or Prism? Patent Citations as a Measure of Knowledge Flows from Public Research," Management Science, INFORMS, vol. 59(2), pages 504-525, October.
    51. Ewen Callaway, 2016. "Beat it, impact factor! Publishing elite turns against controversial metric," Nature, Nature, vol. 535(7611), pages 210-211, July.
    52. Michael Spence, 1973. "Job Market Signaling," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 87(3), pages 355-374.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yves Fassin, 2021. "Does the Financial Times FT50 journal list select the best management and economics journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5911-5943, July.
    2. Nasirov, Shukhrat & Joshi, Amol M., 2023. "Minding the communications gap: How can universities signal the availability and value of their scientific knowledge to commercial organizations?," Research Policy, Elsevier, vol. 52(9).
    3. Maria-Victoria Uribe-Bohorquez & Juan-Camilo Rivera-Ordóñez & Isabel-María García-Sánchez, 2023. "Gender disparities in accounting academia: analysis from the lens of publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(7), pages 3827-3865, July.
    4. Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.
    5. Wang, Peiling & Su, Jing, 2021. "Post-publication expert recommendations in faculty opinions (F1000Prime): Recommended articles and citations," Journal of Informetrics, Elsevier, vol. 15(3).
    6. Jaime A. Teixeira da Silva, 2021. "The Matthew effect impacts science and academic publishing by preferentially amplifying citations, metrics and status," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5373-5377, June.
    7. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    8. Piotr Śpiewanowski & Oleksandr Talavera, 2021. "Journal rankings and publication strategy," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3227-3242, April.
    9. Salandra, Rossella & Criscuolo, Paola & Salter, Ammon, 2021. "Directing scientists away from potentially biased publications: the role of systematic reviews in health care," Research Policy, Elsevier, vol. 50(1).
    10. Li Hou & Qiang Wu & Yundong Xie, 2022. "Does early publishing in top journals really predict long-term scientific success in the business field?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6083-6107, November.
    11. Brooks, Chris & Schopohl, Lisa & Walker, James T., 2023. "Comparing perceptions of the impact of journal rankings between fields," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 90(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    2. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    3. Yves Fassin, 2021. "Does the Financial Times FT50 journal list select the best management and economics journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5911-5943, July.
    4. Brooks, Chris & Schopohl, Lisa & Walker, James T., 2023. "Comparing perceptions of the impact of journal rankings between fields," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 90(C).
    5. Hussain, Simon & Liu, Lana Yan Jun & Miller, Anthony D., 2020. "Accounting as a dichotomised discipline: An analysis of the source materials used in the construction of accounting articles," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 66(C).
    6. Graddy-Reed, Alexandra & Lanahan, Lauren & D'Agostino, Jesse, 2021. "Training across the academy: The impact of R&D funding on graduate students," Research Policy, Elsevier, vol. 50(5).
    7. Matthias Aistleitner & Jakob Kapeller & Stefan Steinerberger, 2018. "Citation Patterns in Economics and Beyond," Working Papers Series 85, Institute for New Economic Thinking.
    8. Hendrik P. Dalen, 2021. "How the publish-or-perish principle divides a science: the case of economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1675-1694, February.
    9. van Dalen, Hendrik Peter, 2021. "How the publish-or-perish principle divides a science: The case of economists," Other publications TiSEM a6a5a855-bb5a-4d52-a841-3, Tilburg University, School of Economics and Management.
    10. Johannes König & David I. Stern & Richard S.J. Tol, 2022. "Confidence Intervals for Recursive Journal Impact Factors," Tinbergen Institute Discussion Papers 22-038/III, Tinbergen Institute.
    11. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    12. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    13. Eunhee Sohn, 2021. "How Local Industry R&D Shapes Academic Research: Evidence from the Agricultural Biotechnology Revolution," Organization Science, INFORMS, vol. 32(3), pages 675-707, May.
    14. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    15. Lanu Kim & Jason H. Portenoy & Jevin D. West & Katherine W. Stovel, 2020. "Scientific journals still matter in the era of academic search engines and preprint archives," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(10), pages 1218-1226, October.
    16. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    17. Olof Ejermo & Claudio Fassio & John Källström, 2020. "Does Mobility across Universities Raise Scientific Productivity?," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 82(3), pages 603-624, June.
    18. van Dalen, Hendrik Peter, 2020. "How the Publish-or-Perish Principle Divides a Science : The Case of Academic Economists," Other publications TiSEM 6fbb6b92-0e06-4271-b6e7-3, Tilburg University, School of Economics and Management.
    19. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    20. J. Atsu Amegashie, 2020. "Citations And Incentives In Academic Contests," Economic Inquiry, Western Economic Association International, vol. 58(3), pages 1233-1244, July.

    More about this item

    Keywords

    Journal rankings; AJG/ABS list; Scientific diffusion; Matthew effect; Difference-in-differences; Coarsened exact matching;
    All these keywords.

    JEL classification:

    • L51 - Industrial Organization - - Regulation and Industrial Policy - - - Economics of Regulation
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • O38 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Government Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:49:y:2020:i:4:s0048733320300317. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.