IDEAS home Printed from https://ideas.repec.org/a/bla/ecinqu/v59y2021i3p961-977.html
   My bibliography  Save this article

Depth and breadth relevance in citation metrics

Author

Listed:
  • David I. Stern
  • Richard S. J. Tol

Abstract

The Euclidean length of a citation list is “depth relevant”: the metric increases when citations are transferred from less to more cited papers. We introduce “breadth relevance,” which favors consistent achievers over one‐hit wonders. The exponent of the CES aggregator then is less than unity rather than greater than unity, as for depth relevance. Using two datasets on citations of economists for the top 50 US and global universities, simply counting citations maximizes the correlation between the citation metrics of researchers and the peer‐reviewed rank of their department. However, citation depth may explain the allocation of researchers across lower‐ranked departments.

Suggested Citation

  • David I. Stern & Richard S. J. Tol, 2021. "Depth and breadth relevance in citation metrics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 961-977, July.
  • Handle: RePEc:bla:ecinqu:v:59:y:2021:i:3:p:961-977
    DOI: 10.1111/ecin.12994
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/ecin.12994
    Download Restriction: no

    File URL: https://libkey.io/10.1111/ecin.12994?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Glenn Ellison, 2013. "How Does the Market Use Citation Data? The Hirsch Index in Economics," American Economic Journal: Applied Economics, American Economic Association, vol. 5(3), pages 63-90, July.
    2. Gibbons, Stephen & Neumayer, Eric & Perkins, Richard, 2015. "Student satisfaction, league tables and university applications: Evidence from Britain," Economics of Education Review, Elsevier, vol. 48(C), pages 148-164.
    3. Michel Lubrano & Luc Bauwens & Alan Kirman & Camelia Protopopescu, 2003. "Ranking Economics Departments in Europe: A Statistical Approach," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1367-1401, December.
    4. Abram Burk, 1936. "Real Income, Expenditure Proportionality, and Frisch's "New Methods of Measuring Marginal Utility"," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 4(1), pages 33-52.
    5. Vladimir G. Deineko & Gerhard J. Woeginger, 2009. "A new family of scientific impact measures: The generalized Kosmulski-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(3), pages 819-826, September.
    6. Powdthavee, Nattavudh & Riyanto, Yohanes E. & Knetsch, Jack L., 2018. "Lower-rated publications do lower academics’ judgments of publication lists: Evidence from a survey experiment of economists," Journal of Economic Psychology, Elsevier, vol. 66(C), pages 33-44.
    7. Kevin A. Bryan, 2019. "Young “Stars” In Economics: What They Do And Where They Go," Economic Inquiry, Western Economic Association International, vol. 57(3), pages 1392-1407, July.
    8. Robert M. Solow, 1956. "A Contribution to the Theory of Economic Growth," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 70(1), pages 65-94.
    9. Kalaitzidakis, Pantelis & Mamuneas, Theofanis P. & Stengos, Thanasis, 1999. "European economics: An analysis based on publications in the core journals," European Economic Review, Elsevier, vol. 43(4-6), pages 1150-1168, April.
    10. Dean Showalter & Richard Jensen, 2019. "University Startup Intensity And Faculty Quality," Economic Inquiry, Western Economic Association International, vol. 57(2), pages 855-875, April.
    11. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    12. Tol, Richard S.J., 2013. "The Matthew effect for cohorts of economists," Journal of Informetrics, Elsevier, vol. 7(2), pages 522-527.
    13. Scott, Loren C & Mitias, Peter M, 1996. "Trends in Rankings of Economics Departments in the U.S.: An Update," Economic Inquiry, Western Economic Association International, vol. 34(2), pages 378-400, April.
    14. Pierre-Philippe Combes & Laurent Linnemer, 2003. "Where are the Economists Who Publish? Publication Concentration and Rankings in Europe Based on Cumulative Publications," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1250-1308, December.
    15. Harvey Goldstein & David J. Spiegelhalter, 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(3), pages 385-409, May.
    16. Quesada, Antonio, 2011. "Axiomatics for the Hirsch index and the Egghe index," Journal of Informetrics, Elsevier, vol. 5(3), pages 476-480.
    17. David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 173-189, March.
    18. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    19. Woeginger, Gerhard J., 2008. "An axiomatic characterization of the Hirsch-index," Mathematical Social Sciences, Elsevier, vol. 56(2), pages 224-232, September.
    20. Daniel S. Hamermesh & Gerard A. Pfann, 2012. "Reputation And Earnings: The Roles Of Quality And Quantity In Academe," Economic Inquiry, Western Economic Association International, vol. 50(1), pages 1-16, January.
    21. Halevi, Gali & Moed, Henk & Bar-Ilan, Judit, 2017. "Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the Literature," Journal of Informetrics, Elsevier, vol. 11(3), pages 823-834.
    22. Frank L DuBois & David Reeb, 2000. "Ranking the International Business Journals," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 31(4), pages 689-704, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. Annual Review 2021
      by noreply@blogger.com (David Stern) in Stochastic Trend on 2021-12-30 06:11:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Johannes König & David I. Stern & Richard S.J. Tol, 2022. "Confidence Intervals for Recursive Journal Impact Factors," Tinbergen Institute Discussion Papers 22-038/III, Tinbergen Institute.
    2. Björn Dressel & David I. Stern, 2021. "Research at public policy schools in the Asia‐Pacific region ranked," Asia and the Pacific Policy Studies, Wiley Blackwell, vol. 8(1), pages 151-166, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David I. Stern & Richard S.J. Tol, 2018. "How to Count Citations If You Must: Comment," Working Paper Series 0118, Department of Economics, University of Sussex Business School.
    2. James B. Davies & Martin G. Kocher & Matthias Sutter, 2008. "Economics research in Canada: a long‐run assessment of journal publications," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(1), pages 22-45, February.
    3. J. Atsu Amegashie, 2020. "Citations And Incentives In Academic Contests," Economic Inquiry, Western Economic Association International, vol. 58(3), pages 1233-1244, July.
    4. Seiler, Christian & Wohlrabe, Klaus, 2012. "Ranking economists on the basis of many indicators: An alternative approach using RePEc data," Journal of Informetrics, Elsevier, vol. 6(3), pages 389-402.
    5. Stephen Bazen & Patrick Moyes, 2012. "Elitism and stochastic dominance," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 39(1), pages 207-251, June.
    6. Jan Ours & Frederic Vermeulen, 2007. "Ranking Dutch Economists," De Economist, Springer, vol. 155(4), pages 469-487, December.
    7. Libman, A., 2010. "Economics in Germany – from National to Global," Journal of the New Economic Association, New Economic Association, issue 8, pages 155-158.
    8. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    9. Jin, Jang C. & Hong, Jin-Heon, 2008. "East Asian rankings of economics departments," Journal of Asian Economics, Elsevier, vol. 19(1), pages 74-82, February.
    10. Tombazos, Christis G., 2005. "A revisionist perspective of European research in economics," European Economic Review, Elsevier, vol. 49(2), pages 251-277, February.
    11. Martin Gregor, 2006. "Hodnocení ekonomických pracovišť a ekonomů: Koho, proč, čím a jak [A survey of rankings of economic departments: Global, american, european and national]," Politická ekonomie, Prague University of Economics and Business, vol. 2006(3), pages 394-414.
    12. Jang C. Jin, 2009. "Publications in mathematical economics and econometrics: ranking of Asian universities and an application of Zipf's law," Asian-Pacific Economic Literature, The Crawford School, The Australian National University, vol. 23(2), pages 116-122, November.
    13. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    14. Baltagi, Badi H., 2007. "Worldwide Econometrics Rankings: 1989–2005," Econometric Theory, Cambridge University Press, vol. 23(5), pages 952-1012, October.
    15. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    16. Klaus Ritzberger, 2008. "Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 267-285, August.
    17. Tilak Mukhopadhyay & Subrata Sarar, 2010. "Rankings of Economics Journals and Departments in India," Indira Gandhi Institute of Development Research, Mumbai Working Papers 2010-021, Indira Gandhi Institute of Development Research, Mumbai, India.
    18. Joseph Macri & Dipendra Sinha, 2006. "Rankings Methodology for International Comparisons of Institutions and Individuals: an Application to Economics in Australia and New Zealand," Journal of Economic Surveys, Wiley Blackwell, vol. 20(1), pages 111-156, February.
    19. Murat Çokgezen, 2019. "Research Performance of Turkish Economists and Economics Departments: Another Update and a Review of the 2000s," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 55(1), pages 133-149, January.
    20. Jang C. Jin, 2009. "Asian University Rankings in International and Development Economics: An Application of Zipf's Law," Review of International Economics, Wiley Blackwell, vol. 17(1), pages 137-143, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:ecinqu:v:59:y:2021:i:3:p:961-977. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/weaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.