IDEAS home Printed from https://ideas.repec.org/a/bpj/bejeap/v9y2009i1n34.html
   My bibliography  Save this article

Don't Get Skewed Over by Journal Rankings

Author

Listed:
  • Wall Howard J

    (Federal Reserve Bank of St. Louis)

Abstract

Nearly all journal rankings in economics use some weighted average of citations to calculate a journal's impact. These rankings are often used, formally or informally, to help assess the publication success of individual economists or institutions. Although ranking methods and opinions are legion, scant attention has been paid to the usefulness of any ranking as representative of the many articles published in a journal. First, because the distributions of citations across articles within a journal are seriously skewed, and the skewness differs across journals, the appropriate measure of central tendency is the median rather than the mean. Second, large shares of articles in the highest-ranked journals are cited less frequently than typical articles in much-lower-ranked journals.

Suggested Citation

  • Wall Howard J, 2009. "Don't Get Skewed Over by Journal Rankings," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 9(1), pages 1-12, August.
    • Handle: RePEc:bpj:bejeap:v:9:y:2009:i:1:n:34
    DOI: 10.2202/1935-1682.2280
    as

    Download full text from publisher

    File URL: https://doi.org/10.2202/1935-1682.2280
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.
    File URL: https://libkey.io/10.2202/1935-1682.2280?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Richard Dusansky & Clayton J. Vernon, 1998. "Rankings of U.S. Economics Departments," Journal of Economic Perspectives, American Economic Association, vol. 12(1), pages 157-170, Winter.
    2. Sandra Rousseau & Tom Verbeke & Ronald Rousseau, 2009. "Evaluating Environmental and Resource Economics Journals: A TOP-Curve Approach," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 3(2), pages 270-287, Summer.
    3. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    RePEc Biblio mentions

    As found on the RePEc Biblio, the curated bibliography for Economics:
    1. > Economics Profession > Ranking in Economics > Ranking Journals

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David L. Anderson & John Tressler, 2010. "The Merits of Using Citation-Based Journal Weighting Schemes to Measure Research Performance in Economics: The Case of New Zealand," Working Papers in Economics 10/03, University of Waikato.
    2. Klaus Wohlrabe, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," CESifo Working Paper Series 5726, CESifo.
    3. John Tressler & David L. Anderson, 2012. "Citations as a Measure of the Research Outputs of New Zealand's Economics Departments: The Problem of 'Long and Variable Lags'," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 19(1), pages 17-40.
    4. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    5. Seiler, Christian & Wohlrabe, Klaus, 2014. "How robust are journal rankings based on the impact factor? Evidence from the economic sciences," Journal of Informetrics, Elsevier, vol. 8(4), pages 904-911.
    6. Hofmeister Robert & Krapf Matthias, 2011. "How Do Editors Select Papers, and How Good are They at Doing It?," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 11(1), pages 1-23, October.
    7. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    8. Lando, Tommaso & Bertoli-Barsotti, Lucio, 2017. "Measuring the citation impact of journals with generalized Lorenz curves," Journal of Informetrics, Elsevier, vol. 11(3), pages 689-703.
    9. Bräuninger, Michael & Haucap, Justus & Muck, Johannes, 2011. "Was lesen und schätzen Ökonomen im Jahr 2011?," DICE Ordnungspolitische Perspektiven 18, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    10. Antonoyiannakis, Manolis, 2018. "Impact Factors and the Central Limit Theorem: Why citation averages are scale dependent," Journal of Informetrics, Elsevier, vol. 12(4), pages 1072-1088.
    11. David L. Anderson & John Tressler, 2011. "The Merits of Using Citations to Measure Research Output in Economics Departments: The New Zealand Case," Working Papers in Economics 11/11, University of Waikato.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 173-189, March.
    2. David Anderson & John Tressler, 2008. "Research output in New Zealand economics departments 2000-2006: A stock approach," New Zealand Economic Papers, Taylor & Francis Journals, vol. 42(2), pages 155-189.
    3. José Alberto Molina & Alfredo Ferrer & David Iñiguez & Alejandro Rivero & Gonzalo Ruiz & Alfonso Tarancón, 2020. "Network analysis to measure academic performance in economics," Empirical Economics, Springer, vol. 58(3), pages 995-1018, March.
    4. Giannias Dimitris A. & Sfakianaki Eleni, 2012. "University Rankings of Different Academic Positions for the Present and the Future: The Case of Greek Departments of Economics," Scientific Annals of Economics and Business, Sciendo, vol. 59(2), pages 43-66, December.
    5. Howard J. Wall, 2009. "Journal rankings in economics: handle with care," Working Papers 2009-014, Federal Reserve Bank of St. Louis.
    6. David L. Anderson & John Tressler, 2008. "Research Output in New Zealand Economics Department 2000-2006," Working Papers in Economics 08/05, University of Waikato.
    7. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2012. "A comparison of top economics departments in the US and EU on the basis of the multidimensional prestige of influential articles in 2010," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 681-698, December.
    8. Goodall, Amanda H., 2009. "Highly cited leaders and the performance of research universities," Research Policy, Elsevier, vol. 38(7), pages 1079-1092, September.
    9. Javier Ruiz-Castillo, 2008. "Economics research in Spain during the 1990s: a literature review," Spanish Economic Review, Springer;Spanish Economic Association, vol. 10(3), pages 221-249, September.
    10. Klaus Wohlrabe, 2018. "Selected Remarks on Frankfurter Allgemeine Zeitung Economists’ Ranking 2018," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 71(20), pages 29-33, October.
    11. Raúl Ramos & Vicente Royuela & Jordi Suriñach, 2007. "An analysis of the determinants in Economics and Business publications by Spanish universities between 1994 and 2004," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(1), pages 117-144, April.
    12. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    13. Peter Harrison Howard & Derek Sylvan, 2020. "Wisdom of the experts: Using survey responses to address positive and normative uncertainties in climate-economic models," Climatic Change, Springer, vol. 162(2), pages 213-232, September.
    14. Singell, Larry D. & Tang, Hui-Hsuan, 2013. "Pomp and circumstance: University presidents and the role of human capital in determining who leads U.S. research institutions," Economics of Education Review, Elsevier, vol. 32(C), pages 219-233.
    15. Jakub Rybacki, 2019. "Are Central Banks' Research Teams Fragile Because of Groupthink?," KAE Working Papers 2019-045, Warsaw School of Economics, Collegium of Economic Analysis.
    16. Alexander Butz & Klaus Wohlrabe, 2016. "Die Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc: Methodik, Ergebnisse, Kritik und Vergleich," ifo Working Paper Series 212, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    17. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    18. Melody Lo & M. C. Sunny Wong & Franklin G. Mixon, 2008. "Ranking Economics Journals, Economics Departments, and Economists Using Teaching‐Focused Research Productivity," Southern Economic Journal, John Wiley & Sons, vol. 74(3), pages 894-906, January.
    19. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:bejeap:v:9:y:2009:i:1:n:34. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.