IDEAS home Printed from https://ideas.repec.org/a/aea/jeclit/v51y2013i1p173-89.html
   My bibliography  Save this article

Uncertainty Measures for Economics Journal Impact Factors

Author

Listed:
  • David I. Stern

Abstract

Academic economists appear to be intensely interested in rankings of journals, institutions, and individuals. Yet there is little discussion of the uncertainty associated with these rankings. To illustrate the uncertainty associated with citations-based rankings, I compute the standard error of the impact factor for all economics journals with a five-year impact factor in the 2011 Journal Citations Report . I use these to derive confidence intervals for the impact factors as well as ranges of possible rank for a subset of thirty journals. I find that the impact factors of the top two journals are well defined and set these journals apart in a clearly defined group. An elite group of 9-11 mainstream journals can also be fairly reliably distinguished. The four bottom ranked journals are also fairly clearly set apart. For the remainder of the distribution, confidence intervals overlap and rankings are quite uncertain. (JEL A14)

Suggested Citation

  • David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 173-189, March.
  • Handle: RePEc:aea:jeclit:v:51:y:2013:i:1:p:173-89
    Note: DOI: 10.1257/jel.51.1.173
    as

    Download full text from publisher

    File URL: http://www.aeaweb.org/articles.php?doi=10.1257/jel.51.1.173
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Christian Zimmermann, 2013. "Academic Rankings with RePEc," Econometrics, MDPI, Open Access Journal, vol. 1(3), pages 1-32, December.
    2. repec:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0679-8 is not listed on IDEAS
    3. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    4. David N. Laband & Robert D. Tollison, 2003. "Dry Holes in Economic Research," Kyklos, Wiley Blackwell, vol. 56(2), pages 161-173, May.
    5. Richard Dusansky & Clayton J. Vernon, 1998. "Rankings of U.S. Economics Departments," Journal of Economic Perspectives, American Economic Association, vol. 12(1), pages 157-170, Winter.
    6. S. Redner, 1998. "How popular is your paper? An empirical study of the citation distribution," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 4(2), pages 131-134, July.
    7. repec:spr:scient:v:78:y:2009:i:2:d:10.1007_s11192-008-2166-9 is not listed on IDEAS
    8. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    9. repec:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0634-8 is not listed on IDEAS
    10. Ma, Chunbo & Stern, David I., 2006. "Environmental and ecological economics: A citation analysis," Ecological Economics, Elsevier, vol. 58(3), pages 491-506, June.
    11. repec:spr:scient:v:51:y:2001:i:2:d:10.1023_a:1012705818635 is not listed on IDEAS
    12. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    13. Burton, M P & Phimister, Euan, 1995. "Core Journals: A Reappraisal of the Diamond List," Economic Journal, Royal Economic Society, vol. 105(429), pages 361-373, March.
    14. Laband, David N & Piette, Michael J, 1994. "The Relative Impacts of Economics Journals: 1970-1990," Journal of Economic Literature, American Economic Association, vol. 32(2), pages 640-666, June.
    15. Liebowitz, S J & Palmer, J P, 1984. "Assessing the Relative Impacts of Economic Journals," Journal of Economic Literature, American Economic Association, vol. 22(1), pages 77-88, March.
    16. repec:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0747-0 is not listed on IDEAS
    17. Tom Coupé, 2003. "Revealed Performances: Worldwide Rankings of Economists and Economics Departments, 1990-2000," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1309-1345, December.
    18. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. My Year in Review 2013
      by David Stern in Stochastic Trend on 2013-12-24 10:20:00
    2. Citations Variance and Journal Ranking Uncertainty
      by David Stern in Stochastic Trend on 2013-09-28 17:45:00
    3. Figshare
      by noreply@blogger.com (David Stern) in Stochastic Trend on 2014-11-16 19:01:00
    4. Legitimate Uses for Impact Factors
      by noreply@blogger.com (David Stern) in Stochastic Trend on 2016-07-12 08:38:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    2. Martin Richards, 2013. "Putting the blue pencil away and taking down the name board," Transportation, Springer, vol. 40(6), pages 1087-1104, November.
    3. repec:spr:scient:v:113:y:2017:i:3:d:10.1007_s11192-017-2507-7 is not listed on IDEAS
    4. Wohlrabe, Klaus & Bornmann, Lutz, 2017. "Normalization of citation impact in economics," MPRA Paper 80384, University Library of Munich, Germany.
    5. Klaus Wohlrabe, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," CESifo Working Paper Series 5726, CESifo Group Munich.
    6. Lehmann, Robert & Wohlrabe, Klaus, 2017. "Who is the ‘Journal Grand Master’? A new ranking based on the Elo rating system," Journal of Informetrics, Elsevier, vol. 11(3), pages 800-809.
    7. Seiler, Christian & Wohlrabe, Klaus, 2014. "How robust are journal rankings based on the impact factor? Evidence from the economic sciences," Journal of Informetrics, Elsevier, vol. 8(4), pages 904-911.
    8. repec:eee:infome:v:11:y:2017:i:3:p:689-703 is not listed on IDEAS
    9. Jakob Kapeller & Matthias Aistleitner & Stefan Steinerberger, 2017. "Citation Patterns in Economics and Beyond: Assessing the Peculiarities of Economics from Two Scientometric Perspectives," ICAE Working Papers 60, Johannes Kepler University, Institute for Comprehensive Analysis of the Economy.
    10. Victoria Anauati & Sebastian Galiani & Ramiro H. Gálvez, 2016. "Quantifying The Life Cycle Of Scholarly Articles Across Fields Of Economic Research," Economic Inquiry, Western Economic Association International, vol. 54(2), pages 1339-1355, April.
    11. Imad A. Moosa, 2016. "A Critique of the Bucket Classification of Journals: The ABDC List as an Example," The Economic Record, The Economic Society of Australia, vol. 92(298), pages 448-463, September.
    12. repec:spr:scient:v:111:y:2017:i:3:d:10.1007_s11192-017-2351-9 is not listed on IDEAS
    13. William Horrace & Christopher Parmeter, 2016. "Accounting for Multiplicity in Inference on Economics Journal Rankings," Working Papers 2016-08, University of Miami, Department of Economics.
    14. Kosnik, Lea-Rachel, 2015. "What have economists been doing for the last 50 years? A text analysis of published academic research from 1960-2010," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 9, pages 1-38.
    15. repec:spr:scient:v:105:y:2015:i:2:d:10.1007_s11192-015-1747-7 is not listed on IDEAS
    16. Lea Kosnik, 2014. "What Have Economists Been Doing for the Last 50 Years? A Text Analysis of Published Academic Research from 1960-2010," Working Papers 1004, University of Missouri-St. Louis, Department of Economics, revised Apr 2015.
    17. repec:aea:jeclit:v:56:y:2018:i:1:p:115-56 is not listed on IDEAS
    18. David L. Anderson & John Tressler, 2016. "Citation-Capture Rates for Economics Journals: Do they Differ from Other Disciplines and Does it Matter?," Economic Papers, The Economic Society of Australia, vol. 35(1), pages 73-85, March.
    19. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    20. repec:spr:scient:v:113:y:2017:i:1:d:10.1007_s11192-017-2471-2 is not listed on IDEAS
    21. Kosteas, Vasilios D., 2015. "Journal impact factors and month of publication," Economics Letters, Elsevier, vol. 135(C), pages 77-79.
    22. Daniel S. Hamermesh, 2015. "Citations in Economics: Measurement, Uses and Impacts," NBER Working Papers 21754, National Bureau of Economic Research, Inc.

    More about this item

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:jeclit:v:51:y:2013:i:1:p:173-89. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Michael P. Albert). General contact details of provider: http://edirc.repec.org/data/aeaaaea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.