IDEAS home Printed from https://ideas.repec.org/p/een/crwfrp/1302.html
   My bibliography  Save this paper

Uncertainty Measures for Economics Journal Impact Factors

Author

Listed:
  • David I. Stern

    (Crawford School of Public Policy, The Australian National University Author-Email: david.stern@anu.edu.au)

Abstract

Academic economists appear to be intensely interested in rankings of journals, institutions, and individuals. Yet there is little discussion of the uncertainty associated with these rankings. To illustrate the uncertainty associated with citations-based rankings, I compute the standard error of the impact factor for all economics journals with a five-year impact factor in the 2011 Journal Citations Report. I use these to derive confidence intervals for the impact factors as well as ranges of possible rank for a subset of thirty journals. I find that the impact factors of the top two journals are well defined and set these journals apart in a clearly defined group. An elite group of 9-11 mainstream journals can also be fairly reliably distinguished. The four bottom ranked journals are also fairly clearly set apart. For the remainder of the distribution, confidence intervals overlap and rankings are quite uncertain.

Suggested Citation

  • David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Crawford School Research Papers 1302, Crawford School of Public Policy, The Australian National University.
    • Handle: RePEc:een:crwfrp:1302
    as

    Download full text from publisher

    File URL: http://ssrn.com/abstract=2199162
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. David N. Laband & Robert D. Tollison, 2003. "Dry Holes in Economic Research," Kyklos, Wiley Blackwell, vol. 56(2), pages 161-173, May.
    2. Loet Leydesdorff & Tobias Opthof, 2010. "Scopus's source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(11), pages 2365-2369, November.
    3. S. Redner, 1998. "How popular is your paper? An empirical study of the citation distribution," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 4(2), pages 131-134, July.
    4. Wolfgang Glänzel, 2009. "The multi-dimensionality of journal impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(2), pages 355-374, February.
    5. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    6. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    7. Ma, Chunbo & Stern, David I., 2006. "Environmental and ecological economics: A citation analysis," Ecological Economics, Elsevier, vol. 58(3), pages 491-506, June.
    8. Michael J Stringer & Marta Sales-Pardo & Luís A Nunes Amaral, 2008. "Effectiveness of Journal Ranking Schemes as a Tool for Locating Information," PLOS ONE, Public Library of Science, vol. 3(2), pages 1-8, February.
    9. Mario de Marchi & Maurizio Rocchi, 2001. "The editorial policies of scientific journals: Testing an impact factor model," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(2), pages 395-404, June.
    10. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    11. Burton, M P & Phimister, Euan, 1995. "Core Journals: A Reappraisal of the Diamond List," Economic Journal, Royal Economic Society, vol. 105(429), pages 361-373, March.
    12. Laband, David N & Piette, Michael J, 1994. "The Relative Impacts of Economics Journals: 1970-1990," Journal of Economic Literature, American Economic Association, vol. 32(2), pages 640-666, June.
    13. Liebowitz, S J & Palmer, J P, 1984. "Assessing the Relative Impacts of Economic Journals," Journal of Economic Literature, American Economic Association, vol. 22(1), pages 77-88, March.
    14. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    15. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Ryan Chute, 2009. "A Principal Component Analysis of 39 Scientific Impact Measures," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-11, June.
    16. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    17. Henk F. Moed & Lisa Colledge & Jan Reedijk & Felix Moya-Anegon & Vicente Guerrero-Bote & Andrew Plume & Mayur Amin, 2012. "Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 367-376, August.
    18. Richard Dusansky & Clayton J. Vernon, 1998. "Rankings of U.S. Economics Departments," Journal of Economic Perspectives, American Economic Association, vol. 12(1), pages 157-170, Winter.
    19. A. I. Pudovkin & Eugene Garfield, 2012. "Rank normalization of impact factors will resolve Vanclay’s dilemma with TRIF," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 409-412, August.
    20. Per O. Seglen, 1992. "The skewness of science," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 43(9), pages 628-638, October.
    21. Anthony F. J. Raan, 2012. "Properties of journal impact in relation to bibliometric research group performance indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 457-469, August.
    22. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    23. Tom Coupé, 2003. "Revealed Performances: Worldwide Rankings of Economists and Economics Departments, 1990-2000," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1309-1345, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joseph Macri & Dipendra Sinha, 2006. "Rankings Methodology for International Comparisons of Institutions and Individuals: an Application to Economics in Australia and New Zealand," Journal of Economic Surveys, Wiley Blackwell, vol. 20(1), pages 111-156, February.
    2. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    3. David Anderson & John Tressler, 2008. "Research output in New Zealand economics departments 2000-2006: A stock approach," New Zealand Economic Papers, Taylor & Francis Journals, vol. 42(2), pages 155-189.
    4. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    5. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    6. Klaus Ritzberger, 2008. "Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 267-285, August.
    7. Ma, Chunbo & Stern, David I., 2006. "Environmental and ecological economics: A citation analysis," Ecological Economics, Elsevier, vol. 58(3), pages 491-506, June.
    8. Koczy, L. & Nichifor, A. & Strobel, M., 2010. "Intellectual Influence: quality versus quantity," Research Memorandum 029, Maastricht University, Maastricht Research School of Economics of Technology and Organization (METEOR).
    9. Johannes König & David I. Stern & Richard S.J. Tol, 2022. "Confidence Intervals for Recursive Journal Impact Factors," Tinbergen Institute Discussion Papers 22-038/III, Tinbergen Institute.
    10. Chagas , André Luis Squarize, 2017. "Publish or Perish: um ranking de revistas da subárea de Economia Regional e Urbana para os pesquisadores brasileiros," Revista Brasileira de Estudos Regionais e Urbanos, Associação Brasileira de Estudos Regionais e Urbanos (ABER), vol. 11(4), pages 515-536.
    11. Konstantinos Chatzimichael & Pantelis Kalaitzidakis & Vangelis Tzouvelekas, 2017. "Measuring the publishing productivity of economics departments in Europe," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 889-908, November.
    12. Javier Ruiz-Castillo, 2008. "Economics research in Spain during the 1990s: a literature review," Spanish Economic Review, Springer;Spanish Economic Association, vol. 10(3), pages 221-249, September.
    13. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    14. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "Zeitschriftenrankings für die Wirtschaftswissenschaften – Konstruktion eines umfassenden Metaindexes," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 286-305, August.
    15. Martin Gregor, 2006. "Hodnocení ekonomických pracovišť a ekonomů: Koho, proč, čím a jak [A survey of rankings of economic departments: Global, american, european and national]," Politická ekonomie, Prague University of Economics and Business, vol. 2006(3), pages 394-414.
    16. Joseph Macri & Michael McAleer & Dipendra Sinha, 2009. "On the Robustness of Alternative Rankings Methodologies For Australian and New Zealand Economics Departments," CIRJE F-Series CIRJE-F-660, CIRJE, Faculty of Economics, University of Tokyo.
    17. Howard J. Wall, 2009. "Journal rankings in economics: handle with care," Working Papers 2009-014, Federal Reserve Bank of St. Louis.
    18. Yu, Xiaohua & Gao, Zhifeng, 2010. "An updated ranking of the economic research institutions in China (2000-2009)," China Economic Review, Elsevier, vol. 21(4), pages 571-581, December.
    19. David L. Anderson & John Tressler, 2010. "The Merits of Using Citation-Based Journal Weighting Schemes to Measure Research Performance in Economics: The Case of New Zealand," Working Papers in Economics 10/03, University of Waikato.
    20. Melody Lo & M. C. Sunny Wong & Franklin G. Mixon, 2008. "Ranking Economics Journals, Economics Departments, and Economists Using Teaching‐Focused Research Productivity," Southern Economic Journal, John Wiley & Sons, vol. 74(3), pages 894-906, January.

    More about this item

    Keywords

    academic; journals; economics;
    All these keywords.

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:een:crwfrp:1302. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: David Stern (email available below). General contact details of provider: https://edirc.repec.org/data/asanuau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.