IDEAS home Printed from https://ideas.repec.org/a/taf/emetrv/v35y2016i1p50-97.html
   My bibliography  Save this article

Robust Ranking of Journal Quality: An Application to Economics

Author

Listed:
  • Chia-Lin Chang
  • Esfandiar Maasoumi
  • Michael McAleer

Abstract

The article focuses on the robustness of rankings of academic journal quality and research impact in general, and in economics, in particular, based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). The article analyzes 299 leading international journals in economics using quantifiable Research Assessment Measures (RAMs), and highlights the similarities and differences in various RAMs, which are based on alternative transformations of citations and influence. All existing RAMs to date have been static, so two new dynamic RAMs are developed to capture changes in impact factor over time and escalating journal self-citations. Alternative RAMs may be calculated annually or updated daily to determine When, Where, and How (frequently) published articles are cited (see Chang et al., 2011a-c). The RAMs are grouped in four distinct classes that include impact factor, mean citations, and non-citations, journal policy, number of high quality articles, journal influence, and article influence. These classes include the most widely used RAMs, namely, the classic 2-year impact factor including journal self-citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Eigenfactor (or Journal Influence), Article Influence, h-index, and Papers Ignored-By Even The Authors (PI-BETA). As all existing RAMs to date have been static, two new dynamic RAMs are developed to capture changes in impact factor over time (5YD2 = 5YIF/2YIF) and Escalating Self-Citations (ESC). We highlight robust rankings based on the harmonic mean of the ranks of RAMs across the four classes. It is shown that emphasizing the 2YIF of a journal, which partly answers the question as to When published articles are cited, to the exclusion of other informative RAMs, which answer Where and How (frequently) published articles are cited, can lead to a distorted evaluation of journal quality, impact, and influence relative to the harmonic mean of the ranks.

Suggested Citation

  • Chia-Lin Chang & Esfandiar Maasoumi & Michael McAleer, 2016. "Robust Ranking of Journal Quality: An Application to Economics," Econometric Reviews, Taylor & Francis Journals, vol. 35(1), pages 50-97, January.
    • Handle: RePEc:taf:emetrv:v:35:y:2016:i:1:p:50-97
    DOI: 10.1080/07474938.2014.956639
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/07474938.2014.956639
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1080/07474938.2014.956639?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Chia-Lin Chang & Michael McAleer, 2013. "Ranking journal quality by harmonic mean of ranks: an application to ISI statistics & probability," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 67(1), pages 27-53, February.
    2. Chia-Lin Chang & Michael McAleer & Les Oxley, 2011. "Great Expectatrics: Great Papers, Great Journals, Great Econometrics," Econometric Reviews, Taylor & Francis Journals, vol. 30(6), pages 583-619.
    3. Chia-Lin Chang & Michael McAleer & Les Oxley, 2011. "What makes a great journal great in the sciences? Which came first, the chicken or the egg?," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(1), pages 17-40, April.
    4. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    5. Chia‐Lin Chang & Michael McAleer & Les Oxley, 2011. "What Makes A Great Journal Great In Economics? The Singer Not The Song," Journal of Economic Surveys, Wiley Blackwell, vol. 25(2), pages 326-361, April.
    6. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    7. Chia-Lin Chang & Michael McAleer & Les Oxley, 2011. "How are journal impact, prestige and article influence related? An application to neuroscience," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(11), pages 2563-2573, January.
    8. J. Peter Neary & James A. Mirrlees & Jean Tirole, 2003. "Evaluating Economics Research in Europe: An Introduction," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1239-1249, December.
    9. Kostas Axarloglou & Vasilis Theoharakis, 2003. "Diversity in Economics: An Analysis of Journal Quality Perceptions," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1402-1423, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chia-Lin Chang & Michael Mcaleer, 2014. "Just How Good Are The Top Three Journals In Finance? An Assessment Based On Quantity And Quality Citations," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 9(01), pages 1-31.
    2. Chang, Chia-Lin & McAleer, Michael & Oxley, Les, 2013. "Coercive journal self citations, impact factor, Journal Influence and Article Influence," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 93(C), pages 190-197.
    3. Dejian Yu & Wanru Wang & Shuai Zhang & Wenyu Zhang & Rongyu Liu, 2017. "A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 521-542, April.
    4. Yu, Tian & Yu, Guang & Wang, Ming-Yang, 2014. "Classification method for detecting coercive self-citation in journals," Journal of Informetrics, Elsevier, vol. 8(1), pages 123-135.
    5. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    6. Klaus Wohlrabe, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," CESifo Working Paper Series 5726, CESifo.
    7. Chia-Lin Chang & Michael Mcaleer, 2013. "What Do Experts Know About Forecasting Journal Quality? A Comparison With Isi Research Impact In Finance," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 8(01), pages 1-30.
    8. Chia-Lin Chang & Michael McAleer, 2013. "Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc," Econometrics, MDPI, vol. 1(3), pages 1-19, November.
    9. Chia-Lin Chang & Michael McAleer & Les Oxley, 2012. "Journal Impact Factor, Eigenfactor, Journal Influence and Article Influence," Documentos de Trabajo del ICAE 2012-15, Universidad Complutense de Madrid, Facultad de Ciencias Económicas y Empresariales, Instituto Complutense de Análisis Económico.
    10. Chia-Lin Chang & Michael McAleer, 2014. "Quality Weighted Citations Versus Total Citations in the Sciences and Social Sciences," Documentos de Trabajo del ICAE 2014-06, Universidad Complutense de Madrid, Facultad de Ciencias Económicas y Empresariales, Instituto Complutense de Análisis Económico.
    11. Chang Chia-Lin & McAleer Michael, 2014. "Ranking Economics and Econometrics ISI Journals by Quality Weighted Citations," Review of Economics, De Gruyter, vol. 65(1), pages 35-52, April.
    12. Chia-Lin Chang & Michael McAleer, "undated". "Quality Weighted Citations Versus Total Citations in the Sciences and Social Sciences, with an Application to Finance and Accounting," Documentos de Trabajo del ICAE 2015-01, Universidad Complutense de Madrid, Facultad de Ciencias Económicas y Empresariales, Instituto Complutense de Análisis Económico, revised Jan 2015.
    13. Chia-Lin Chang & Michael McAleer, 2013. "Ranking journal quality by harmonic mean of ranks: an application to ISI statistics & probability," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 67(1), pages 27-53, February.
    14. McAleer, M.J. & Oláh, J. & Popp, J., 2018. "Pros and Cons of the Impact Factor in a Rapidly Changing Digital World," Econometric Institute Research Papers EI2018-11, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    15. William C. Horrace & Christopher F. Parmeter, 2017. "Accounting for Multiplicity in Inference on Economics Journal Rankings," Southern Economic Journal, John Wiley & Sons, vol. 84(1), pages 337-347, July.
    16. Franklin G. Mixon, Jr. & Kamal P. Upadhyaya, 2021. "Scholarly Impact of Core Econometrics Journals: A Catalog and Citations-Based Ranking," International Econometric Review (IER), Econometric Research Association, vol. 13(4), pages 118-131, December.
    17. Francesco Bartolucci & Valentino Dardanoni & Franco Peracchi, 2015. "Ranking scientific journals via latent class models for polytomous item response data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 1025-1049, October.
    18. Chia-Lin Chang & Michael McAleer, 2015. "Bibliometric Rankings of Journals Based on the Thomson Reuters Citations Database," Journal of Reviews on Global Economics, Lifescience Global, vol. 4, pages 120-125.
    19. Polterovich, Victor, 2022. "Библиометрическое Равновесие [Bibliometric Equilibrium]," MPRA Paper 111802, University Library of Munich, Germany.
    20. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    21. Chia-Lin Chang & Michael McAleer, 2019. "Modeling Latent Carbon Emission Prices for Japan: Theory and Practice," Energies, MDPI, vol. 12(21), pages 1-21, November.
    22. Konstantinos Metaxoglou, 2021. "Canadian Journal of Economics: A historic overview," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 54(3), pages 1418-1453, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    2. Chia-Lin Chang & Michael Mcaleer, 2014. "Just How Good Are The Top Three Journals In Finance? An Assessment Based On Quantity And Quality Citations," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 9(01), pages 1-31.
    3. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    4. Chia-Lin Chang & Michael Mcaleer, 2013. "What Do Experts Know About Forecasting Journal Quality? A Comparison With Isi Research Impact In Finance," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 8(01), pages 1-30.
    5. Chia-Lin Chang & Michael McAleer, 2015. "Bibliometric Rankings of Journals Based on the Thomson Reuters Citations Database," Journal of Reviews on Global Economics, Lifescience Global, vol. 4, pages 120-125.
    6. Chang, C-L. & McAleer, M.J. & Oxley, L., 2012. "Journal Impact Factor, Eigenfactor, Journal Influence and Article Influence," Econometric Institute Research Papers EI 2012-27, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    7. Chang, Chia-Lin & McAleer, Michael & Oxley, Les, 2013. "Coercive journal self citations, impact factor, Journal Influence and Article Influence," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 93(C), pages 190-197.
    8. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    9. Chia-Lin Chang & Michael McAleer, 2013. "Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc," Econometrics, MDPI, vol. 1(3), pages 1-19, November.
    10. Chia-Lin Chang & Michael McAleer, "undated". "Quality Weighted Citations Versus Total Citations in the Sciences and Social Sciences, with an Application to Finance and Accounting," Documentos de Trabajo del ICAE 2015-01, Universidad Complutense de Madrid, Facultad de Ciencias Económicas y Empresariales, Instituto Complutense de Análisis Económico, revised Jan 2015.
    11. Jan Ours & Frederic Vermeulen, 2007. "Ranking Dutch Economists," De Economist, Springer, vol. 155(4), pages 469-487, December.
    12. Chia-Lin Chang & Michael McAleer, 2019. "Modeling Latent Carbon Emission Prices for Japan: Theory and Practice," Energies, MDPI, vol. 12(21), pages 1-21, November.
    13. Chang, C-L. & McAleer, M.J., 2012. "What do Experts Know About Ranking Journal Quality? A Comparison with ISI Research Impact in Finance," Econometric Institute Research Papers EI2012-01, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    14. Chia-Lin Chang & Michael McAleer, 2014. "How Should Journal Quality be Ranked? An Application to Agricultural, Energy, Environmental and Resource Economics," Journal of Reviews on Global Economics, Lifescience Global, vol. 3, pages 33-47.
    15. Chia-Lin Chang & Michael McAleer, 2013. "Ranking journal quality by harmonic mean of ranks: an application to ISI statistics & probability," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 67(1), pages 27-53, February.
    16. Mirucki, Jean & Nicot, Bernadette & Poshyvak, Maria, 2007. "What Can EconLit Reveal Us About Ukraine's Scholarly Production?," MPRA Paper 29089, University Library of Munich, Germany.
    17. Frank Neri & Joan R. Rodgers, 2006. "Ranking Australian Economics Departments by Research Productivity," The Economic Record, The Economic Society of Australia, vol. 82(s1), pages 74-84, September.
    18. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    19. Rodgers, Joan R. & Valadkhani, Abbas, 2005. "Ranking of Australian Economics Departments Based on Their Total and Per Academic Staff Research Output," Economics Working Papers wp05-18, School of Economics, University of Wollongong, NSW, Australia.
    20. Silvia Ferrini, 2007. "L’Impact Factor: luci e ombre," Department of Economic Policy, Finance and Development (DEPFID) University of Siena 002, Department of Economic Policy, Finance and Development (DEPFID), University of Siena.

    More about this item

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • Y10 - Miscellaneous Categories - - Data: Tables and Charts - - - Data: Tables and Charts

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:emetrv:v:35:y:2016:i:1:p:50-97. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: http://www.tandfonline.com/LECR20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.