IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v91y2012i3d10.1007_s11192-012-0618-8.html
   My bibliography  Save this article

Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

Author

Listed:
  • Jerome K. Vanclay

    (Southern Cross University)

  • Lutz Bornmann

    (Administrative Headquarters)

Abstract

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.

Suggested Citation

  • Jerome K. Vanclay & Lutz Bornmann, 2012. "Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 751-771, June.
  • Handle: RePEc:spr:scient:v:91:y:2012:i:3:d:10.1007_s11192-012-0618-8
    DOI: 10.1007/s11192-012-0618-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-012-0618-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11192-012-0618-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. L. Egghe, 2011. "The single publication H-index and the indirect H-index of a researcher," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1003-1004, September.
    2. Loet Leydesdorff & Lutz Bornmann & Rüdiger Mutz & Tobias Opthof, 2011. "Turning the tables on citation analysis one more time: Principles for comparing sets of documents," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(7), pages 1370-1381, July.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "Evaluating research: from informed peer review to bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 499-514, June.
    4. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    5. Jacob B. Slyder & Beth R. Stein & Brent S. Sams & David M. Walker & B. Jacob Beale & Jeffrey J. Feldhaus & Carolyn A. Copenheaver, 2011. "Citation pattern and lifespan: a comparison of discipline, institution, and individual," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 955-966, December.
    6. Deryl Northcott & Simon Linacre, 2010. "Producing Spaces for Academic Discourse: The Impact of Research Assessment Exercises and Journal Quality Rankings," Australian Accounting Review, CPA Australia, vol. 20(1), pages 38-54, March.
    7. Imad Moosa, 2011. "The demise of the ARC journal ranking scheme: an ex post analysis of the accounting and finance journals," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 51(3), pages 809-836, September.
    8. Vanclay, Jerome K., 2011. "An evaluation of the Australian Research Council's journal ranking," Journal of Informetrics, Elsevier, vol. 5(2), pages 265-274.
    9. David Pontille & Didier Torny, 2010. "The controversial policies of journal ratings: evaluating social sciences and humanities," Research Evaluation, Oxford University Press, vol. 19(5), pages 347-360, December.
    10. András Schubert, 2009. "Using the h-index for assessing single publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(3), pages 559-565, March.
    11. Vanclay, Jerome K., 2008. "Ranking forestry journals using the h-index," Journal of Informetrics, Elsevier, vol. 2(4), pages 326-334.
    12. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Caprasecca, Alessandro, 2009. "Allocative efficiency in public research funding: Can bibliometrics help?," Research Policy, Elsevier, vol. 38(1), pages 206-215, February.
    13. Lutz Bornmann & Hermann Schier & Werner Marx & Hans-Dieter Daniel, 2011. "Does the h index for assessing single publications really work? A case study on papers published in chemistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 835-843, December.
    14. Claire Donovan & Linda Butler, 2007. "Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: an economics pilot study," Research Evaluation, Oxford University Press, vol. 16(4), pages 231-242, December.
    15. Leo Egghe, 2008. "Modelling successive h-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(3), pages 377-387, December.
    16. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    17. Frances Ruane & Richard S. J. Tol, 2008. "Rational (successive) h-indices: An application to economics in the Republic of Ireland," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(2), pages 395-405, May.
    18. Félix Moya-Anegón & Zaida Chinchilla-Rodríguez & Benjamín Vargas-Quesada & Elena Corera-Álvarez & Francisco José Muñoz-Fernández & Antonio González-Molina & Victor Herrero-Solana, 2007. "Coverage analysis of Scopus: A journal metric approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 73(1), pages 53-78, October.
    19. András Schubert, 2007. "Successive h-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 70(1), pages 201-205, January.
    20. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & Nicolas Robinson-García & Daniel Torres-Salinas, 2013. "Benchmarking research performance at the university level with information theoretic measures," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 435-452, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    2. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    3. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    4. Giovanni Anania & Annarosa Caruso, 2013. "Two simple new bibliometric indexes to better evaluate research in disciplines where publications typically receive less citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 617-631, August.
    5. Leo Egghe & Ronald Rousseau, 2021. "The h-index formalism," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6137-6145, July.
    6. Vanclay, Jerome K., 2011. "An evaluation of the Australian Research Council's journal ranking," Journal of Informetrics, Elsevier, vol. 5(2), pages 265-274.
    7. Nicholas McGuigan, 2015. "The Impact of Journal Rankings on Australasian Accounting Education Scholarship - A Personal View," Accounting Education, Taylor & Francis Journals, vol. 24(3), pages 187-207, June.
    8. Nikolaos A. Kazakis, 2014. "Bibliometric evaluation of the research performance of the Greek civil engineering departments in National and European context," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 505-525, October.
    9. Cabrerizo, F.J. & Alonso, S. & Herrera-Viedma, E. & Herrera, F., 2010. "q2-Index: Quantitative and qualitative evaluation based on the number and impact of papers in the Hirsch core," Journal of Informetrics, Elsevier, vol. 4(1), pages 23-28.
    10. Judit Bar-Ilan & Mark Levene, 2015. "The hw-rank: an h-index variant for ranking web pages," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2247-2253, March.
    11. Franc Mali, 2013. "Why an Unbiased External R&D Evaluation System is Important for the Progress of Social Sciences—the Case of a Small Social Science Community," Social Sciences, MDPI, Open Access Journal, vol. 2(4), pages 1-14, December.
    12. Yu Liu & Wei Zuo & Ying Gao & Yanhong Qiao, 2013. "Comprehensive geometrical interpretation of h-type indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 605-615, August.
    13. Nan Zhang & Shanshan Wan & Peiling Wang & Peng Zhang & Qiang Wu, 2018. "A bibliometric analysis of highly cited papers in the field of Economics and Business based on the Essential Science Indicators database," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 1039-1053, August.
    14. Petridis, Konstantinos & Malesios, Chrisovalantis & Arabatzis, Garyfallos & Thanassoulis, Emmanuel, 2013. "Efficiency analysis of forestry journals: Suggestions for improving journals’ quality," Journal of Informetrics, Elsevier, vol. 7(2), pages 505-521.
    15. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    16. Parreiras, R.O. & Kokshenev, I. & Carvalho, M.O.M. & Willer, A.C.M. & Dellezzopolles, C.F. & Nacif, D.B. & Santana, J.A., 2019. "A flexible multicriteria decision-making methodology to support the strategic management of Science, Technology and Innovation research funding programs," European Journal of Operational Research, Elsevier, vol. 272(2), pages 725-739.
    17. Miguel A. García-Pérez, 2009. "A multidimensional extension to Hirsch’s h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 779-785, December.
    18. Peter Jacso, 2012. "Grim tales about the impact factor and the h-index in the Web of Science and the Journal Citation Reports databases: reflections on Vanclay’s criticism," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 325-354, August.
    19. Kaur, Jasleen & Ferrara, Emilio & Menczer, Filippo & Flammini, Alessandro & Radicchi, Filippo, 2015. "Quality versus quantity in scientific impact," Journal of Informetrics, Elsevier, vol. 9(4), pages 800-808.
    20. Ricardo Arencibia-Jorge & Ronald Rousseau, 2009. "Influence of individual researchers’ visibility on institutional impact: an example of Prathap’s approach to successive h-indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(3), pages 507-516, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:91:y:2012:i:3:d:10.1007_s11192-012-0618-8. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.springer.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.