IDEAS home Printed from https://ideas.repec.org/p/esx/essedp/14461.html

A Tale of Two Metrics: Research Assessment vs Recognised Excellence

Author

Listed:
  • Régibeau, P
  • Rockett, K

Abstract

We build an economics department entirely composed of Nobel Prize winners and evaluate it using standard research assessment metrics. Performing the same evaluation on existing departments, we find that the rating of our Nobel Prize department does not stand out from other departments. Compared to recent research evaluations, our Nobel Prize department's ranking is less stable. This suggests a significant effect of score "targeting" induced by the ranking exercise. We find some evidence that modifying the assessment criteria to increase the totatl number of publications considered can help distinguish the top.

Suggested Citation

  • Régibeau, P & Rockett, K, 2014. "A Tale of Two Metrics: Research Assessment vs Recognised Excellence," Economics Discussion Papers 14461, University of Essex, Department of Economics.
    • Handle: RePEc:esx:essedp:14461
    as

    Download full text from publisher

    File URL: https://repository.essex.ac.uk/14461/
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Sofronis Clerides & Panos Pashardes & Alexandros Polycarpou, 2006. "RAE Ratings and Research Quality: The Case of Economics Departments," University of Cyprus Working Papers in Economics 7-2006, University of Cyprus Department of Economics.
    2. Joshua S. Gans & George B. Shepherd, 1994. "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 165-179, Winter.
    3. Margit Osterloh & Bruno S. Frey, 2015. "Ranking Games," Evaluation Review, , vol. 39(1), pages 102-129, February.
    4. David N. Laband, 2013. "On the Use and Abuse of Economics Journal Rankings," Economic Journal, Royal Economic Society, vol. 0, pages 223-254, August.
    5. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    6. John Hudson, 2013. "Ranking Journals," Economic Journal, Royal Economic Society, vol. 0, pages 202-222, August.
    7. Sofronis Clerides & Panos Pashardes & Alexandros Polycarpou, 2011. "Peer Review vs Metric‐based Assessment: Testing for Bias in the RAE Ratings of UK Economics Departments," Economica, London School of Economics and Political Science, vol. 78(311), pages 565-583, July.
    8. Daniel Sgroi & Andrew J. Oswald, 2013. "How Should Peer‐review Panels Behave?," Economic Journal, Royal Economic Society, vol. 0, pages 255-278, August.
    9. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Caprasecca, Alessandro, 2009. "Allocative efficiency in public research funding: Can bibliometrics help?," Research Policy, Elsevier, vol. 38(1), pages 206-215, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stephan B. Bruns & David I. Stern, 2016. "Research assessment using early citation information," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 917-935, August.
    2. Oswald, Andrew J., 2015. "The Objective Measurement of World-Leading Research," IZA Discussion Papers 8829, Institute of Labor Economics (IZA).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:esx:essedp:757 is not listed on IDEAS
    2. Erich Battistin & Marco Ovidi, 2017. "Rising Stars," Working Papers 843, Queen Mary University of London, School of Economics and Finance.
    3. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    4. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    5. Stephan B. Bruns & David I. Stern, 2016. "Research assessment using early citation information," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 917-935, August.
    6. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    7. Lyudmyla Shkulipa, 2021. "Evaluation of accounting journals by coverage of accounting topics in 2018–2019," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7251-7327, September.
    8. Oswald, Andrew J., 2015. "The Objective Measurement of World-Leading Research," IZA Discussion Papers 8829, Institute of Labor Economics (IZA).
    9. Daniel Sgroi & Andrew J. Oswald, 2013. "How Should Peer‐review Panels Behave?," Economic Journal, Royal Economic Society, vol. 0, pages 255-278, August.
    10. David I Stern, 2014. "High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-11, November.
    11. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    12. Margit Osterloh & Bruno S. Frey, 2010. "Academic rankings and research governance," IEW - Working Papers 482, Institute for Empirical Research in Economics - University of Zurich.
    13. Stelios Katranidis & Theodore Panagiotidis & Costas Zontanos, 2014. "An Evaluation Of The Greek Universities’ Economics Departments," Bulletin of Economic Research, Wiley Blackwell, vol. 66(2), pages 173-182, April.
    14. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    15. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    16. Stelios Katranidis & Theodore Panagiotidis & Costas Zontanos, 2017. "Economists, Research Performance and National Inbreeding: North Versus South," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 46(1), pages 145-163, February.
    17. Vasilios D. Kosteas, 2018. "Predicting long-run citation counts for articles in top economics journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1395-1412, June.
    18. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    19. Csaba, László, 2014. "Átmenettan és közgazdaságtan. Módszertani tanulságok egy részterület műveléséből [Transitology" and economics. Methodological lessons to be drawn from work in a partial territory]," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(1), pages 53-67.
    20. David L. Anderson & John Tressler, 2016. "Citation-Capture Rates for Economics Journals: Do they Differ from Other Disciplines and Does it Matter?," Economic Papers, The Economic Society of Australia, vol. 35(1), pages 73-85, March.
    21. Maria Rosaria Carillo & Erasmo Papagni & Alessandro Sapio, 2012. "Do collaborations enhance the high-quality output of scientific institutions? Evidence from the Italian Research Assessment Exercise (2001-2003)," Discussion Papers 4_2012, CRISEI, University of Naples "Parthenope", Italy.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:esx:essedp:14461. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Essex Economics Web Manager The email address of this maintainer does not seem to be valid anymore. Please ask Essex Economics Web Manager to update the entry or send us the correct address (email available below). General contact details of provider: https://edirc.repec.org/data/edessuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.