IDEAS home Printed from https://ideas.repec.org/p/lan/wpaper/602544.html
   My bibliography  Save this paper

The Assessment of Research Quality: Peer Review or Metrics?

Author

Listed:
  • J Taylor

Abstract

This paper investigates the extent to which the outcomes of the 2008 Research Assessment Exercise, determined by peer review, can be explained by a set of quantitative indicators, some of which were made available to the review panels. Three cognate units of assessment are examined in detail: business & management, economics & econometrics, and accounting & finance. The paper focuses on the extent to which the quality of research output, as determined by the RAE panel, can be explained by the journal quality indicator published by the Association of Business Schools. The main finding is that although a high proportion of the variation between universities in their RAE outcomes can be explained by quantitative indicators, there is insufficient evidence to support the claim by the ABS that its Journal Quality Guide is a sufficiently accurate predictor of research quality to justify a predominant role in the research assessment process. A further finding is that there appears to be an element of bias in the decisions reached by the business & management panel and by the economics & econometrics panel.

Suggested Citation

  • J Taylor, 2009. "The Assessment of Research Quality: Peer Review or Metrics?," Working Papers 602544, Lancaster University Management School, Economics Department.
    • Handle: RePEc:lan:wpaper:602544
    as

    Download full text from publisher

    File URL: http://www.lancaster.ac.uk/media/lancaster-university/content-assets/documents/lums/economics/working-papers/ResearchQuality.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Timothy Clark & Mike Wright, 2007. "Reviewing Journal Rankings and Revisiting Peer Reviews: Editorial Perspectives," Journal of Management Studies, Wiley Blackwell, vol. 44(4), pages 612-621, June.
    2. Jim Taylor, 1995. "A Statistical Analysis of the 1992 Research Assessment Exercise," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(2), pages 241-261, March.
    3. Jim Taylor, "undated". "Measuring Research Performance in Business Managment Studies in the United Kingdom: the 1992 Research Assessment Exercise," Working Papers ec15/94, Department of Economics, University of Lancaster.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. J Taylor & I Walker, 2009. "Peer assessment of research: how many publications per staff?," Working Papers 603570, Lancaster University Management School, Economics Department.
    2. repec:lan:wpaper:983 is not listed on IDEAS

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:lan:wpaper:1047 is not listed on IDEAS
    2. repec:lan:wpaper:984 is not listed on IDEAS
    3. Doyle, J. R. & Arthurs, A. J. & Green, R. H. & McAulay, L. & Pitt, M. R. & Bottomley, P. A. & Evans, W., 1996. "The judge, the model of the judge, and the model of the judged as judge: Analyses of the UK 1992 research assessment exercise data for business and management studies," Omega, Elsevier, vol. 24(1), pages 13-28, February.
    4. Hensel, Przemysław G., 2019. "Supporting replication research in management journals: Qualitative analysis of editorials published between 1970 and 2015," European Management Journal, Elsevier, vol. 37(1), pages 45-57.
    5. Michelle Greenwood, 2016. "Approving or Improving Research Ethics in Management Journals," Journal of Business Ethics, Springer, vol. 137(3), pages 507-520, September.
    6. Timothy Clark & Mike Wright, 2009. "So, Farewell Then . . . Reflections on Editing the Journal of Management Studies," Journal of Management Studies, Wiley Blackwell, vol. 46(1), pages 1-9, January.
    7. Wijnberg, Nachoem M., 2011. "Classification systems and selection systems: The risks of radical innovation and category spanning," Scandinavian Journal of Management, Elsevier, vol. 27(3), pages 297-306, September.
    8. Lakshmi Balachandran Nair, 2021. "From ‘Whodunit’ to ‘How’: Detective Stories and Auditability in Qualitative Business Ethics Research," Journal of Business Ethics, Springer, vol. 172(2), pages 195-209, August.
    9. Hussain, Simon, 2010. "Accounting journals and the ABS quality ratings," The British Accounting Review, Elsevier, vol. 42(1), pages 1-16.
    10. Tüselmann, Heinz & Sinkovics, Rudolf R. & Pishchulov, Grigory, 2015. "Towards a consolidation of worldwide journal rankings – A classification using random forests and aggregate rating via data envelopment analysis," Omega, Elsevier, vol. 51(C), pages 11-23.
    11. Gianni De Fraja & Giovanni Facchini & John Gathergood, 2016. "How Much Is That Star in the Window? Professorial Salaries and Research Performance in UK Universities," Discussion Papers 2016-13, University of Nottingham, GEP.
    12. Tanzila Ahmed & Ben Johnson & Charles Oppenheim & Catherine Peck, 2004. "Highly cited old papers and the reasons why they continue to be cited. Part II., The 1953 Watson and Crick article on the structure of DNA," Scientometrics, Springer;Akadémiai Kiadó, vol. 61(2), pages 147-156, October.
    13. Louis de Mesnard, 2014. "On the marketization of the academic review process. (VF) Sur la marchandisation du processus de referee des revues académiques," Working Papers CREGO 1141001, Université de Bourgogne - CREGO EA7317 Centre de recherches en gestion des organisations.
    14. Mike Wright, 2014. "Academic entrepreneurship, technology transfer and society: where next?," The Journal of Technology Transfer, Springer, vol. 39(3), pages 322-334, June.
    15. Luis Antonio Orozco Castro, 2015. "Diversidad y heterogeneidad en redes de colaboración científica. Un estudio de las escuelas de administración de América Latina," Books, Universidad Externado de Colombia, Facultad de Administración de Empresas, edition 1, number 44, August.
    16. Ken Starkey & Armand Hatchuel & Sue Tempest, 2009. "Management Research and the New Logics of Discovery and Engagement," Journal of Management Studies, Wiley Blackwell, vol. 46(3), pages 547-558, May.
    17. V. A. Traag & L. Waltman, 2019. "Systematic analysis of agreement between metrics and peer review in the UK REF," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    18. David Audretsch & Erik Lehmann & Mike Wright, 2014. "Technology transfer in a global economy," The Journal of Technology Transfer, Springer, vol. 39(3), pages 301-312, June.
    19. Vicente Safón, 2019. "Inter-ranking reputational effects: an analysis of the Academic Ranking of World Universities (ARWU) and the Times Higher Education World University Rankings (THE) reputational relationship," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 897-915, November.
    20. Oswald, Andrew J., 2008. "Can We Test for Bias in Scientific Peer-Review?," IZA Discussion Papers 3665, Institute of Labor Economics (IZA).
    21. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    22. Cappelletti-Montano, Beniamino & Columbu, Silvia & Montaldo, Stefano & Musio, Monica, 2022. "Interpreting the outcomes of research assessments: A geometrical approach," Journal of Informetrics, Elsevier, vol. 16(1).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:lan:wpaper:602544. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Giorgio Motta (email available below). General contact details of provider: https://edirc.repec.org/data/delanuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.