IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0168533.html
   My bibliography  Save this article

Beyond Academia – Interrogating Research Impact in the Research Excellence Framework

Author

Listed:
  • Emma Terämä
  • Melanie Smallman
  • Simon J Lock
  • Charlotte Johnson
  • Martin Zaltz Austwick

Abstract

Big changes to the way in which research funding is allocated to UK universities were brought about in the Research Excellence Framework (REF), overseen by the Higher Education Funding Council, England. Replacing the earlier Research Assessment Exercise, the purpose of the REF was to assess the quality and reach of research in UK universities–and allocate funding accordingly. For the first time, this included an assessment of research ‘impact’, accounting for 20% of the funding allocation. In this article we use a text mining technique to investigate the interpretations of impact put forward via impact case studies in the REF process. We find that institutions have developed a diverse interpretation of impact, ranging from commercial applications to public and cultural engagement activities. These interpretations of impact vary from discipline to discipline and between institutions, with more broad-based institutions depicting a greater variety of impacts. Comparing the interpretations with the score given by REF, we found no evidence of one particular interpretation being more highly rewarded than another. Importantly, we also found a positive correlation between impact score and [overall research] quality score, suggesting that impact is not being achieved at the expense of research excellence.

Suggested Citation

  • Emma Terämä & Melanie Smallman & Simon J Lock & Charlotte Johnson & Martin Zaltz Austwick, 2016. "Beyond Academia – Interrogating Research Impact in the Research Excellence Framework," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-18, December.
  • Handle: RePEc:plo:pone00:0168533
    DOI: 10.1371/journal.pone.0168533
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0168533
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0168533&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0168533?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Steven Wooding & Thed N Van Leeuwen & Sarah Parks & Shitij Kapur & Jonathan Grant, 2015. "UK Doubles Its “World-Leading” Research in Life Sciences and Medicine in Six Years: Testing the Claim?," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-10, July.
    2. Teresa Penfield & Matthew J. Baker & Rosa Scoble & Michael C. Wykes, 2013. "Assessment, evaluations, and definitions of research impact: A review," Research Evaluation, Oxford University Press, vol. 23(1), pages 21-32, October.
    3. Lisa Klautzer & Stephen Hanney & Edward Nason & Jennifer Rubin & Jonathan Grant & Steven Wooding, 2011. "Assessing policy and practice impacts of social science research: the application of the Payback Framework to assess the Future of Work programme," Research Evaluation, Oxford University Press, vol. 20(3), pages 201-209, September.
    4. Creso M. Sá & Andrew Kretz & Kristjan Sigurdson, 2013. "Accountability, performance assessment, and evaluation: Policy pressures and responses from research councils," Research Evaluation, Oxford University Press, vol. 22(2), pages 105-117, January.
    5. Sarah Bell & Ben Shaw & Annette Boaz, 2011. "Real-world approaches to assessing the impact of environmental research on policy," Research Evaluation, Oxford University Press, vol. 20(3), pages 227-237, September.
    6. David Bholat & Stephen Hans & Pedro Santos & Cheryl Schonhardt-Bailey, 2015. "Text mining for central banks," Handbooks, Centre for Central Banking Studies, Bank of England, number 33, April.
    7. J Lomas, 1990. "Finding Audiences, Changing Beliefs: The Structure of Research Use in Canadian Health Policy," Centre for Health Economics and Policy Analysis Working Paper Series 1990-10, Centre for Health Economics and Policy Analysis (CHEPA), McMaster University, Hamilton, Canada.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Levine, Kendra K., 2019. "New Metrics for Measuring Academic Research Outside the Ivory Tower," Institute of Transportation Studies, Research Reports, Working Papers, Proceedings qt05s6857s, Institute of Transportation Studies, UC Berkeley.
    2. Kayvan Kousha & Emma Stuart & Mahshid Abdoli & Mike Thelwall, 2024. "How do museums and galleries help academics create societal impact? An analysis of the UK REF2021 impact case studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(12), pages 7759-7782, December.
    3. Han Zheng & L. G. Pee & Dan Zhang, 2021. "Societal impact of research: a text mining study of impact types," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7397-7417, September.
    4. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    2. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    3. Aaryan Gupta & Vinya Dengre & Hamza Abubakar Kheruwala & Manan Shah, 2020. "Comprehensive review of text-mining applications in finance," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 6(1), pages 1-25, December.
    4. Fraccaroli, Nicolò & Giovannini, Alessandro & Jamet, Jean-François & Persson, Eric, 2022. "Ideology and monetary policy. The role of political parties’ stances in the European Central Bank’s parliamentary hearings," European Journal of Political Economy, Elsevier, vol. 74(C).
    5. Martin Baumgaertner & Johannes Zahner, 2021. "Whatever it takes to understand a central banker - Embedding their words using neural networks," MAGKS Papers on Economics 202130, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    6. Paola Cerchiello & Giancarlo Nicola, 2018. "Assessing News Contagion in Finance," Econometrics, MDPI, vol. 6(1), pages 1-19, February.
    7. Flood, M. D. & Jagadish, H. V. & Raschid, L., 2016. "Big data challenges and opportunities in financial stability monitoring," Financial Stability Review, Banque de France, issue 20, pages 129-142, April.
    8. Youngjoon Lee & Soohyon Kim & Ki Young Park, 2018. "Deciphering Monetary Policy Committee Minutes with Text Mining Approach: A Case of South Korea," Working papers 2018rwp-132, Yonsei University, Yonsei Economics Research Institute.
    9. Johannes Zahner, 2020. "Above, but close to two percent. Evidence on the ECB’s inflation target using text mining," MAGKS Papers on Economics 202046, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    10. Stefan Angrick & Naoyuki Yoshino, 2020. "From Window Guidance to Interbank Rates: Tracing the Transition of Monetary Policy in Japan and China," International Journal of Central Banking, International Journal of Central Banking, vol. 16(3), pages 279-316, June.
    11. Ashok Banerjee & Ayush Kanodia & Partha Ray, 2021. "Deciphering Indian inflationary expectations through text mining: an exploratory approach," Indian Economic Review, Springer, vol. 56(1), pages 49-66, June.
    12. Leonardo N. Ferreira, 2021. "Forecasting with VAR-teXt and DFM-teXt Models:exploring the predictive power of central bank communication," Working Papers Series 559, Central Bank of Brazil, Research Department.
    13. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    14. Jon Ellingsen & Vegard H. Larsen & Leif Anders Thorsrud, 2020. "News Media vs. FRED-MD for Macroeconomic Forecasting," CESifo Working Paper Series 8639, CESifo.
    15. Strzałkowski, Andrzej, 2024. "Adaptation and operationalisation of sustainable degrowth for policy: Why we need to translate research papers into legislative drafts?," Ecological Economics, Elsevier, vol. 220(C).
    16. Carol L. McWilliam, 1997. "Using a Participatory Research Process to Make a Difference in Policy on Aging," Canadian Public Policy, University of Toronto Press, vol. 23(s1), pages 70-89, Spring.
    17. Kurowski, Łukasz & Smaga, Paweł, 2023. "Analysing financial stability reports as crisis predictors with the use of text-mining," The Journal of Economic Asymmetries, Elsevier, vol. 28(C).
    18. R. Erasmus & H. Hollander, 2020. "A Forward Guidance Indicator For The South African Reserve Bank: Implementing A Text Analysis Algorithm," Studies in Economics and Econometrics, Taylor & Francis Journals, vol. 44(3), pages 41-72, December.
    19. Philip ME Garboden, 2019. "Sources and Types of Big Data for Macroeconomic Forecasting," Working Papers 2019-3, University of Hawaii Economic Research Organization, University of Hawaii at Manoa.
    20. Gerlak, Andrea K. & Guido, Zack & Owen, Gigi & McGoffin, Mariana Sofia Rodriguez & Louder, Elena & Davies, Julia & Smith, Kelly Jay & Zimmer, Andy & Murveit, Anna M. & Meadow, Alison & Shrestha, Padme, 2023. "Stakeholder engagement in the co-production of knowledge for environmental decision-making," World Development, Elsevier, vol. 170(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0168533. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.