IDEAS home Printed from https://ideas.repec.org/a/oup/scippl/v49y2022i2p289-301..html
   My bibliography  Save this article

Deconstructing impact: A framework for impact evaluation in grant applications
[Evidencing Impact from Art Research: Analysis of Impact Case Studies from the REF 2014]

Author

Listed:
  • Lai Ma
  • Rachael Agnew

Abstract

Impact assessments in grant applications can be biased, random, or inconsistent. One reason is that there is not a framework to assist the review process. To develop fair and transparent evaluative criteria, it is necessary to understand what kinds of outcome and impact can reasonably be achieved and expected. Using content analysis, 100 UK Research Excellence Framework (REF) 2014 impact case studies were analysed based on the definition of the output, outcome, and impact in the logic model. The analysis shows that (1) outcomes and medium-term impacts, not long-term impacts, are reported in the case studies, (2) impacts can be categorised as use and experience based, and (3) there is a need to recognise the creation and co-creation of impacts. A framework for impact evaluation in grant applications has been developed. In this framework, the criteria for evaluating impact statements focus on process-oriented impacts and that ‘impacts’ can be outputs, outcomes, and medium-term impacts.

Suggested Citation

  • Lai Ma & Rachael Agnew, 2022. "Deconstructing impact: A framework for impact evaluation in grant applications [Evidencing Impact from Art Research: Analysis of Impact Case Studies from the REF 2014]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 289-301.
    • Handle: RePEc:oup:scippl:v:49:y:2022:i:2:p:289-301.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/scipol/scab080
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Stefan P. L. de Jong & Jorrit Smit & Leonie van Drooge, 2016. "Scientists’ response to societal impact policies: A policy paradox," Science and Public Policy, Oxford University Press, vol. 43(1), pages 102-114.
    2. J Britt Holbrook & Robert Frodeman, 2011. "Peer review and the ex ante assessment of societal impacts," Research Evaluation, Oxford University Press, vol. 20(3), pages 239-246, September.
    3. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    4. Stefan P L de Jong & Reetta Muhonen, 2020. "Who benefits from ex ante societal impact evaluation in the European funding arena? A cross-country comparison of societal impact capacity in the social sciences and humanities," Research Evaluation, Oxford University Press, vol. 29(1), pages 22-33.
    5. Stevie Upton & Paul Vallance & John Goddard, 2014. "From outcomes to process: evidence for a new approach to research impact assessment," Research Evaluation, Oxford University Press, vol. 23(4), pages 352-365.
    6. Katherine Caves & Sarah Lueling, 2021. "Research and the real world: Analyzing the short- and long-term impact of knowledge transfer [ISRIA Statement: Ten-Point Guidelines for an Effective Process of Research Impact Assessment]," Research Evaluation, Oxford University Press, vol. 30(3), pages 393-404.
    7. Heyeres, Marion & Tsey, Komla & Yang, Yinghong & Yan, Li & Jiang, Hua, 2019. "The characteristics and reporting quality of research impact case studies: A systematic review," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 10-23.
    8. Gabrielle N. Samuel & Gemma E. Derrick, 2015. "Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014," Research Evaluation, Oxford University Press, vol. 24(3), pages 229-241.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stefan P L de Jong & Corina Balaban, 2022. "How universities influence societal impact practices: Academics’ sense-making of organizational impact strategies [Between Relevance and Excellence? Research Impact Agenda and the Production of Pol," Science and Public Policy, Oxford University Press, vol. 49(4), pages 609-620.
    2. Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    3. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    4. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    5. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    6. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    7. J. Britt Holbrook, 2017. "The future of the impact agenda depends on the revaluation of academic freedom," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-9, December.
    8. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    9. Stefan P L de Jong & Corina Balaban & Maria Nedeva, 2022. "From ‘productive interactions’ to ‘enabling conditions’: The role of organizations in generating societal impact of academic research [One Size Does Not Fit All! New Perspectives on the University ," Science and Public Policy, Oxford University Press, vol. 49(4), pages 643-645.
    10. Janire Fonseca Peso & Antonia Caro González & Nemanja Milosevic, 2020. "Innovative Co-Creative Participatory Methodologies for a Dreamt-of Quality Education in Europe," Sustainability, MDPI, vol. 12(16), pages 1-14, August.
    11. Jesús de Frutos-Belizón & Fernando Martín-Alcázar & Gonzalo Sánchez-Gardey, 2021. "The research–practice gap in the field of HRM: a qualitative study from the academic side of the gap," Review of Managerial Science, Springer, vol. 15(6), pages 1465-1515, August.
    12. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    13. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    14. Peter Weißhuhn & Katharina Helming & Johanna Ferretti, 2018. "Research impact assessment in agriculture—A review of approaches and impact areas," Research Evaluation, Oxford University Press, vol. 27(1), pages 36-42.
    15. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    16. Alba Viana Lora & Marta Gemma Nel-lo Andreu, 2020. "Alternative Metrics for Assessing the Social Impact of Tourism Research," Sustainability, MDPI, vol. 12(10), pages 1-12, May.
    17. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    18. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    19. Jonna Brenninkmeijer, 2022. "Achieving societal and academic impacts of research: A comparison of networks, values, and strategies [University Research Funding and Publication Performance - an International Comparison]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 728-738.
    20. Florian Findler, 2021. "Toward a sustainability assessment framework of research impacts: Contributions of a business school," Sustainable Development, John Wiley & Sons, Ltd., vol. 29(6), pages 1190-1203, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:scippl:v:49:y:2022:i:2:p:289-301.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/spp .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.