IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v95y2022ics0149718922001112.html
   My bibliography  Save this article

What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors

Author

Listed:
  • Dotti, Nicola Francesco
  • Walczyk, Julia

Abstract

The discussion on the ‘societal impact of university research’ has emerged within the recent decade with different utilisation from the British REF impact stories to the EU Framework Programmes’ criteria. The fundamental idea is that (university) research should contribute to addressing societal challenges, such as climate change, and not just pursue ‘scientific excellence’. Several approaches have been developed, from case studies of the ‘productive interactions’ that co-create knowledge with societal stakeholders to the so-called ‘altmetrics’ tracking online dissemination of research outputs. Nonetheless, these experiences do not always point in the same direction, and the notion of ‘societal impact’ seems still unclearly defined. Given the growing policy pressure on universities to address societal challenges, this paper reviews the scientific literature on the so-called ‘societal impact’ of university research. By querying three scientific archives (Web of Science, Scopus and ScienceDirect), 135 relevant scientific publications are selected and discussed, looking for conceptualisations, monitoring methods and success factors. The review maps a value-laden discussion arguing for longer-term, multi-dimensional perspectives on university research impacts beyond bibliometric indicators. Findings highlight a progressive shift from ‘attribution’, i.e., looking for causal relationships between research and societal changes, to 'contribution', acknowledging researchers' efforts to engage with societal challenges.

Suggested Citation

  • Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    • Handle: RePEc:eee:epplan:v:95:y:2022:i:c:s0149718922001112
    DOI: 10.1016/j.evalprogplan.2022.102157
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718922001112
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2022.102157?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lyall, Catherine & Bruce, Ann & Firn, John & Firn, Marion & Tait, Joyce, 2004. "Assessing end-use relevance of public sector research organisations," Research Policy, Elsevier, vol. 33(1), pages 73-87, January.
    2. Paola Coletti & Nicola Francesco Dotti, 2021. "Knowledge co-production promoting policy change in time of decentralisation: a comparison of two cases from Brussels," Local Government Studies, Taylor & Francis Journals, vol. 47(2), pages 276-295, March.
    3. Thomas Niederkrotenthaler & Thomas Dorner & Manfred Maier, 2011. "Measuring impact of research on society," Nature, Nature, vol. 469(7328), pages 34-34, January.
    4. Narzanin Massoumi & Tom Mills & David Miller, 2020. "Secrecy, coercion and deception in research on ‘terrorism’ and ‘extremism’," Contemporary Social Science, Taylor & Francis Journals, vol. 15(2), pages 134-152, April.
    5. Stina Hansson & Merritt Polk, 2018. "Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact," Research Evaluation, Oxford University Press, vol. 27(2), pages 132-144.
    6. David Budtz Pedersen & Jonas Følsgaard Grønvad & Rolf Hvidtfeldt, 2020. "Methods for mapping the impact of social sciences and humanities—A literature review," Research Evaluation, Oxford University Press, vol. 29(1), pages 4-21.
    7. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    8. Anna-Lena Guske & Klaus Jacob & Martin Hirschnitz-Garbers & Jan Peuckert & Stefan Schridde & Sven Stinner & Franziska Wolff & Dominik Zahrnt & Florence Ziesemer, 2019. "Stories that Change Our World? Narratives of the Sustainable Economy," Sustainability, MDPI, vol. 11(21), pages 1-14, November.
    9. Stefan de Jong & Katharine Barker & Deborah Cox & Thordis Sveinsdottir & Peter Van den Besselaar, 2014. "Understanding societal impact through productive interactions: ICT research as a case," Research Evaluation, Oxford University Press, vol. 23(2), pages 89-102.
    10. João Carlos Nabout & Fabrício Barreto Teresa & Karine Borges Machado & Vitor Hugo Mendonça Prado & Luis Mauricio Bini & José Alexandre Felizola Diniz-Filho, 2018. "Do traditional scientometric indicators predict social media activity on scientific knowledge? An analysis of the ecological literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1007-1015, May.
    11. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    12. Rau, Henrike & Goggins, Gary & Fahy, Frances, 2018. "From invisibility to impact: Recognising the scientific and societal relevance of interdisciplinary sustainability research," Research Policy, Elsevier, vol. 47(1), pages 266-276.
    13. Claire Donovan, 2007. "The qualitative future of research evaluation," Science and Public Policy, Oxford University Press, vol. 34(8), pages 585-597, October.
    14. Gaunand, A. & Hocdé, A. & Lemarié, S. & Matt, M. & Turckheim, E.de, 2015. "How does public agricultural research impact society? A characterization of various patterns," Research Policy, Elsevier, vol. 44(4), pages 849-861.
    15. Tharapos, Meredith & Marriott, Neil, 2020. "Beauty is in the eye of the beholder: Research quality in accounting education," The British Accounting Review, Elsevier, vol. 52(5).
    16. Barbara L Riley & Alison Kernoghan & Lisa Stockton & Steve Montague & Jennifer Yessis & Cameron D Willis, 2018. "Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’," Research Evaluation, Oxford University Press, vol. 27(1), pages 16-27.
    17. Roberta Kwok, 2013. "Research impact: Altmetrics make their mark," Nature, Nature, vol. 500(7463), pages 491-493, August.
    18. Phillip Phan & Donald S. Siegel & Mike Wright, 2016. "Science Parks and Incubators: Observations, Synthesis and Future Research," World Scientific Book Chapters, in: Phillip H Phan & Sarfraz A Mian & Wadid Lamine (ed.), TECHNOLOGY ENTREPRENEURSHIP AND BUSINESS INCUBATION Theory • Practice • Lessons Learned, chapter 9, pages 249-272, World Scientific Publishing Co. Pte. Ltd..
    19. Reed, M.S. & Ferré, M. & Martin-Ortega, J. & Blanche, R. & Lawford-Rolfe, R. & Dallimer, M. & Holden, J., 2021. "Evaluating impact from research: A methodological framework," Research Policy, Elsevier, vol. 50(4).
    20. Merja Otronen, 2004. "The evaluation of social relevance in an environmental research programme," Research Evaluation, Oxford University Press, vol. 13(1), pages 43-50, April.
    21. Mike Thelwall, 2021. "Measuring Societal Impacts Of Research With Altmetrics? Common Problems And Mistakes," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1302-1314, December.
    22. Matt, M. & Gaunand, A. & Joly, P-B. & Colinet, L., 2017. "Opening the black box of impact – Ideal-type impact pathways in a public agricultural research organization," Research Policy, Elsevier, vol. 46(1), pages 207-218.
    23. Jenny Wooldridge & Mike B. King, 2019. "Altmetric scores: An early indicator of research impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 70(3), pages 271-282, March.
    24. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    25. Barend van der Meulen & Arie Rip, 2000. "Evaluation of societal quality of public sector research in the Netherlands," Research Evaluation, Oxford University Press, vol. 9(1), pages 11-25, April.
    26. Bella Reichard & Mark S Reed & Jenn Chubb & Ged Hall & Lucy Jowett & Alisha Peart & Andrea Whittle, 2020. "Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014," Palgrave Communications, Palgrave Macmillan, vol. 6(1), pages 1-17, December.
    27. Jörg Becker & Jan Brocke & Marcel Heddier & Stefan Seidel, 2015. "In Search of Information Systems (Grand) Challenges," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(6), pages 377-390, December.
    28. Alba Viana Lora & Marta Gemma Nel-lo Andreu, 2020. "Alternative Metrics for Assessing the Social Impact of Tourism Research," Sustainability, MDPI, vol. 12(10), pages 1-12, May.
    29. Heyeres, Marion & Tsey, Komla & Yang, Yinghong & Yan, Li & Jiang, Hua, 2019. "The characteristics and reporting quality of research impact case studies: A systematic review," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 10-23.
    30. David Doloreux & Igone Porto Gomez, 2017. "A review of (almost) 20 years of regional innovation systems research," European Planning Studies, Taylor & Francis Journals, vol. 25(3), pages 371-387, March.
    31. Faure, Guy & Barret, Danielle & Blundo-Canto, Genowefa & Dabat, Marie-Hélène & Devaux-Spatarakis, Agathe & Le Guerroué, Jean Louis & Marquié, Catherine & Mathé, Syndhia & Temple, Ludovic & Toillier, A, 2018. "How different agricultural research models contribute to impacts: Evidence from 13 case studies in developing countries," Agricultural Systems, Elsevier, vol. 165(C), pages 128-136.
    32. Julie Davies & Emily Yarrow & Jawad Syed, 2020. "The curious under‐representation of women impact case leaders: Can we disengender inequality regimes?," Gender, Work and Organization, Wiley Blackwell, vol. 27(2), pages 129-148, March.
    33. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    34. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    35. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    36. Reetta Muhonen & Paul Benneworth & Julia Olmos-Peñuela, 2020. "From productive interactions to impact pathways: Understanding the key dimensions in developing SSH research societal impact," Research Evaluation, Oxford University Press, vol. 29(1), pages 34-47.
    37. Lutz Bornmann & Robin Haunschild, 2016. "How to normalize Twitter counts? A first attempt based on journals in the Twitter Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1405-1422, June.
    38. Tsey, Komla & Onnis, Leigh-ann & Whiteside, Mary & McCalman, Janya & Williams, Megan & Heyeres, Marion & Lui, Siu Man (Carrie) & Klieve, Helen & Cadet-James, Yvonne & Baird, Leslie & Brown, Catherine , 2019. "Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 176-186.
    39. J Britt Holbrook & Robert Frodeman, 2011. "Peer review and the ex ante assessment of societal impacts," Research Evaluation, Oxford University Press, vol. 20(3), pages 239-246, September.
    40. Stina Hansson & Merritt Polk, 2019. "Comments to Belcher et al. 2018’s critique of Hansson and Polk 2018," Research Evaluation, Oxford University Press, vol. 28(2), pages 202-205.
    41. König, Bettina & Diehl, Katharina & Tscherning, Karen & Helming, Katharina, 2013. "A framework for structuring interdisciplinary research management," Research Policy, Elsevier, vol. 42(1), pages 261-272.
    42. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    43. Hanan Khazragui & John Hudson, 2015. "Measuring the benefits of university research: impact and the REF in the UK," Research Evaluation, Oxford University Press, vol. 24(1), pages 51-62.
    44. Nelius Boshoff & Mpho Sefatsa, 2019. "Creating research impact through the productive interactions of an individual: an example from South African research on maritime piracy," Research Evaluation, Oxford University Press, vol. 28(2), pages 145-157.
    45. Parisa Maroufkhani & Ralf Wagner & Wan Khairuzzaman Wan Ismail, 2018. "Entrepreneurial ecosystems: a systematic review," Journal of Enterprising Communities: People and Places in the Global Economy, Emerald Group Publishing Limited, vol. 12(4), pages 545-564, July.
    46. Inge van der Weijden & Maaike Verbree & Peter van den Besselaar, 2012. "From bench to bedside: The societal orientation of research leaders: The case of biomedical and health research in the Netherlands," Science and Public Policy, Oxford University Press, vol. 39(3), pages 285-303, May.
    47. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    48. K. Brad Wray, 2016. "No new evidence for a citation benefit for Author-Pay Open Access Publications in the social sciences and humanities," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 1031-1035, March.
    49. Samantha Cruz Rivera & Derek G Kyte & Olalekan Lee Aiyegbusi & Thomas J Keeley & Melanie J Calvert, 2017. "Assessing the impact of healthcare research: A systematic review of methodological frameworks," PLOS Medicine, Public Library of Science, vol. 14(8), pages 1-24, August.
    50. Kate Williams, 2020. "Playing the fields: Theorizing research impact and its assessment," Research Evaluation, Oxford University Press, vol. 29(2), pages 191-202.
    51. María José Aranguren & Patricia Canto-Farachala & James R Wilson, 2021. "Transformative academic institutions: An experimental framework for understanding regional impacts of research," Research Evaluation, Oxford University Press, vol. 30(2), pages 191-200.
    52. Han Zheng & L. G. Pee & Dan Zhang, 2021. "Societal impact of research: a text mining study of impact types," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7397-7417, September.
    53. Alexis Dewaele & Kristof Vandael & Stefan Meysman & Ann Buysse, 2021. "Understanding collaborative interactions in relation to research impact in social sciences and humanities: A meta-ethnography," Research Evaluation, Oxford University Press, vol. 30(2), pages 179-190.
    54. Nelius Boshoff & Stefan P L de Jong, 2020. "Conceptualizing the societal impact of research in terms of elements of logic models: a survey of researchers in sub-Saharan Africa," Research Evaluation, Oxford University Press, vol. 29(1), pages 48-65.
    55. Lutz Bornmann & Robin Haunschild, 2017. "Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 937-943, February.
    56. Juha-Pekka Lauronen, 2020. "The dilemmas and uncertainties in assessing the societal impact of research," Science and Public Policy, Oxford University Press, vol. 47(2), pages 207-218.
    57. W. H. Voorberg & V. J. J. M. Bekkers & L. G. Tummers, 2015. "A Systematic Review of Co-Creation and Co-Production: Embarking on the social innovation journey," Public Management Review, Taylor & Francis Journals, vol. 17(9), pages 1333-1357, October.
    58. Steven Hill, 2016. "Assessing (for) impact: future assessment of the societal impact of research," Palgrave Communications, Palgrave Macmillan, vol. 2(1), pages 1-7, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marra, Mita, 2022. "Productive interactions in digital training partnerships: Lessons learned for regional development and university societal impact assessment," Evaluation and Program Planning, Elsevier, vol. 95(C).
    2. Kalpazidou Schmidt, Evanthia, 2023. "Creating a developmental framework for evaluating RRI implementation in research organisations," Evaluation and Program Planning, Elsevier, vol. 100(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    2. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    3. Alba Viana Lora & Marta Gemma Nel-lo Andreu, 2020. "Alternative Metrics for Assessing the Social Impact of Tourism Research," Sustainability, MDPI, vol. 12(10), pages 1-12, May.
    4. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    5. Genowefa Blundo-Canto & Bernard Triomphe & Guy Faure & Danielle Barret & Aurelle de Romemont & Etienne Hainzelin, 2019. "Building a culture of impact in an international agricultural research organization: Process and reflective learning," Research Evaluation, Oxford University Press, vol. 28(2), pages 136-144.
    6. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    7. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    8. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    9. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    10. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    11. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    12. Matt, M. & Colinet, L. & Gaunand, A. & Joly, P.B., 2015. "A typology of impact pathways generated by a public agricultural research organization," Working Papers 2015-03, Grenoble Applied Economics Laboratory (GAEL).
    13. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    14. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    15. Juha-Pekka Lauronen, 2022. "The epistemic, production, and accountability prospects of social impact: An analysis of strategic research proposals," Research Evaluation, Oxford University Press, vol. 31(2), pages 214-225.
    16. D’Este, Pablo & Robinson-García, Nicolás, 2023. "Interdisciplinary research and the societal visibility of science: The advantages of spanning multiple and distant scientific fields," Research Policy, Elsevier, vol. 52(2).
    17. Hans Jonker & Florian Vanlee & Walter Ysebaert, 2022. "Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7289-7306, December.
    18. Rene Brauer & Mirek Dymitrow, 2020. "The Language of Sustainable Tourism as a Proxy Indicator of Quality," Sustainability, MDPI, vol. 13(1), pages 1-17, December.
    19. Marra, Mita, 2022. "Productive interactions in digital training partnerships: Lessons learned for regional development and university societal impact assessment," Evaluation and Program Planning, Elsevier, vol. 95(C).
    20. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:95:y:2022:i:c:s0149718922001112. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.