IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v50y2021i4s0048733320302225.html
   My bibliography  Save this article

Evaluating impact from research: A methodological framework

Author

Listed:
  • Reed, M.S.
  • Ferré, M.
  • Martin-Ortega, J.
  • Blanche, R.
  • Lawford-Rolfe, R.
  • Dallimer, M.
  • Holden, J.

Abstract

Interest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits.

Suggested Citation

  • Reed, M.S. & Ferré, M. & Martin-Ortega, J. & Blanche, R. & Lawford-Rolfe, R. & Dallimer, M. & Holden, J., 2021. "Evaluating impact from research: A methodological framework," Research Policy, Elsevier, vol. 50(4).
    • Handle: RePEc:eee:respol:v:50:y:2021:i:4:s0048733320302225
    DOI: 10.1016/j.respol.2020.104147
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733320302225
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2020.104147?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    2. Michael Woolcock, 2013. "Using Case Studies to Explore the External Validity of ‘Complex’ Development Interventions," CID Working Papers 270, Center for International Development at Harvard University.
    3. Richard Mallett & Jessica Hagen-Zanker & Rachel Slater & Maren Duvendack, 2012. "The benefits and challenges of using systematic reviews in international development research," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 4(3), pages 445-455, September.
    4. Greenhalgh, Trisha & Robert, Glenn & Macfarlane, Fraser & Bate, Paul & Kyriakidou, Olympia & Peacock, Richard, 2005. "Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review," Social Science & Medicine, Elsevier, vol. 61(2), pages 417-430, July.
    5. Reed, Mark S. & Fraser, Evan D.G. & Dougill, Andrew J., 2006. "An adaptive learning process for developing and applying sustainability indicators with local communities," Ecological Economics, Elsevier, vol. 59(4), pages 406-418, October.
    6. Rau, Henrike & Goggins, Gary & Fahy, Frances, 2018. "From invisibility to impact: Recognising the scientific and societal relevance of interdisciplinary sustainability research," Research Policy, Elsevier, vol. 47(1), pages 266-276.
    7. Gaunand, A. & Hocdé, A. & Lemarié, S. & Matt, M. & Turckheim, E.de, 2015. "How does public agricultural research impact society? A characterization of various patterns," Research Policy, Elsevier, vol. 44(4), pages 849-861.
    8. Zoe Bulaitis, 2017. "Measuring impact in the humanities: Learning from accountability and economics in a contemporary history of cultural value," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-11, December.
    9. Woolcock, Michael, 2013. "Using Case Studies to Explore the External Validity of 'Complex' Development Interventions," Working Paper Series rwp13-048, Harvard University, John F. Kennedy School of Government.
    10. Matt, M. & Gaunand, A. & Joly, P-B. & Colinet, L., 2017. "Opening the black box of impact – Ideal-type impact pathways in a public agricultural research organization," Research Policy, Elsevier, vol. 46(1), pages 207-218.
    11. Julian, David A. & Jones, Ann & Deyo, Diana, 1995. "Open systems evaluation and the logic model: Program planning and evaluation tools," Evaluation and Program Planning, Elsevier, vol. 18(4), pages 333-341.
    12. John D Brewer, 2011. "The impact of impact," Research Evaluation, Oxford University Press, vol. 20(3), pages 255-256, September.
    13. Bozeman, Barry & Youtie, Jan, 2017. "Socio-economic impacts and public value of government-funded research: Lessons from four US National Science Foundation initiatives," Research Policy, Elsevier, vol. 46(8), pages 1387-1398.
    14. Jack Spaapen & Leonie van Drooge, 2011. "Introducing ‘productive interactions’ in social impact assessment," Research Evaluation, Oxford University Press, vol. 20(3), pages 211-218, September.
    15. Gabrielle N. Samuel & Gemma E. Derrick, 2015. "Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014," Research Evaluation, Oxford University Press, vol. 24(3), pages 229-241.
    16. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    17. Alis Oancea, 2019. "Research governance and the future(s) of research assessment," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    18. Douthwaite, Boru & Kuby, Thomas & van de Fliert, Elske & Schulz, Steffen, 2003. "Impact pathway evaluation: an approach for achieving and attributing impact in complex systems," Agricultural Systems, Elsevier, vol. 78(2), pages 243-265, November.
    19. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    20. Edward T. Game & Heather Tallis & Lydia Olander & Steven M. Alexander & Jonah Busch & Nancy Cartwright & Elizabeth L. Kalies & Yuta J. Masuda & Anne-Christine Mupepele & Jiangxiao Qiu & Andrew Rooney , 2018. "Cross-discipline evidence principles for sustainability policy," Nature Sustainability, Nature, vol. 1(9), pages 452-454, September.
    21. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    22. J. Edler & L. Georghiou & K. Blind & E. Uyarra, 2012. "Evaluating the demand side: New challenges for evaluation," Research Evaluation, Oxford University Press, vol. 21(1), pages 33-47, February.
    23. Alberto Martín-Martín & Enrique Orduna-Malea & Emilio Delgado López-Cózar, 2018. "Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2175-2188, September.
    24. Zaifunizam Ariffin & Steven Eric Krauss & Haslinda Abdullah & Adriana Ortega & Zuraidah Ali, 2017. "Peer Support and Youth Engagement," International Journal of Academic Research in Business and Social Sciences, Human Resource Management Academic Research Society, International Journal of Academic Research in Business and Social Sciences, vol. 7(8), pages 629-634, August.
    25. Paul J. Gertler & Sebastian Martinez & Patrick Premand & Laura B. Rawlings & Christel M. J. Vermeersch, . "Impact Evaluation in Practice, First Edition [La evaluación de impacto en la práctica]," World Bank Publications, The World Bank, number 2550, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Daniele Rotolo & Michael Hopkins & Nicola Grassano, 2023. "Do funding sources complement or substitute? Examining the impact of cancer research publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 50-66, January.
    2. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    3. Bhubaneswor Dhakal & Narendra Chand & Him Lal Shrestha & Anita Shrestha & Nischal Dhakal & Bikash Adhikari & Shyam Krishna Shrestha & Krishna Bahadur Karki & Padam Bhandari, 2022. "Paradoxes of Aggravated Vulnerability, Marginalization, and Peril of Forest-Based Communities after Increasing Conservative Forest and Protected Areas in Nepal: A Policy Lesson on Land-Based Climate C," World, MDPI, vol. 3(3), pages 1-31, August.
    4. Keishiro Hara & Iori Miura & Masanori Suzuki & Toshihiro Tanaka, 2023. "Designing research strategy and technology innovation for sustainability by adopting “imaginary future generations”—A case study using metallurgy," Futures & Foresight Science, John Wiley & Sons, vol. 5(3-4), September.
    5. Haddad, Carolina R. & Bergek, Anna, 2023. "Towards an integrated framework for evaluating transformative innovation policy," Research Policy, Elsevier, vol. 52(2).
    6. Isabel Vogel & Chris Barnett, 2023. "Laying the Foundations for Impact: Lessons from the GCRF Evaluation," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 35(2), pages 281-297, April.
    7. Daniel Black & Susanne Charlesworth & Maria Ester Dal Poz & Erika Cristina Francisco & Adina Paytan & Ian Roderick & Timo von Wirth & Kevin Winter, 2023. "Comparing Societal Impact Planning and Evaluation Approaches across Four Urban Living Labs (in Food-Energy-Water Systems)," Sustainability, MDPI, vol. 15(6), pages 1-20, March.
    8. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    9. Anh Huu Nguyen & Thinh Gia Hoang & Duy Thanh Nguyen & Loan Quynh Thi Nguyen & Duong Thuy Doan, 2023. "The Development of Green Bond in Developing Countries: Insights from Southeast Asia Market Participants," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 35(1), pages 196-218, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Genowefa Blundo-Canto & Bernard Triomphe & Guy Faure & Danielle Barret & Aurelle de Romemont & Etienne Hainzelin, 2019. "Building a culture of impact in an international agricultural research organization: Process and reflective learning," Research Evaluation, Oxford University Press, vol. 28(2), pages 136-144.
    2. Matt, M. & Gaunand, A. & Joly, P-B. & Colinet, L., 2017. "Opening the black box of impact – Ideal-type impact pathways in a public agricultural research organization," Research Policy, Elsevier, vol. 46(1), pages 207-218.
    3. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    4. A. Gaunand & L. Colinet & P.-B. Joly & M. Matt, 2022. "Counting what really counts? Assessing the political impact of science," The Journal of Technology Transfer, Springer, vol. 47(3), pages 699-721, June.
    5. Ciarli, Tommaso & Ràfols, Ismael, 2019. "The relation between research priorities and societal demands: The case of rice," Research Policy, Elsevier, vol. 48(4), pages 949-967.
    6. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    7. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    8. Katherine Casey & Rachel Glennerster & Edward Miguel & Maarten Voors, 2023. "Skill Versus Voice in Local Development," The Review of Economics and Statistics, MIT Press, vol. 105(2), pages 311-326, March.
    9. Vivek Kumar Singh & Prashasti Singh & Mousumi Karmakar & Jacqueline Leta & Philipp Mayr, 2021. "The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5113-5142, June.
    10. Quentin Stoeffler & Michael Carter & Catherine Guirkinger & Wouter Gelade, 2022. "The Spillover Impact of Index Insurance on Agricultural Investment by Cotton Farmers in Burkina Faso," The World Bank Economic Review, World Bank, vol. 36(1), pages 114-140.
    11. de Almeida, Liliane & Augusto de Jesus Pacheco, Diego & Caten, Carla Schwengber ten & Jung, Carlos Fernando, 2021. "A methodology for identifying results and impacts in technological innovation projects," Technology in Society, Elsevier, vol. 66(C).
    12. Arnaud Vaganay, 2016. "Cluster Sampling Bias in Government-Sponsored Evaluations: A Correlational Study of Employment and Welfare Pilots in England," PLOS ONE, Public Library of Science, vol. 11(8), pages 1-21, August.
    13. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    14. Hakiman, Kamran & Sheely, Ryan, 2023. "Unlocking the Potential of Participatory Planning: How Flexible and Adaptive Governance Interventions Can Work in Practice," OSF Preprints kucjs, Center for Open Science.
    15. Matt, M. & Colinet, L. & Gaunand, A. & Joly, P.B., 2015. "A typology of impact pathways generated by a public agricultural research organization," Working Papers 2015-03, Grenoble Applied Economics Laboratory (GAEL).
    16. Milante Gary & Woolcock Michael, 2017. "New Approaches to Identifying State Fragility," Journal of Globalization and Development, De Gruyter, vol. 8(1), pages 1-10, June.
    17. Turner, James A & Guesmi, Bouali & Gil, José M. & Heanue, Kevin & Sierra, Miguel & Percy, Helen & Bortagaray, Isabel & Chams, Nour & Milne, Cath, 2022. "Evaluation capacity building in response to the agricultural research impact agenda: Emerging insights from Ireland, Catalonia (Spain), New Zealand, and Uruguay," Evaluation and Program Planning, Elsevier, vol. 94(C).
    18. Civera, Alice & Lehmann, Erik E. & Paleari, Stefano & Stockinger, Sarah A.E., 2020. "Higher education policy: Why hope for quality when rewarding quantity?," Research Policy, Elsevier, vol. 49(8).
    19. Paúl Carrión-Mero & Néstor Montalván-Burbano & Fernando Morante-Carballo & Adolfo Quesada-Román & Boris Apolo-Masache, 2021. "Worldwide Research Trends in Landslide Science," IJERPH, MDPI, vol. 18(18), pages 1-24, September.
    20. Woolcock, Michael, 2014. "Engaging with Fragile and Conflict-Affected States," Working Paper Series rwp14-038, Harvard University, John F. Kennedy School of Government.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:50:y:2021:i:4:s0048733320302225. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.