IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v73y2019icp10-23.html
   My bibliography  Save this article

The characteristics and reporting quality of research impact case studies: A systematic review

Author

Listed:
  • Heyeres, Marion
  • Tsey, Komla
  • Yang, Yinghong
  • Yan, Li
  • Jiang, Hua

Abstract

Despite the growing expectation that researchers report the impact of their research using a case study approach, systematic reviews of research impact have focused on frameworks, indicators, methods of data collection and assessment rather than impact case studies. Our aim is to provide an overview of the characteristics of published research impact case studies, including translation activities, and their reporting quality. We searched for peer-reviewed impact studies published between 2000 and 2018 using a case study approach and selected 25 suitable papers. We applied descriptive statistics to study characteristics, conducted thematic analysis of research translation activities and assessed reporting quality using the 10-point ISRIA statement. 24 papers reported intermediate impacts, such as advocacy, or the development of statements, tools, or technology. 4 reported on longer-term societal impacts, such as health outcomes and economic return on investment. 7 reported on translation activities. Papers scored well against the ISRIA statement on 5 domains of reporting quality. Weakest scores centred around identification of stakeholder needs and stakeholder involvement, and ethics and conflict of interest. We identified the need for more consistency in reporting through a case study approach, more systematic reporting of translation pathways and greater transparency concerning estimated costs and benefits of the research and its translation and impact assessment.

Suggested Citation

  • Heyeres, Marion & Tsey, Komla & Yang, Yinghong & Yan, Li & Jiang, Hua, 2019. "The characteristics and reporting quality of research impact case studies: A systematic review," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 10-23.
    • Handle: RePEc:eee:epplan:v:73:y:2019:i:c:p:10-23
    DOI: 10.1016/j.evalprogplan.2018.11.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718918300867
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2018.11.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Annette Boaz & Siobhan Fitzpatrick & Ben Shaw, 2009. "Assessing the impact of research on policy: A literature review," Science and Public Policy, Oxford University Press, vol. 36(4), pages 255-270, May.
    2. repec:dau:papers:123456789/13723 is not listed on IDEAS
    3. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    4. Carlyn E Orians & Joanne Abed & Christina H Drew & Shyanika Wijesinha Rose & Jennifer H Cohen & Jerry Phelps, 2009. "Scientific and public health impacts of the NIEHS Extramural Asthma Research Program: insights from primary data," Research Evaluation, Oxford University Press, vol. 18(5), pages 375-385, December.
    5. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    6. Alene, Arega D. & Coulibaly, Ousmane, 2009. "The impact of agricultural research on productivity and poverty in sub-Saharan Africa," Food Policy, Elsevier, vol. 34(2), pages 198-209, April.
    7. Lisa Klautzer & Stephen Hanney & Edward Nason & Jennifer Rubin & Jonathan Grant & Steven Wooding, 2011. "Assessing policy and practice impacts of social science research: the application of the Payback Framework to assess the Future of Work programme," Research Evaluation, Oxford University Press, vol. 20(3), pages 201-209, September.
    8. Walker, Tom & Ryan, Jim & Kelley, Tim, 2010. "Impact Assessment of Policy-Oriented International Agricultural Research: Evidence and Insights from Case Studies," World Development, Elsevier, vol. 38(10), pages 1453-1461, October.
    9. Samantha Cruz Rivera & Derek G Kyte & Olalekan Lee Aiyegbusi & Thomas J Keeley & Melanie J Calvert, 2017. "Assessing the impact of healthcare research: A systematic review of methodological frameworks," PLOS Medicine, Public Library of Science, vol. 14(8), pages 1-24, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lai Ma & Rachael Agnew, 2022. "Deconstructing impact: A framework for impact evaluation in grant applications [Evidencing Impact from Art Research: Analysis of Impact Case Studies from the REF 2014]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 289-301.
    2. Florian Findler, 2021. "Toward a sustainability assessment framework of research impacts: Contributions of a business school," Sustainable Development, John Wiley & Sons, Ltd., vol. 29(6), pages 1190-1203, November.
    3. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tsey, Komla & Onnis, Leigh-ann & Whiteside, Mary & McCalman, Janya & Williams, Megan & Heyeres, Marion & Lui, Siu Man (Carrie) & Klieve, Helen & Cadet-James, Yvonne & Baird, Leslie & Brown, Catherine , 2019. "Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 176-186.
    2. Juha-Pekka Lauronen, 2022. "Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization," SAGE Open, , vol. 12(2), pages 21582440221, April.
    3. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    4. Alessandro Magrini & Fabio Bartolini & Alessandra Coli & Barbara Pacini, 2019. "A structural equation model to assess the impact of agricultural research expenditure on multiple dimensions," Quality & Quantity: International Journal of Methodology, Springer, vol. 53(4), pages 2063-2080, July.
    5. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    6. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    7. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    8. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    9. A. Gaunand & L. Colinet & P.-B. Joly & M. Matt, 2022. "Counting what really counts? Assessing the political impact of science," The Journal of Technology Transfer, Springer, vol. 47(3), pages 699-721, June.
    10. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    11. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    12. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    13. İlkay Unay-Gailhard & Mark A. Brennen, 2022. "How digital communications contribute to shaping the career paths of youth: a review study focused on farming as a career option," Agriculture and Human Values, Springer;The Agriculture, Food, & Human Values Society (AFHVS), vol. 39(4), pages 1491-1508, December.
    14. Mahin Ghafari & Vali Baigi & Zahra Cheraghi & Amin Doosti-Irani, 2016. "The Prevalence of Asymptomatic Bacteriuria in Iranian Pregnant Women: A Systematic Review and Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-10, June.
    15. Elizabeth T Cafiero-Fonseca & Andrew Stawasz & Sydney T Johnson & Reiko Sato & David E Bloom, 2017. "The full benefits of adult pneumococcal vaccination: A systematic review," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-23, October.
    16. Santos Urbina & Sofía Villatoro & Jesús Salinas, 2021. "Self-Regulated Learning and Technology-Enhanced Learning Environments in Higher Education: A Scoping Review," Sustainability, MDPI, vol. 13(13), pages 1-12, June.
    17. Oded Berger-Tal & Alison L Greggor & Biljana Macura & Carrie Ann Adams & Arden Blumenthal & Amos Bouskila & Ulrika Candolin & Carolina Doran & Esteban Fernández-Juricic & Kiyoko M Gotanda & Catherine , 2019. "Systematic reviews and maps as tools for applying behavioral ecology to management and policy," Behavioral Ecology, International Society for Behavioral Ecology, vol. 30(1), pages 1-8.
    18. Nadine Desrochers & Adèle Paul‐Hus & Jen Pecoskie, 2017. "Five decades of gratitude: A meta‐synthesis of acknowledgments research," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(12), pages 2821-2833, December.
    19. Alene Sze Jing Yong & Yi Heng Lim & Mark Wing Loong Cheong & Ednin Hamzah & Siew Li Teoh, 2022. "Willingness-to-pay for cancer treatment and outcome: a systematic review," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 23(6), pages 1037-1057, August.
    20. Xue-Ying Xu & Hong Kong & Rui-Xiang Song & Yu-Han Zhai & Xiao-Fei Wu & Wen-Si Ai & Hong-Bo Liu, 2014. "The Effectiveness of Noninvasive Biomarkers to Predict Hepatitis B-Related Significant Fibrosis and Cirrhosis: A Systematic Review and Meta-Analysis of Diagnostic Test Accuracy," PLOS ONE, Public Library of Science, vol. 9(6), pages 1-16, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:73:y:2019:i:c:p:10-23. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.