IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v27y2018i2p132-144..html
   My bibliography  Save this article

Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact

Author

Listed:
  • Stina Hansson
  • Merritt Polk

Abstract

There is a call for more transdisciplinary (TD) research, from academia, society, and funding agencies. Consequently, the field of TD research is searching for ways of proving the value and providing evidence to support the effectiveness of such research. The main challenge for evaluating TD research is attribution, that is how to link societal change to the TD research process. However, little attention has been paid to the relationship between the quality of the research process and the effects and impacts that are being evaluated. Building upon earlier attempts at evaluating TD research, this article tests three key aspects of effective sustainability research: its relevance, credibility, and legitimacy. To explore the link between the quality of process and societal effects, we analyze and compare outputs, outcomes, and impact of five TD projects. Overall, our analysis shows that while relevance, credibility, and legitimacy gave important insights regarding the links between process and impacts, they are not adequate for evaluating TD research impact. Process qualities such as practitioner motivation and perceived importance of the project, together with breadth of perspectives, the openness/flexibility of participants, and in-depth exchanges of expertise and knowledge, contributed to producing internally relevant, credible, and legitimate results. However, we also saw a need to develop the relevance, credibility, and legitimacy framework, in relation to the external dynamics of the project process, heterogeneous stakeholder groups, and the credibility of practice-based knowledge, which together with institutional factors and the political context significantly shape the possibility of impact.

Suggested Citation

  • Stina Hansson & Merritt Polk, 2018. "Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact," Research Evaluation, Oxford University Press, vol. 27(2), pages 132-144.
  • Handle: RePEc:oup:rseval:v:27:y:2018:i:2:p:132-144.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvy004
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sonia Talwar & Arnim Wiek & John Robinson, 2011. "User engagement in sustainability research," Science and Public Policy, Oxford University Press, vol. 38(5), pages 379-390, June.
    2. Irwin Feller, 2006. "Multiple actors, multiple settings, multiple criteria: issues in assessing interdisciplinary research," Research Evaluation, Oxford University Press, vol. 15(1), pages 5-15, April.
    3. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    4. Brian M. Belcher & Katherine E. Rasmussen & Matthew R. Kemshaw & Deborah A. Zornes, 2016. "Defining and assessing research quality in a transdisciplinary context," Research Evaluation, Oxford University Press, vol. 25(1), pages 1-17.
    5. Elizabeth Koier & Edwin Horlings, 2015. "How accurately does output reflect the nature and design of transdisciplinary research programmes?," Research Evaluation, Oxford University Press, vol. 24(1), pages 37-50.
    6. John Robinson & James Tansey, 2006. "Co-production, emergent properties and strong interactive social research: the Georgia Basin Futures Project," Science and Public Policy, Oxford University Press, vol. 33(2), pages 151-160, March.
    7. Christian Pohl & Stephan Rist & Anne Zimmermann & Patricia Fry & Ghana S Gurung & Flurina Schneider & Chinwe Ifejika Speranza & Boniface Kiteme & Sébastian Boillat & Elvira Serrano & Gertrude Hirsch H, 2010. "Researchers' roles in knowledge co-production: experience from sustainability research in Kenya, Switzerland, Bolivia and Nepal," Science and Public Policy, Oxford University Press, vol. 37(4), pages 267-281, May.
    8. Laura Meagher & Catherine Lyall & Sandra Nutley, 2008. "Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research," Research Evaluation, Oxford University Press, vol. 17(3), pages 163-173, September.
    9. Cash, David & Clark, William & Alcock, Frank & Dickson, Nancy & Eckley, Noelle & Jager, Jill, 2002. "Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making," Working Paper Series rwp02-046, Harvard University, John F. Kennedy School of Government.
    10. Vanessa Watson, 2014. "Co-production and collaboration in planning - The difference," Planning Theory & Practice, Taylor & Francis Journals, vol. 15(1), pages 62-76, March.
    11. de Jong, Stefan P.L. & Wardenaar, Tjerk & Horlings, Edwin, 2016. "Exploring the promises of transdisciplinary research: A quantitative study of two climate research programmes," Research Policy, Elsevier, vol. 45(7), pages 1397-1409.
    12. Lutz Bornmann & Robin Haunschild, 2017. "Measuring field-normalized impact of papers on specific societal groups: An altmetrics study based on Mendeley Data," Research Evaluation, Oxford University Press, vol. 26(3), pages 230-241.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Genowefa Blundo-Canto & Bernard Triomphe & Guy Faure & Danielle Barret & Aurelle de Romemont & Etienne Hainzelin, 2019. "Building a culture of impact in an international agricultural research organization: Process and reflective learning," Research Evaluation, Oxford University Press, vol. 28(2), pages 136-144.
    2. D’Este, Pablo & Robinson-García, Nicolás, 2023. "Interdisciplinary research and the societal visibility of science: The advantages of spanning multiple and distant scientific fields," Research Policy, Elsevier, vol. 52(2).
    3. Heikki Tuomenvirta & Hilppa Gregow & Atte Harjanne & Sanna Luhtala & Antti Mäkelä & Karoliina Pilli-Sihvola & Sirkku Juhola & Mikael Hildén & Pirjo Peltonen-Sainio & Ilkka T. Miettinen & Mikko Halonen, 2019. "Identifying Policy Actions Supporting Weather-Related Risk Management and Climate Change Adaptation in Finland," Sustainability, MDPI, vol. 11(13), pages 1-15, July.
    4. Edwards, David M. & Meagher, Laura R., 2020. "A framework to evaluate the impacts of research on policy and practice: A forestry pilot study," Forest Policy and Economics, Elsevier, vol. 114(C).
    5. Upham, Dr Paul & Sovacool, Prof Benjamin & Ghosh, Dr Bipashyee, 2022. "Just transitions for industrial decarbonisation: A framework for innovation, participation, and justice," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    6. Ana Maria Perez Arredondo, 2022. "Research-Practice-Collaborations Addressing One Health and Urban Transformation. A Case Study," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 34(4), pages 1745-1756, August.
    7. Brian M Belcher & Luisa F Ramirez & Rachel Davel & Rachel Claus, 2019. "A response to Hansson and Polk (2018) “Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impac," Research Evaluation, Oxford University Press, vol. 28(2), pages 196-201.
    8. Stina Hansson & Merritt Polk, 2019. "Comments to Belcher et al. 2018’s critique of Hansson and Polk 2018," Research Evaluation, Oxford University Press, vol. 28(2), pages 202-205.
    9. Theresa Tribaldos & Flurina Schneider, 2021. "Enabling Players to Develop Theories of Change for Sustainable Development: A Serious Game," Simulation & Gaming, , vol. 52(5), pages 664-678, October.
    10. Melissa Robson-Williams & Bruce Small & Roger Robson-Williams & Nick Kirk, 2021. "Handrails through the Swamp? A Pilot to Test the Integration and Implementation Science Framework in Complex Real-World Research," Sustainability, MDPI, vol. 13(10), pages 1-23, May.
    11. Stephanie Jahn & Jens Newig & Daniel J. Lang & Judith Kahle & Matthias Bergmann, 2022. "Demarcating transdisciplinary research in sustainability science—Five clusters of research modes based on evidence from 59 research projects," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(2), pages 343-357, April.
    12. Cian O’Donovan & Aleksandra (Ola) Michalec & Joshua R Moon, 2022. "Capabilities for transdisciplinary research," Research Evaluation, Oxford University Press, vol. 31(1), pages 145-158.
    13. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    14. Livia Fritz & Ulli Vilsmaier & Garance Clément & Laurie Daffe & Anna Pagani & Melissa Pang & Daniel Gatica-Perez & Vincent Kaufmann & Marie Santiago Delefosse & Claudia R. Binder, 2022. "Explore, engage, empower: methodological insights into a transformative mixed methods study tackling the COVID-19 lockdown," Palgrave Communications, Palgrave Macmillan, vol. 9(1), pages 1-13, December.
    15. Robert Axelsson & Magnus Ljung & Malgorzata Blicharska & Michael Frisk & Marianne Henningsson & Grzegorz Mikusiński & Lennart Folkeson & Görgen Göransson & Sofia Jönsson-Ekström & Anders Sjölund & Jan, 2020. "The Challenge of Transdisciplinary Research: A Case Study of Learning by Evaluation for Sustainable Transport Infrastructures," Sustainability, MDPI, vol. 12(17), pages 1-24, August.
    16. Rick Hölsgens & Eva Wascher & Carolin Bauer & Judith Boll & Stephanie Bund & Saskia Dankwart-Kammoun & Irina Heese & Katharina Schrot & Jürgen Schultze & Robert Tenambergen, 2023. "Transdisciplinary Research along the Logic of Empowerment : Perspectives from Four Urban and Regional Transformation Projects," Sustainability, MDPI, vol. 15(5), pages 1-20, March.
    17. Gerlak, Andrea K. & Guido, Zack & Owen, Gigi & McGoffin, Mariana Sofia Rodriguez & Louder, Elena & Davies, Julia & Smith, Kelly Jay & Zimmer, Andy & Murveit, Anna M. & Meadow, Alison & Shrestha, Padme, 2023. "Stakeholder engagement in the co-production of knowledge for environmental decision-making," World Development, Elsevier, vol. 170(C).
    18. Livia Fritz & Claudia R. Binder, 2018. "Participation as Relational Space: A Critical Approach to Analysing Participation in Sustainability Research," Sustainability, MDPI, vol. 10(8), pages 1-29, August.
    19. Joanne Vinke‐de Kruijf & Laura Verbrugge & Barbara Schröter & Robert‐Jan den Haan & Juliette Cortes Arevalo & Jan Fliervoet & Jennifer Henze & Christian Albert, 2022. "Knowledge co‐production and researcher roles in transdisciplinary environmental management projects," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(2), pages 393-405, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stina Hansson & Merritt Polk, 2019. "Comments to Belcher et al. 2018’s critique of Hansson and Polk 2018," Research Evaluation, Oxford University Press, vol. 28(2), pages 202-205.
    2. Cian O’Donovan & Aleksandra (Ola) Michalec & Joshua R Moon, 2022. "Capabilities for transdisciplinary research," Research Evaluation, Oxford University Press, vol. 31(1), pages 145-158.
    3. de Jong, Stefan P.L. & Wardenaar, Tjerk & Horlings, Edwin, 2016. "Exploring the promises of transdisciplinary research: A quantitative study of two climate research programmes," Research Policy, Elsevier, vol. 45(7), pages 1397-1409.
    4. Laurens K. Hessels & Stefan P.L. De Jong & Stijn Brouwer, 2018. "Collaboration between Heterogeneous Practitioners in Sustainability Research: A Comparative Analysis of Three Transdisciplinary Programmes," Sustainability, MDPI, vol. 10(12), pages 1-16, December.
    5. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    6. Chams, Nour & Guesmi, Bouali & Gil, Jose M. & Molins, Mireia & Cubel, Rosa, 2021. "Between “Research Producers” and “Research Adopters”: The Role of Knowledge and Innovation Transfer on Sustainability Impact," 2021 Conference, August 17-31, 2021, Virtual 315264, International Association of Agricultural Economists.
    7. Stephanie Jahn & Jens Newig & Daniel J. Lang & Judith Kahle & Matthias Bergmann, 2022. "Demarcating transdisciplinary research in sustainability science—Five clusters of research modes based on evidence from 59 research projects," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(2), pages 343-357, April.
    8. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    9. Edwards, David M. & Meagher, Laura R., 2020. "A framework to evaluate the impacts of research on policy and practice: A forestry pilot study," Forest Policy and Economics, Elsevier, vol. 114(C).
    10. Katja Brundiers & Arnim Wiek & Braden Kay, 2013. "The Role of Transacademic Interface Managers in Transformational Sustainability Research and Education," Sustainability, MDPI, vol. 5(11), pages 1-23, October.
    11. Morgan, Edward A. & Osborne, Natalie & Mackey, Brendan, 2022. "Evaluating planning without plans: Principles, criteria and indicators for effective forest landscape approaches," Land Use Policy, Elsevier, vol. 115(C).
    12. Gregory Trencher & Masafumi Nagao & Chiahsin Chen & Kentaro Ichiki & Tobai Sadayoshi & Mariko Kinai & Mio Kamitani & Shojiro Nakamura & Aiko Yamauchi & Masaru Yarime, 2017. "Implementing Sustainability Co-Creation between Universities and Society: A Typology-Based Understanding," Sustainability, MDPI, vol. 9(4), pages 1-28, April.
    13. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    14. Jovana Janinovic & Sanja Pekovic & Dijana Vuckovic & Stevo Popovic & Rajka Djokovic & Mirjana Pejiæ Bach, 2020. "Innovative strategies for creating and assessing research quality and societal impact in social sciences and humanities," Interdisciplinary Description of Complex Systems - scientific journal, Croatian Interdisciplinary Society Provider Homepage: http://indecs.eu, vol. 18(4), pages 449-458.
    15. Bührer, Susanne & Feidenheimer, Alexander & Walz, Rainer & Lindner, Ralf & Beckert, Bernd & Wallwaey, Elisa, 2022. "Concepts and methods to measure societal impacts: An overview," Discussion Papers "Innovation Systems and Policy Analysis" 74, Fraunhofer Institute for Systems and Innovation Research (ISI).
    16. Livia Fritz & Claudia R. Binder, 2018. "Participation as Relational Space: A Critical Approach to Analysing Participation in Sustainability Research," Sustainability, MDPI, vol. 10(8), pages 1-29, August.
    17. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    18. Ramirez, Luisa F. & Belcher, Brian M., 2020. "Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru," Forest Policy and Economics, Elsevier, vol. 114(C).
    19. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    20. Benedikt Fecher & Freia Kuper & Nataliia Sokolovska & Alex Fenton & Stefan Hornbostel & Gert G. Wagner, 2021. "Understanding the Societal Impact of the Social Sciences and Humanities: Remarks on Roles, Challenges, and Expectations," RatSWD Working Papers 276, German Data Forum (RatSWD).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:27:y:2018:i:2:p:132-144.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.