IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v25y2016i1p1-17..html
   My bibliography  Save this article

Defining and assessing research quality in a transdisciplinary context

Author

Listed:
  • Brian M. Belcher
  • Katherine E. Rasmussen
  • Matthew R. Kemshaw
  • Deborah A. Zornes

Abstract

Research increasingly seeks both to generate knowledge and to contribute to real-world solutions, with strong emphasis on context and social engagement. As boundaries between disciplines are crossed, and as research engages more with stakeholders in complex systems, traditional academic definitions and criteria of research quality are no longer sufficient—there is a need for a parallel evolution of principles and criteria to define and evaluate research quality in a transdisciplinary research (TDR) context. We conducted a systematic review to help answer the question: What are appropriate principles and criteria for defining and assessing TDR quality? Articles were selected and reviewed seeking: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, proposed principles of research quality, proposed criteria for research quality assessment, proposed indicators and measures of research quality, and proposed processes for evaluating TDR. We used the information from the review and our own experience in two research organizations that employ TDR approaches to develop a prototype TDR quality assessment framework, organized as an evaluation rubric. We provide an overview of the relevant literature and summarize the main aspects of TDR quality identified there. Four main principles emerge: relevance, including social significance and applicability; credibility, including criteria of integration and reflexivity, added to traditional criteria of scientific rigor; legitimacy, including criteria of inclusion and fair representation of stakeholder interests, and; effectiveness, with criteria that assess actual or potential contributions to problem solving and social change.

Suggested Citation

  • Brian M. Belcher & Katherine E. Rasmussen & Matthew R. Kemshaw & Deborah A. Zornes, 2016. "Defining and assessing research quality in a transdisciplinary context," Research Evaluation, Oxford University Press, vol. 25(1), pages 1-17.
  • Handle: RePEc:oup:rseval:v:25:y:2016:i:1:p:1-17.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvv025
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Cash, David & Clark, William & Alcock, Frank & Dickson, Nancy & Eckley, Noelle & Jager, Jill, 2002. "Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making," Working Paper Series rwp02-046, Harvard University, John F. Kennedy School of Government.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stina Hansson & Merritt Polk, 2018. "Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact," Research Evaluation, Oxford University Press, vol. 27(2), pages 132-144.
    2. Stephanie Jahn & Jens Newig & Daniel J. Lang & Judith Kahle & Matthias Bergmann, 2022. "Demarcating transdisciplinary research in sustainability science—Five clusters of research modes based on evidence from 59 research projects," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(2), pages 343-357, April.
    3. Flurina Schneider & Zarina Patel & Katsia Paulavets & Tobias Buser & Jacqueline Kado & Stefanie Burkhart, 2023. "Fostering transdisciplinary research for sustainability in the Global South: Pathways to impact for funding programmes," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-11, December.
    4. Ingrid Kofler, 2023. "Beyond Disciplinary Constraints: Designing Transdisciplinary Research and Collaboration in Real-World Laboratories," Societies, MDPI, vol. 13(9), pages 1-18, September.
    5. Ramirez, Luisa F. & Belcher, Brian M., 2020. "Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru," Forest Policy and Economics, Elsevier, vol. 114(C).
    6. Brian Belcher & Janet Halliwell, 2021. "Conceptualizing the elements of research impact: towards semantic standards," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-6, December.
    7. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    8. Stefanie Meilinger, 2022. "Research–Practice–Collaborations in Engineering," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 34(4), pages 1727-1734, August.
    9. Bührer, Susanne & Feidenheimer, Alexander & Walz, Rainer & Lindner, Ralf & Beckert, Bernd & Wallwaey, Elisa, 2022. "Concepts and methods to measure societal impacts: An overview," Discussion Papers "Innovation Systems and Policy Analysis" 74, Fraunhofer Institute for Systems and Innovation Research (ISI).
    10. Cian O’Donovan & Aleksandra (Ola) Michalec & Joshua R Moon, 2022. "Capabilities for transdisciplinary research," Research Evaluation, Oxford University Press, vol. 31(1), pages 145-158.
    11. Susan Roelofs & Nancy Edwards & Sarah Viehbeck & Cody Anderson, 2019. "Formative, embedded evaluation to strengthen interdisciplinary team science: Results of a 4-year, mixed methods, multi-country case study," Research Evaluation, Oxford University Press, vol. 28(1), pages 37-50.
    12. Ratner, B. D. & Dubois, Mark J. & Morrison, T. H. & Tezzo, X. & Song, A. M. & Mbaru, E. & Chimatiro, S. K. & Cohen, P. J., 2022. "A framework to guide research engagement in the policy process, with application to small-scale fisheries," Papers published in Journals (Open Access), International Water Management Institute, pages 1-27(4):45..
    13. Shunshun Shi & Wenyu Zhang & Shuai Zhang & Jie Chen, 2018. "Does prestige dimension influence the interdisciplinary performance of scientific entities in knowledge flow? Evidence from the e-government field," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 1237-1264, November.
    14. Marlen Gabriele Arnold, 2022. "The challenging role of researchers coping with tensions, dilemmas and paradoxes in transdisciplinary settings," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(2), pages 326-342, April.
    15. Sofie Sandin & Mats Benner, 2022. "Research evaluations for an energy transition? Insights from a review of Swedish research evaluation reports," Research Evaluation, Oxford University Press, vol. 31(1), pages 80-92.
    16. Marina Knickel & Karlheinz Knickel & Francesca Galli & Damian Maye & Johannes S. C. Wiskerke, 2019. "Towards a Reflexive Framework for Fostering Co—Learning and Improvement of Transdisciplinary Collaboration," Sustainability, MDPI, vol. 11(23), pages 1-22, November.
    17. Brian M Belcher & Luisa F Ramirez & Rachel Davel & Rachel Claus, 2019. "A response to Hansson and Polk (2018) “Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impac," Research Evaluation, Oxford University Press, vol. 28(2), pages 196-201.
    18. Leonhard Hennen & Jürgen Kopfmüller & Maria Maia & Linda Nierling & Constanze Scherz, 2023. "Ways towards Transformation—Conceptual Approaches and Challenges," Sustainability, MDPI, vol. 15(20), pages 1-27, October.
    19. Stina Hansson & Merritt Polk, 2019. "Comments to Belcher et al. 2018’s critique of Hansson and Polk 2018," Research Evaluation, Oxford University Press, vol. 28(2), pages 202-205.
    20. Bethany K Laursen & Nicole Motzer & Kelly J Anderson, 2022. "Pathways for assessing interdisciplinarity: A systematic review," Research Evaluation, Oxford University Press, vol. 31(3), pages 326-343.
    21. Lawarée, Justin & Jacob, Steve & Ouimet, Mathieu, 2020. "A scoping review of knowledge syntheses in the field of evaluation across four decades of practice," Evaluation and Program Planning, Elsevier, vol. 79(C).
    22. Vučković Dijana & Pekovic Sanja & Popović Stevo & Janinovic Jovana, 2023. "Assessing the Appraisal of Research Quality in Social Sciences and Humanities: A Case Study of the University of Montenegro," Business Systems Research, Sciendo, vol. 14(1), pages 131-152, September.
    23. Martin Ricker, 2017. "Letter to the Editor: About the quality and impact of scientific articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1851-1855, June.
    24. Sandra Rousseau & Ronald Rousseau, 2021. "Bibliometric Techniques And Their Use In Business And Economics Research," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1428-1451, December.
    25. Katja Bender, 2022. "Research–Practice–Collaborations in International Sustainable Development and Knowledge Production: Reflections from a Political-Economic Perspective," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 34(4), pages 1691-1703, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kristjanson, Patti & Reid, Robin & Dickson, Nancy & Clark, William C. & Vishnubhotla, Prasad & Romney, Dannie & Bezkorowajnyj, Peter & Said, Mohammed & Kaelo, Dickson & Makui, Ogeli & Nkedianye, David, 2008. "Linking International Agricultural Research Knowledge with Action for Sustainable Poverty Alleviation: What Works?," Working Paper Series rwp08-045, Harvard University, John F. Kennedy School of Government.
    2. Zeigermann, Ulrike & Böcher, Michael, 2020. "Challenges for bridging the gap between knowledge and governance in sustainability policy – The case of OECD ‘Focal Points’ for Policy Coherence for Development," Forest Policy and Economics, Elsevier, vol. 114(C).
    3. Bullock Graham, 2015. "Signaling the credibility of private actors as public agents: transparency, independence, and expertise in environmental evaluations of products and companies," Business and Politics, De Gruyter, vol. 17(2), pages 177-219, August.
    4. Gillian L. Galford & Julie Nash & Alan K. Betts & Sam Carlson & Sarah Ford & Ann Hoogenboom & Deborah Markowitz & Andrew Nash & Elizabeth Palchak & Sarah Pears & Kristen L. Underwood, 2016. "Bridging the climate information gap: a framework for engaging knowledge brokers and decision makers in state climate assessments," Climatic Change, Springer, vol. 138(3), pages 383-395, October.
    5. Dragana Bojovic & Andria Nicodemou & Asun Lera St.Clair & Isadora Christel & Francisco J. Doblas-Reyes, 2022. "Exploring the landscape of seasonal forecast provision by Global Producing Centres," Climatic Change, Springer, vol. 172(1), pages 1-23, May.
    6. Sébastien Chailleux, 2020. "Making the subsurface political: How enhanced oil recovery techniques reshaped the energy transition," Environment and Planning C, , vol. 38(4), pages 733-750, June.
    7. Garrett Ward Richards, 2019. "The Science–Policy Relationship Hierarchy (SPRHi) model of co-production: how climate science organizations have influenced the policy process in Canadian case studies," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(1), pages 67-95, March.
    8. Pieter T. Boer & Geert W. J. Frederix & Talitha L. Feenstra & Pepijn Vemer, 2016. "Unremarked or Unperformed? Systematic Review on Reporting of Validation Efforts of Health Economic Decision Models in Seasonal Influenza and Early Breast Cancer," PharmacoEconomics, Springer, vol. 34(9), pages 833-845, September.
    9. Kenji Otsuka, 2022. "Co‐optation in co‐production: Maintaining credibility and legitimacy in transboundary environmental governance in East Asia," Review of Policy Research, Policy Studies Organization, vol. 39(6), pages 771-797, November.
    10. Krister Andersson, 2008. "Motivation to Engage in Social Learning about Sustainability: An Institutional Analysis," CID Working Papers 26, Center for International Development at Harvard University.
    11. Nadia Sitas & Heidi E. Prozesky & Karen J. Esler & Belinda Reyers, 2014. "Exploring the Gap between Ecosystem Service Research and Management in Development Planning," Sustainability, MDPI, vol. 6(6), pages 1-23, June.
    12. Nils C. Bandelow & Johanna Hornung, 2022. "Narratives, evidence and public policy in crisis situations," Review of Policy Research, Policy Studies Organization, vol. 39(6), pages 704-707, November.
    13. McDermott, T.K.J. & Surminski, S., 2018. "Normative interpretations of climate risk assessment and how it affects local decision making – a study at the city scale in Cork, Ireland," Working Papers 309607, National University of Ireland, Galway, Socio-Economic Marine Research Unit.
    14. Coughlan de Perez, Erin & Stephens, Elisabeth & van Aalst, Maarten & Bazo, Juan & Fournier-Tombs, Eleonore & Funk, Sebastian & Hess, Jeremy J. & Ranger, Nicola & Lowe, Rachel, 2022. "Epidemiological versus meteorological forecasts: Best practice for linking models to policymaking," International Journal of Forecasting, Elsevier, vol. 38(2), pages 521-526.
    15. Doyeon Lee & Keunhwan Kim, 2022. "Public R&D Projects-Based Investment and Collaboration Framework for an Overarching South Korean National Strategy of Personalized Medicine," IJERPH, MDPI, vol. 19(3), pages 1-25, January.
    16. Felix Creutzig & Christoph von Stechow & David Klein & Carol Hunsberger & Nico Bauer & Alexander Popp & Ottmar Edenhofer, 2012. "Can Bioenergy Assessments Deliver?," Economics of Energy & Environmental Policy, International Association for Energy Economics, vol. 0(Number 2).
    17. Gudmundsson, Henrik & Ericsson, Eva & Hugosson, Muriel Beser & Rosqvist, Lena Smidfelt, 2009. "Framing the role of Decision Support in the case of Stockholm Congestion Charging Trial," Transportation Research Part A: Policy and Practice, Elsevier, vol. 43(3), pages 258-268, March.
    18. Stina Hansson & Merritt Polk, 2019. "Comments to Belcher et al. 2018’s critique of Hansson and Polk 2018," Research Evaluation, Oxford University Press, vol. 28(2), pages 202-205.
    19. Leimona, Beria & Lusiana, Betha & van Noordwijk, Meine & Mulyoutami, Elok & Ekadinata, Andree & Amaruzaman, Sacha, 2015. "Boundary work: Knowledge co-production for negotiating payment for watershed services in Indonesia," Ecosystem Services, Elsevier, vol. 15(C), pages 45-62.
    20. de Jager, Monique & Hengeveld, Geerten M. & Mooij, Wolf M. & Slooten, Elisabeth, 2019. "Modelling the spatial dynamics of Maui dolphins using individual-based models," Ecological Modelling, Elsevier, vol. 402(C), pages 59-65.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:25:y:2016:i:1:p:1-17.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.