IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v102y2024ics014971892300143x.html
   My bibliography  Save this article

Evaluation in design science: A framework to support project studies in the context of University Research Centres

Author

Listed:
  • Moutinho, José da Assunção
  • Fernandes, Gabriela
  • Rabechini, Roque

Abstract

This paper proposes a discussion of the evaluation of an artefact developed under the Design Science paradigm using the Delphi method. It evaluates the Ecosystem framework of University Research Centres in Project Studies, considering a set of criteria pre-established in the literature. The Delphi method is an evaluation implemented in an electronic platform involving twenty-one participants, among whom were academics, practitioners, and PhD candidates in the field of project management. It reached consensus and stability in two rounds: the results indicate a consensus among the participants in the applicability, novelty, simplicity, completeness, fidelity to modelled phenomena, consistency and internal coherence, scalability, flexibility, interest, elegance, and reusability criteria. Usability was the only criterion that did not attain the predefined percentage of consensus among the participants (70%). Given the framework’s characteristics, Delphi participants indicated the need to produce complementary guidelines for its implementation.

Suggested Citation

  • Moutinho, José da Assunção & Fernandes, Gabriela & Rabechini, Roque, 2024. "Evaluation in design science: A framework to support project studies in the context of University Research Centres," Evaluation and Program Planning, Elsevier, vol. 102(C).
    • Handle: RePEc:eee:epplan:v:102:y:2024:i:c:s014971892300143x
    DOI: 10.1016/j.evalprogplan.2023.102366
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S014971892300143X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2023.102366?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Siemieniako, Dariusz & Kubacki, Krzysztof & Mitręga, Maciej, 2021. "Inter-organisational relationships for social impact: A systematic literature review," Journal of Business Research, Elsevier, vol. 132(C), pages 453-469.
    2. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    3. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    4. John Venable & Jan Pries-Heje & Richard Baskerville, 2016. "FEDS: a Framework for Evaluation in Design Science Research," European Journal of Information Systems, Taylor & Francis Journals, vol. 25(1), pages 77-89, January.
    5. Rowe, Gene & Wright, George, 1996. "The impact of task characteristics on the performance of structured group forecasting techniques," International Journal of Forecasting, Elsevier, vol. 12(1), pages 73-89, March.
    6. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    7. Ekaterina Albats & Irina Fiegenbaum & James A. Cunningham, 2018. "A micro level study of university industry collaborative lifecycle key performance indicators," The Journal of Technology Transfer, Springer, vol. 43(2), pages 389-431, April.
    8. A. Georges L. Romme, 2003. "Making a Difference: Organization as Design," Organization Science, INFORMS, vol. 14(5), pages 558-573, October.
    9. Bill Kuechler & Vijay Vaishnavi, 2008. "On theory development in design science research: anatomy of a research project," European Journal of Information Systems, Taylor & Francis Journals, vol. 17(5), pages 489-504, October.
    10. Lee Cronbach, 1951. "Coefficient alpha and the internal structure of tests," Psychometrika, Springer;The Psychometric Society, vol. 16(3), pages 297-334, September.
    11. Victoria Galan-Muros & Todd Davey, 2019. "The UBC ecosystem: putting together a comprehensive framework for university-business cooperation," The Journal of Technology Transfer, Springer, vol. 44(4), pages 1311-1346, August.
    12. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    13. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    14. Gilbert Silvius & Ron Schipper, 2019. "Planning Project Stakeholder Engagement from a Sustainable Development Perspective," Administrative Sciences, MDPI, vol. 9(2), pages 1-22, June.
    15. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    2. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    3. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    4. Sandeep Singhai & Ritika Singh & Harish Kumar Sardana & Anuradha Madhukar, 2021. "Analysis of Factors Influencing Technology Transfer: A Structural Equation Modeling Based Approach," Sustainability, MDPI, vol. 13(10), pages 1-15, May.
    5. Delke, Vincent & Schiele, Holger & Buchholz, Wolfgang & Kelly, Stephen, 2023. "Implementing Industry 4.0 technologies: Future roles in purchasing and supply management," Technological Forecasting and Social Change, Elsevier, vol. 196(C).
    6. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    7. Bokrantz, Jon & Skoogh, Anders & Berlin, Cecilia & Stahre, Johan, 2017. "Maintenance in digitalised manufacturing: Delphi-based scenarios for 2030," International Journal of Production Economics, Elsevier, vol. 191(C), pages 154-169.
    8. Belton, Ian & Wright, George & Sissons, Aileen & Bolger, Fergus & Crawford, Megan M. & Hamlin, Iain & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina, 2021. "Delphi with feedback of rationales: How large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged?," Technological Forecasting and Social Change, Elsevier, vol. 170(C).
    9. Joshua Holstein & Max Schemmer & Johannes Jakubik & Michael Vössing & Gerhard Satzger, 2023. "Sanitizing data for analysis: Designing systems for data understanding," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-18, December.
    10. C. J. Torrecilla-Salinas & O. Troyer & M. J. Escalona & M. Mejías, 2019. "A Delphi-based expert judgment method applied to the validation of a mature Agile framework for Web development projects," Information Technology and Management, Springer, vol. 20(1), pages 9-40, March.
    11. Wei Qi & Xiumei Guo & Xia Wu & Dora Marinova & Jin Fan, 2018. "Do the sunk cost effect and cognitive dissonance increase risk perception? An empirical study in the context of city smog," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(5), pages 2269-2289, September.
    12. Mauksch, Stefanie & von der Gracht, Heiko A. & Gordon, Theodore J., 2020. "Who is an expert for foresight? A review of identification methods," Technological Forecasting and Social Change, Elsevier, vol. 154(C).
    13. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151.
    14. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    15. Ståle Navrud & Jon Strand, 2018. "Valuing Global Ecosystem Services: What Do European Experts Say? Applying the Delphi Method to Contingent Valuation of the Amazon Rainforest," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 70(1), pages 249-269, May.
    16. Flostrand, Andrew & Pitt, Leyland & Bridson, Shannon, 2020. "The Delphi technique in forecasting– A 42-year bibliographic analysis (1975–2017)," Technological Forecasting and Social Change, Elsevier, vol. 150(C).
    17. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    18. Antonelli, Marta & Basile, Linda & Gagliardi, Francesca & Isernia, Pierangelo, 2022. "The future of the Mediterranean agri-food systems: Trends and perspectives from a Delphi survey," Land Use Policy, Elsevier, vol. 120(C).
    19. von Briel, Frederik, 2018. "The future of omnichannel retail: A four-stage Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 132(C), pages 217-229.
    20. Uwe Cantner & Martin Kalthaus & Indira Yarullina, 2022. "Outcomes of Science-Industry Collaboration: Factors and Interdependencies," Jena Economics Research Papers 2022-003, Friedrich-Schiller-University Jena.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:102:y:2024:i:c:s014971892300143x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.