IDEAS home Printed from https://ideas.repec.org/a/wly/fufsci/v6y2024i1ne172.html

Quality indicators for Delphi studies

Author

Listed:
  • Jon Landeta
  • Aitziber Lertxundi

Abstract

The Delphi method is a technique of social research that seeks to obtain a reliable group opinion from experts. It was first created for military purposes in the mid‐1950s. Since then, its use in the scientific field has continued to spread to different disciplines and aims. Despite this expansion, however, not set of indicators of the quality of Delphi studies has yet to be developed that might provide the reader—whether an expert in the technique or not—with some framework of reference whereby to gauge what credibility should be afforded to the results of the study. In this paper, following a thorough review of the literature on the criteria used to assess Delphi studies and the items of evaluation recommended for inclusion in Delphi reports, we determine what characteristics a quality evaluation indicator for this technique should have and propose a battery of indicators based on these characteristics, which should for preference be included in the final report of a Delphi study. The proposed indicators focus on three areas that are particularly relevant to the quality of Delphi research: the quality of the panel of participating experts, the way in which relevant information is obtained from the experts, and the quality of the interaction generated among the experts.

Suggested Citation

  • Jon Landeta & Aitziber Lertxundi, 2024. "Quality indicators for Delphi studies," Futures & Foresight Science, John Wiley & Sons, vol. 6(1), March.
    • Handle: RePEc:wly:fufsci:v:6:y:2024:i:1:n:e172
    DOI: 10.1002/ffo2.172
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/ffo2.172
    Download Restriction: no
    File URL: https://libkey.io/10.1002/ffo2.172?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Pill, Juri, 1971. "The Delphi method: Substance, context, a critique and an annotated bibliography," Socio-Economic Planning Sciences, Elsevier, vol. 5(1), pages 57-71, February.
    2. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151, January.
    3. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    4. Cynthia P. Iglesias & Alexander Thompson & Wolf H. Rogowski & Katherine Payne, 2016. "Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations," PharmacoEconomics, Springer, vol. 34(11), pages 1161-1172, November.
    5. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151.
    6. Mauksch, Stefanie & von der Gracht, Heiko A. & Gordon, Theodore J., 2020. "Who is an expert for foresight? A review of identification methods," Technological Forecasting and Social Change, Elsevier, vol. 154(C).
    7. Olaf Helmer & Nicholas Rescher, 1959. "On the Epistemology of the Inexact Sciences," Management Science, INFORMS, vol. 6(1), pages 25-52, October.
    8. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Che-Jen Su & Xin Wang & Cheng-Fang Hu & Kunihiko Moriya & Nicolas Lorgnier & Yi-Fang Lan & Jun Liu, 2025. "Research priorities through the lens of remote work in hospitality, leisure, sports, and tourism," Service Business, Springer;Pan-Pacific Business Association, vol. 19(3), pages 1-39, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    2. Aldossary, Naief A. & Rezgui, Yacine & Kwan, Alan, 2015. "Consensus-based low carbon domestic design framework for sustainable homes," Renewable and Sustainable Energy Reviews, Elsevier, vol. 51(C), pages 417-432.
    3. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    4. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    5. Engelke, Henning & Mauksch, Stefanie & Darkow, Inga-Lena & von der Gracht, Heiko A., 2015. "Opportunities for social enterprise in Germany — Evidence from an expert survey," Technological Forecasting and Social Change, Elsevier, vol. 90(PB), pages 635-646.
    6. Badir S. Alsaeed & Dexter V. L. Hunt & Soroosh Sharifi, 2024. "A Sustainable Water Resources Management Assessment Framework (SWRM-AF) for Arid and Semi-Arid Regions—Part 2: Refining the Conceptual Framework Using the Delphi Technique," Sustainability, MDPI, vol. 16(11), pages 1-26, May.
    7. Alcon, Francisco & Tapsuwan, Sorada & Martínez-Paz, José M. & Brouwer, Roy & de Miguel, María D., 2014. "Forecasting deficit irrigation adoption using a mixed stakeholder assessment methodology," Technological Forecasting and Social Change, Elsevier, vol. 83(C), pages 183-193.
    8. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    9. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    10. Kattirtzi, Michael & Winskel, Mark, 2020. "When experts disagree: Using the Policy Delphi method to analyse divergent expert expectations and preferences on UK energy futures," Technological Forecasting and Social Change, Elsevier, vol. 153(C).
    11. Isabela Caroline de Sousa & Tiago F. A. C. Sigahi & Izabela Simon Rampasso & Gustavo Hermínio Salati Marcondes de Moraes & Walter Leal Filho & João Henrique Paulino Pires Eustachio & Rosley Anholon, 2024. "A Delphi–Fuzzy Delphi Study on SDGs 9 and 12 after COVID-19: Case Study in Brazil," Forecasting, MDPI, vol. 6(3), pages 1-18, July.
    12. Horrillo, A. & Escribano, M. & Mesias, F.J. & Elghannam, A. & Gaspar, P., 2016. "Is there a future for organic production in high ecological value ecosystems?," Agricultural Systems, Elsevier, vol. 143(C), pages 114-125.
    13. C. J. Torrecilla-Salinas & O. Troyer & M. J. Escalona & M. Mejías, 2019. "A Delphi-based expert judgment method applied to the validation of a mature Agile framework for Web development projects," Information Technology and Management, Springer, vol. 20(1), pages 9-40, March.
    14. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    15. Sebastian Hinderer & Leif Brändle & Andreas Kuckertz, 2021. "Transition to a Sustainable Bioeconomy," Sustainability, MDPI, vol. 13(15), pages 1-16, July.
    16. Hyun Baek & Sun-Kyoung Park, 2015. "Sustainable Development Plan for Korea through Expansion of Green IT: Policy Issues for the Effective Utilization of Big Data," Sustainability, MDPI, vol. 7(2), pages 1-21, January.
    17. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    18. Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
    19. Winkler, Jens & Kuklinski, Christian Paul Jian-Wei & Moser, Roger, 2015. "Decision making in emerging markets: The Delphi approach's contribution to coping with uncertainty and equivocality," Journal of Business Research, Elsevier, vol. 68(5), pages 1118-1126.
    20. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fufsci:v:6:y:2024:i:1:n:e172. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)2573-5152 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.