IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v147y2019icp72-82.html
   My bibliography  Save this article

Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process

Author

Listed:
  • Belton, Ian
  • MacDonald, Alice
  • Wright, George
  • Hamlin, Iain

Abstract

This paper provides a practical, systematic approach to the design and delivery of a Delphi survey. We prescribe a sequence of six steps to do with (i) setting up the Delphi process – including selecting respondents and generating a requisite number of focal issues, (ii) software/delivery choice, (iii) developing question items and response scales, (iv) providing feedback between a requisite number of Delphi rounds, (v) preventing and dealing with panellist drop out, and (vi) analysing and presenting the Delphi yield. At each step, the Delphi administrator has a range of choice options and we provide discussion of the pros and cons of each option - in order that the overall design and delivery of a particular Delphi survey is both well-founded and defensible.

Suggested Citation

  • Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    • Handle: RePEc:eee:tefoso:v:147:y:2019:i:c:p:72-82
    DOI: 10.1016/j.techfore.2019.07.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162519306572
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2019.07.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. David M. Boje & J. Keith Murnighan, 1982. "Group Confidence Pressures in Iterative Decisions," Management Science, INFORMS, vol. 28(10), pages 1187-1196, October.
    2. Hung, Hsin-Ling & Altschuld, James W. & Lee, Yi-Fang, 2008. "Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation," Evaluation and Program Planning, Elsevier, vol. 31(2), pages 191-198, May.
    3. Weijters, Bert & Cabooter, Elke & Schillewaert, Niels, 2010. "The effect of rating scale format on response styles: The number of response categories and response category labels," International Journal of Research in Marketing, Elsevier, vol. 27(3), pages 236-247.
    4. de Loë, Rob C. & Melnychuk, Natalya & Murray, Dan & Plummer, Ryan, 2016. "Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities," Technological Forecasting and Social Change, Elsevier, vol. 104(C), pages 78-88.
    5. Moonjung Choi & Han-Lim Choi, 2015. "Foresight for Science and Technology Priority Setting in Korea," Foresight-Russia Форсайт, CyberLeninka;Федеральное государственное автономное образовательное учреждение высшего образования «Национальный исследовательский университет «Высшая школа экономики», vol. 9(3 (eng)), pages 54-65.
    6. Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
    7. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    8. von der Gracht, Heiko A. & Darkow, Inga-Lena, 2010. "Scenarios for the logistics services industry: A Delphi-based analysis for 2025," International Journal of Production Economics, Elsevier, vol. 127(1), pages 46-59, September.
    9. Aengenheyster, Stefan & Cuhls, Kerstin & Gerhold, Lars & Heiskanen-Schüttler, Maria & Huck, Jana & Muszynska, Monika, 2017. "Real-Time Delphi in practice — A comparative analysis of existing software-based tools," Technological Forecasting and Social Change, Elsevier, vol. 118(C), pages 15-27.
    10. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151, January.
    11. repec:eee:intfor:v:27:y:2011:i:1:p:1-13 is not listed on IDEAS
    12. Nick Novakowski & Barry Wellar, 2008. "Using the Delphi Technique in Normative Planning Research: Methodological Design Considerations," Environment and Planning A, , vol. 40(6), pages 1485-1500, June.
    13. Moonjung Choi & Han-Lim Choi, 2015. "Foresight for Science and Technology Priority Setting in Korea," Foresight and STI Governance (Foresight-Russia till No. 3/2015), National Research University Higher School of Economics, vol. 9(3), pages 54-65.
    14. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    15. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    16. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    17. von der Gracht, Heiko A., 2012. "Consensus measurement in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 79(8), pages 1525-1536.
    18. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151.
    19. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    20. Geist, Monica R., 2010. "Using the Delphi method to engage stakeholders: A comparison of two studies," Evaluation and Program Planning, Elsevier, vol. 33(2), pages 147-154, May.
    21. Li, Na & Chen, Kaihua & Kou, Mingting, 2017. "Technology foresight in China: Academic studies, governmental practices and policy applications," Technological Forecasting and Social Change, Elsevier, vol. 119(C), pages 246-255.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Belton, Ian & Wright, George & Sissons, Aileen & Bolger, Fergus & Crawford, Megan M. & Hamlin, Iain & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina, 2021. "Delphi with feedback of rationales: How large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged?," Technological Forecasting and Social Change, Elsevier, vol. 170(C).
    2. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    3. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    4. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    5. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    6. Frevel, Nicolas & Beiderbeck, Daniel & Schmidt, Sascha L., 2022. "The impact of technology on sports – A prospective study," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    7. Gebhardt, Maximilian & Spieske, Alexander & Birkel, Hendrik, 2022. "The future of the circular economy and its effect on supply chain dependencies: Empirical evidence from a Delphi study," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).
    8. Fritschy, Carolin & Spinler, Stefan, 2019. "The impact of autonomous trucks on business models in the automotive and logistics industry–a Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 148(C).
    9. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    10. Peppel, Marcel & Ringbeck, Jürgen & Spinler, Stefan, 2022. "How will last-mile delivery be shaped in 2040? A Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 177(C).
    11. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    12. Engelke, Henning & Mauksch, Stefanie & Darkow, Inga-Lena & von der Gracht, Heiko A., 2015. "Opportunities for social enterprise in Germany — Evidence from an expert survey," Technological Forecasting and Social Change, Elsevier, vol. 90(PB), pages 635-646.
    13. Bonaccorsi, Andrea & Apreda, Riccardo & Fantoni, Gualtiero, 2020. "Expert biases in technology foresight. Why they are a problem and how to mitigate them," Technological Forecasting and Social Change, Elsevier, vol. 151(C).
    14. Beiderbeck, Daniel & Evans, Nicolas & Frevel, Nicolas & Schmidt, Sascha L., 2023. "The impact of technology on the future of football – A global Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 187(C).
    15. Schlecht, Laura & Schneider, Sabrina & Buchwald, Arne, 2021. "The prospective value creation potential of Blockchain in business models: A delphi study," Technological Forecasting and Social Change, Elsevier, vol. 166(C).
    16. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    17. Rengarajan, Srinath & Moser, Roger & Narayanamurthy, Gopalakrishnan, 2021. "Strategy tools in dynamic environments – An expert-panel study," Technological Forecasting and Social Change, Elsevier, vol. 165(C).
    18. Spickermann, Alexander & Grienitz, Volker & von der Gracht, Heiko A., 2014. "Heading towards a multimodal city of the future?," Technological Forecasting and Social Change, Elsevier, vol. 89(C), pages 201-221.
    19. de Loë, Rob C. & Melnychuk, Natalya & Murray, Dan & Plummer, Ryan, 2016. "Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities," Technological Forecasting and Social Change, Elsevier, vol. 104(C), pages 78-88.
    20. Kattirtzi, Michael & Winskel, Mark, 2020. "When experts disagree: Using the Policy Delphi method to analyse divergent expert expectations and preferences on UK energy futures," Technological Forecasting and Social Change, Elsevier, vol. 153(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:147:y:2019:i:c:p:72-82. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.