IDEAS home Printed from https://ideas.repec.org/a/wly/fufsci/v4y2022i2ne2118.html
   My bibliography  Save this article

A critical evaluation of 42, large‐scale, science and technology foresight Delphi surveys

Author

Listed:
  • Ian Belton
  • Kerstin Cuhls
  • George Wright

Abstract

Large‐scale Delphi surveys on technology foresight started in the 1960s and involve an average of about 2000 participants answering, potentially, up to about 450 items. This contrasts sharply with the participation and content of the more common, smaller‐scale Delphi surveys. Previously, Belton et al. developed “six steps” to underpin a well‐founded and defensible Delphi process and we apply these steps in a novel evaluation of the quality of 42 large‐scale technology foresight surveys. Using a detailed analysis of two exemplar studies and a content analysis of all 42 surveys, we explore whether such surveys differ systematically from “traditional” smaller‐scale Delphi surveys and, if so, why this may be and what it may mean for the quality of data produced. We conclude that there are some problematic issues within these surveys—to do with (i) data quality in both the numerical summarizing of participant's between‐round feedback and in the reporting of final round numerical responses, (ii) the infrequent elicitation of rationales to justify participants' proffered numerical responses, and, when such rationales are elicited, (iii) the between‐round summary and presentation of the rationales. We speculate on the reasons for these design differences in the extant large‐scale surveys and conclude that extra‐survey political influences, such as differing objectives and the need to demonstrate wide‐ranging expert participation, may be the underlying cause. We conclude with considerations and recommendations for the design of future large‐scale Delphi surveys to enable the underlying process to become better‐founded and more defensible to procedural evaluation.

Suggested Citation

  • Ian Belton & Kerstin Cuhls & George Wright, 2022. "A critical evaluation of 42, large‐scale, science and technology foresight Delphi surveys," Futures & Foresight Science, John Wiley & Sons, vol. 4(2), June.
  • Handle: RePEc:wly:fufsci:v:4:y:2022:i:2:n:e2118
    DOI: 10.1002/ffo2.118
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/ffo2.118
    Download Restriction: no

    File URL: https://libkey.io/10.1002/ffo2.118?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mohamed Ramadan A. Rezk & Amr Radwan & Nahed Salem & Mahmoud M. Sakr & Manuela TvaronaviÄ ienÄ—, 2019. "Foresight for sustainable energy policy in Egypt: results from a Delphi survey," Insights into Regional Development, VsI Entrepreneurship and Sustainability Center, vol. 1(4), pages 357-369, December.
    2. Celiktas, Melih Soner & Kocar, Gunnur, 2010. "From potential forecast to foresight of Turkey's renewable energy with Delphi approach," Energy, Elsevier, vol. 35(5), pages 1973-1980.
    3. Tang, Yong & Sun, Honghang & Yao, Qiang & Wang, Yibo, 2014. "The selection of key technologies by the silicon photovoltaic industry based on the Delphi method and AHP (analytic hierarchy process): Case study of China," Energy, Elsevier, vol. 75(C), pages 474-482.
    4. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    5. Rowe, Gene & Wright, George, 1996. "The impact of task characteristics on the performance of structured group forecasting techniques," International Journal of Forecasting, Elsevier, vol. 12(1), pages 73-89, March.
    6. Attila Havas, 2003. "Evolving foresight in a small transition economy," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 22(2-3), pages 179-201.
    7. Georg Aichholzer, 2001. "Delphi Austria - An Example of Tailoring Foresight to the Needs of a Small Country," ITA manu:scripts 01_02, Institute of Technology Assessment (ITA).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    2. Chun-Chieh Tseng & Jun-Yi Zeng & Min-Liang Hsieh & Chih-Hung Hsu, 2022. "Analysis of Innovation Drivers of New and Old Kinetic Energy Conversion Using a Hybrid Multiple-Criteria Decision-Making Model in the Post-COVID-19 Era: A Chinese Case," Mathematics, MDPI, vol. 10(20), pages 1-25, October.
    3. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    4. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    5. Daniel E. O'Leary, 2007. "Knowledge representation of rules: a note," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 15(1‐2), pages 73-84, January.
    6. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    7. Song, Haiyan & Gao, Bastian Z. & Lin, Vera S., 2013. "Combining statistical and judgmental forecasts via a web-based tourism demand forecasting system," International Journal of Forecasting, Elsevier, vol. 29(2), pages 295-310.
    8. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    9. Billig, Eric & Thrän, Daniela, 2016. "Evaluation of biomethane technologies in Europe – Technical concepts under the scope of a Delphi-Survey embedded in a multi-criteria analysis," Energy, Elsevier, vol. 114(C), pages 1176-1186.
    10. Perera, H. Niles & Hurley, Jason & Fahimnia, Behnam & Reisi, Mohsen, 2019. "The human factor in supply chain forecasting: A systematic review," European Journal of Operational Research, Elsevier, vol. 274(2), pages 574-600.
    11. Makkonen, Mari & Pätäri, Satu & Jantunen, Ari & Viljainen, Satu, 2012. "Competition in the European electricity markets – outcomes of a Delphi study," Energy Policy, Elsevier, vol. 44(C), pages 431-440.
    12. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    13. Gebhardt, Maximilian & Spieske, Alexander & Birkel, Hendrik, 2022. "The future of the circular economy and its effect on supply chain dependencies: Empirical evidence from a Delphi study," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).
    14. Wentholt, M.T.A. & Rowe, G. & König, A. & Marvin, H.J.P. & Frewer, L.J., 2009. "The views of key stakeholders on an evolving food risk governance framework: Results from a Delphi study," Food Policy, Elsevier, vol. 34(6), pages 539-548, December.
    15. Kauko, Karlo & Palmroos, Peter, 2014. "The Delphi method in forecasting financial markets— An experimental study," International Journal of Forecasting, Elsevier, vol. 30(2), pages 313-327.
    16. Delke, Vincent & Schiele, Holger & Buchholz, Wolfgang & Kelly, Stephen, 2023. "Implementing Industry 4.0 technologies: Future roles in purchasing and supply management," Technological Forecasting and Social Change, Elsevier, vol. 196(C).
    17. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    18. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    19. Nur, Suardi & Burton, Bruce & Bergmann, Ariel, 2023. "Evidence on optimal risk allocation models for Indonesian geothermal projects under PPP contracts," Utilities Policy, Elsevier, vol. 81(C).
    20. Bokrantz, Jon & Skoogh, Anders & Berlin, Cecilia & Stahre, Johan, 2017. "Maintenance in digitalised manufacturing: Delphi-based scenarios for 2030," International Journal of Production Economics, Elsevier, vol. 191(C), pages 154-169.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fufsci:v:4:y:2022:i:2:n:e2118. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)2573-5152 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.