IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v218y2025ics0040162525002549.html

Biases in expert judgements in large-scale S&T Delphi Surveys: How to cope with them?

Author

Listed:
  • Sokolov, Alexander
  • Grebenyuk, Anna
  • Urashima, Kuniko

Abstract

When preparing to a new large-scale science and technology (S&T) Foresight study we decided to analyse available previous Delphi surveys in order to assess the results they achieve. The most important issue for us was how to avoid biases in expert judgements, in particular related to reaching convergence in the second round of the survey compared to the first round. This article presents the results, which turned out to be unexpected for us.

Suggested Citation

  • Sokolov, Alexander & Grebenyuk, Anna & Urashima, Kuniko, 2025. "Biases in expert judgements in large-scale S&T Delphi Surveys: How to cope with them?," Technological Forecasting and Social Change, Elsevier, vol. 218(C).
    • Handle: RePEc:eee:tefoso:v:218:y:2025:i:c:s0040162525002549
    DOI: 10.1016/j.techfore.2025.124223
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162525002549
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2025.124223?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Calof, Jonathan & Colton, Brian, 2024. "Developing foresight that impacts senior management decisions," Technological Forecasting and Social Change, Elsevier, vol. 198(C).
    2. Timofei Nestik, 2018. "The Psychological Aspects of Corporate Foresight," Foresight and STI Governance, National Research University Higher School of Economics, vol. 12(2), pages 78-90.
    3. Pill, Juri, 1971. "The Delphi method: Substance, context, a critique and an annotated bibliography," Socio-Economic Planning Sciences, Elsevier, vol. 5(1), pages 57-71, February.
    4. Moonjung Choi & Han-Lim Choi, 2015. "Foresight for Science and Technology Priority Setting in Korea," Foresight-Russia Форсайт, CyberLeninka;Федеральное государственное автономное образовательное учреждение высшего образования «Национальный исследовательский университет «Высшая школа экономики», vol. 9(3 (eng)), pages 54-65.
    5. Anastassios Pouris & Portia Raphasha, 2015. "Priorities Setting with Foresight in South Africa," Foresight-Russia Форсайт, CyberLeninka;Федеральное государственное автономное образовательное учреждение высшего образования «Национальный исследовательский университет «Высшая школа экономики», vol. 9(3 (eng)), pages 66-79.
    6. repec:plo:pone00:0020476 is not listed on IDEAS
    7. Alexander Sokolov, 2009. "Future of S&T: Delphi Survey Results," Foresight and STI Governance, National Research University Higher School of Economics, vol. 3(3), pages 40-58.
    8. Anastassios Pouris & Portia Raphasha, 2015. "Priorities Setting with Foresight in South Africa," Foresight and STI Governance, National Research University Higher School of Economics, vol. 9(3), pages 66-79.
    9. Ozcan Saritas & Serhat Burmaoglu, 2015. "The evolution of the use of Foresight methods: a scientometric analysis of global FTA research output," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(1), pages 497-508, October.
    10. Moonjung Choi & Han-Lim Choi, 2015. "Foresight for Science and Technology Priority Setting in Korea," Foresight and STI Governance, National Research University Higher School of Economics, vol. 9(3), pages 54-65.
    11. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    12. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    13. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    14. Li, Na & Chen, Kaihua & Kou, Mingting, 2017. "Technology foresight in China: Academic studies, governmental practices and policy applications," Technological Forecasting and Social Change, Elsevier, vol. 119(C), pages 246-255.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Niyazov, Sukhayl & Maibakh, Olesia & Sukharev, Alexei & Kulakova, Tatiana & Ufimtsev, Alexey, 2026. "Corrigendum to “Evaluating Delphi survey accuracy in transportation: Evidence from Japanese technology foresight” [Technol. Forecast. Soc. Change 224 (2026) 124496]," Technological Forecasting and Social Change, Elsevier, vol. 226(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sokolov, Alexander & Shashnov, Sergey & Kotsemir, Maxim & Grebenyuk, Anna, 2019. "Quantitative analysis for a better-focused international STI collaboration policy: A case of BRICS," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 221-242.
    2. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    3. Belton, Ian & Wright, George & Sissons, Aileen & Bolger, Fergus & Crawford, Megan M. & Hamlin, Iain & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina, 2021. "Delphi with feedback of rationales: How large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged?," Technological Forecasting and Social Change, Elsevier, vol. 170(C).
    4. Prommer, Lisa & Tiberius, Victor & Kraus, Sascha, 2020. "Exploring the future of startup leadership development," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    5. Laura Studen & Victor Tiberius, 2020. "Social Media, Quo Vadis? Prospective Development and Implications," Future Internet, MDPI, vol. 12(9), pages 1-22, August.
    6. Volkmar, Gioia & Fischer, Peter M. & Reinecke, Sven, 2022. "Artificial Intelligence and Machine Learning: Exploring drivers, barriers, and future developments in marketing management," Journal of Business Research, Elsevier, vol. 149(C), pages 599-614.
    7. Fritschy, Carolin & Spinler, Stefan, 2019. "The impact of autonomous trucks on business models in the automotive and logistics industry–a Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 148(C).
    8. Alexander Sokolov & Sergey Shashnov & Maxim Kotsemir, 2021. "From BRICS to BRICS plus: selecting promising areas of S&T Cooperation with developing countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8815-8859, November.
    9. Peppel, Marcel & Ringbeck, Jürgen & Spinler, Stefan, 2022. "How will last-mile delivery be shaped in 2040? A Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 177(C).
    10. Merfeld, Katrin & Wilhelms, Mark-Philipp & Henkel, Sven & Kreutzer, Karin, 2019. "Carsharing with shared autonomous vehicles: Uncovering drivers, barriers and future developments – A four-stage Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 144(C), pages 66-81.
    11. Ozcan Saritas & Derrick Ababio Anim, 2017. "The Last and Next 10 Years of Foresight," HSE Working papers WP BRP 77/STI/2017, National Research University Higher School of Economics.
    12. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    13. Prianto Budi Saptono & Gustofan Mahmud & Intan Pratiwi & Dwi Purwanto & Ismail Khozen & Muhamad Akbar Aditama & Siti Khodijah & Maria Eurelia Wayan & Rina Yuliastuty Asmara & Ferry Jie, 2023. "Development of Climate-Related Disclosure Indicators for Application in Indonesia: A Delphi Method Study," Sustainability, MDPI, vol. 15(14), pages 1-25, July.
    14. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    15. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    16. Georg Aichholzer, 2002. "Das ExpertInnen-Delphi: methodische Grundlagen und Anwendungsfeld ‘Technology Foresight‘ (The Expert Delphi: Methodology and Application in 'Technology Foresight')," ITA manu:scripts 02_01, Institute of Technology Assessment (ITA).
    17. Hayes, Tom, 2007. "Delphi study of the future of marketing of higher education," Journal of Business Research, Elsevier, vol. 60(9), pages 927-931, September.
    18. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    19. Haarhaus, Tim & Liening, Andreas, 2020. "Building dynamic capabilities to cope with environmental uncertainty: The role of strategic foresight," Technological Forecasting and Social Change, Elsevier, vol. 155(C).
    20. Ribeiro, Barbara E. & Quintanilla, Miguel A., 2015. "Transitions in biofuel technologies: An appraisal of the social impacts of cellulosic ethanol using the Delphi method," Technological Forecasting and Social Change, Elsevier, vol. 92(C), pages 53-68.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:218:y:2025:i:c:s0040162525002549. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.